ForumSections

Testing a trading system for robustness on MT5

New comment

I am convinced I may be overfitting my EA to historical data somewhat.

Based on what I know, the methods available to test a system's robustness (and identify overfitting) in MT5 include:

  • Using the built-in forward test/out-of-sample function in the strategy tester (and testing on other OOS data manually)
  • 2D/3D visualizer (parameter stability analysis i.e. sensitivity analysis)
  • Exporting backtest/forward test data to excel for analysis (e.g. pivot tables and colour coding)
  • Validating results on other instruments and other timeframes

But I know that outside of MT5 there are many more robustness tests available such as:

  • Randomized Out of Sample Testing
  • Vs. Random
  • Vs. Others
  • Noise Testing
  • Monte Carlo Analysis
  • Monte Carlo Reshuffle
  • Monte Carlo Resample
  • Monte Carlo Permutation
  • Monte Carlo Randomized
  • Variance Testing
  • Delayed Testing
  • Parameter Optimization / Parameter Stability Testing
  • Noise Testing Parameter Optimization

I know that QuantAnalyzer has some MonteCarlo analysis and 'what if' scenario testing that I might incorporate.

My Questions:

  1. Am I missing any of MT5's robustness testing features?
  2. Are there other tools that easily interface with MT5 (like QuantAnalyzer) that would give me access to other robustness testing? I would really like to do Randomized Out of Sample Testing, Noise Testing, Variance testing 


Thank you all,

Jim

traderjim42:
QuantAnalyzer
traderjim42:

My Questions:

  1. Am I missing any of MT5's robustness testing features?
  2. Are there other tools that easily interface with MT5 (like QuantAnalyzer) that would give me access to other robustness testing? I would really like to do Randomized Out of Sample Testing, Noise Testing, Variance testing 

As you have mentioned QuantAnalyzer - that's the top currently in this field. 

Any complex systems and math EAs are prone to curve-fitting. Try it on different brokers, that could be an eye opener.  

Oleksandr Medviediev #:

As you have mentioned QuantAnalyzer - that's the top currently in this field. 

Any complex systems and math EAs are prone to curve-fitting. Try it on different brokers, that could be an eye opener.  


Thanks for your response, that's good to know. 

The performance of my EA changes a lot between brokers, but I tend to attribute this to the different data feeds that different brokers use.

STEP-1: set your own quality criteria for EA metrics - what is acceptable and what is not.
Oleksandr Medviediev #:
STEP-1: set your own quality criteria for EA metrics - what is acceptable and what is not.
Also a good idea, thanks
New comment