Optimisation robustness and avoiding overfitting MT5
When I first saw this post, I thought it was spam and removed it, but then realised it was in fact a question and restored it.
It is so long that I doubt most users will take the time to read it. You may want to condense a bit.
Also, you mention "In instances similar to the screenshot above", but there are no images. So, given that your rating is still too low, attach the images instead.
Let me help you with what I know since I'm currently working with robustness and overfitting in the backtest/forward test.
for optimization first step,
- for optimization more then 3 parameter it is better to do it recursively to avoid overfitting.
for second step,
The reason I decided to focus this analysis on the forward tests data and not the backtest data is because the forward test occurs on the most recent market history
- as far as I know the data for foward test is an out-of-sample data, meaning that it is not what actually happened on the real market for that specified time. e.g when you compare last month chart with what forward test chart it shows different chart because the data is out-of-sample.
for third step,
- I think diversifying is a great way to reduce risk and it can help the equity curve since I also do it and it really help with smoothing of equity curve.
I found this article hope this helps
https://www.mql5.com/en/blogs/post/753653
- 2023.09.10
- www.mql5.com
When I first saw this post, I thought it was spam and removed it, but then realised it was in fact a question and restored it.
It is so long that I doubt most users will take the time to read it. You may want to condense a bit.
Also, you mention "In instances similar to the screenshot above", but there are no images. So, given that your rating is still too low, attach the images instead.
For those who dare to read, please see the attached image I was referring to. This is an example of several 'plausible' setfiles that could be traded in live conditions. However, trading all of these seems impractical, and I would like to refine a process that could identify the 2-4 most robust setfiles.
Open to any advice on this.
Let me help you with what I know since I'm currently working with robustness and overfitting in the backtest/forward test.
for optimization first step,
- for optimization more then 3 parameter it is better to do it recursively to avoid overfitting.
for second step,
- as far as I know the data for foward test is an out-of-sample data, meaning that it is not what actually happened on the real market for that specified time. e.g when you compare last month chart with what forward test chart it shows different chart because the data is out-of-sample.
for third step,
- I think diversifying is a great way to reduce risk and it can help the equity curve since I also do it and it really help with smoothing of equity curve.
I found this article hope this helps
https://www.mql5.com/en/blogs/post/753653
Thanks Luandre,
1. - By recursively, you mean optimise 3 parameters, and use that to identify promising settings, and then in a separate optimisation, run a more refined optimisation on another 3 different parameters ? That makes sense to me. I will test this.
2. When I say forward test, I am tasking about the out of sample forward test that the mt5 strategy tester runs after the optimisation (when the strategy tester graph resets the equity). I do not agree that this is not actually what happened in the real market, since it is using real market data (albeit historical data). The whole point of doing this forward test is to see whether the forward test results are supportive of the backtest/optimisation results (i.e. create similar results).
3. Agreed.
Thanks for your comment
James
Yes that is correct, the second optimization will use the the parameter from the first optimization. But this is something that I currently working on also. I open up a thread on this forum with title 'optimization many parameter' if you want to know about it.
It is an actual data with different period if I understand. Let's say your period is from september to december and at december the EA will start the forward testing. As far as I know the data for december will not the actual data from the december it could be from august or somewhere near that period. That's why it is called out-of-sample because it is not from those period. It is kinda useless if the data that will be used is from that december period since backtest also use that data
Luandre Ezra #:
Okay thanks I'll look for that thread - 'optimization many parameter'
I think you may be confused about the forward testing option.
Let's say you select a 4 month period for your backtest, and you select the 'Forward 1/4' option. This will run a backtest/optimisation on the first 3 months of the period, and it will then run the forward test on the the 4th and final month of the period. So the 4th month is still considered out of sample because it was not included in the backtest/optimisation sample data (which was only the first 3 months).
- Free trading apps
- Over 8,000 signals for copying
- Economic news for exploring financial markets
You agree to website policy and terms of use
Context:
This post is intended to be useful to those looking for ideas on how to set up an optimisation/setfile discovery process for themselves. Much more selfishly, I am also looking for advice from the more experienced on my optimisation process (which I will lay out below). I have been trading for years but MUCH more successfully in recent years using EA's (one in particular that I have developed). I am currently using the best process I can based on my current understanding, but I would love your insight to help better my understanding and improve my process.
The EA:
The EA I use could be categorised as a grid EA, but it has a 'delay' input which prevents it from trading constantly. And it's not using pending orders like typical grid EA's might. For example, after a very solid bullish move in price, the EA is likely to enter (it may be long or short, depending on the settings). Tough to explain, but it is highly versatile and can trade 'mean reversion' style or 'momentum/trend' style. The main aspect to understand is that once it has been triggered to open a trade, it will keep entering trades as the subsequent trades go into drawdown, looking to close that sequence of trades together with an overall profit, and the 'stop loss' is an equity stop loss which the user can set to any level they want. If you were trading a very small account you could set this equity stop loss to ~$100-200 and use lot sizes of 0.01-0.02 and larger accounts would be proportionally scaled up. Other features of the EA include an EMA filter, the ability to let the distance between trades be determined by ATR values, and more. The EA provides enough opportunity for diversification across a number of different approaches, even if these approaches are all applied to the same currency pair (i.e. I can organise a portfolio of strategies to be trading simultaneously, without the possibility of them all taking a loss on the same day)
The Strategy:
The fundamental strategy is to optimise the EA rather frequently to ensure it is in tune with current market conditions, therefore constantly taking small profits and avoiding the comparatively large losses. An example could be each month, running a 4 month backtest, with a 1/4 forward test (i.e. last month is the forward test). Like all trading, there is a trade off between aggression and defensiveness, and a balance must be struck. The settings you choose to run must be safe enough to not have unwanted drawdown levels, but aggressive enough to be able to build up profit buffers that can withstand inevitable drawdown without losing weeks upon weeks of profits (and remain profitable in the long run). Diversification is an important part of the overall strategy as discussed above.
The Optimisation Process - This is where I need your advice:
I am going to lay out what I do, and what I consider, and please shoot holes in my process where you see issues.
1. I have identified over time which fx pairs tend to work well with the EA in general. I will run a 4 month optimisation with a 1/4 (i.e. 1 month) forward test on each of these pairs. I run complex criterion as the optimisation object. On each of these optimisations, I will be optimising about 6-7 parameters (e.g. one might be pips start 3, and test up to 30 in steps of 3). I know that this is a risk for overfitting, but I have tried optimisations with fewer parameters being tested (e.g. 3-4 parameters) and I will often get poor results (i.e. results that I wouldnt trade live), especially on longer tests of 4+ months. So I figure that if you are getting poor results, then you can't be overfitting, so increasing the parameters to 6-7 (and increasing the degree of freedom it has to find historically successful results) is justified in order to find good settings, even though it by definition must be increasing the risk of overfitting. I do generally aim to keep the degrees of freedom as low as possible and I dont optimise any superfluous settings (unless Im being experimental).
2. Once the test is complete, I will export the forward test data into an excel workbook and filter for results that were consistently good between the backtest and the forward test. To do this I will filter out all passes that have a score lower than 99 on the forward or backtest. An example is below:
The reason I decided to focus this analysis on the forward tests data and not the backtest data is because the forward test occurs on the most recent market history, and the backtest data is factored in through the 'Back Result' column anyway where I filter out anything less than a 99 score.
At this point, depending on the size of the backtest and the instrument, there may only be a 1-2 acceptable settings. In that case I would just run them both, assuming they are sufficiently different to each other, then move onto another pair to repeat the process.
However in the example above, we see a number of potential settings we could use. To refine further, I would look at the equity curves on the back/forward tests and looks to select for steady consistent gains versus settings that have large one-off wins.
I will also trust the results that have more trades, although I am generally quite happy with 100+ trades per month as I think this is statistically significant enough.
I will also look for any parameters that seem to be appearing more often in the successful shortlist of 99+ settings, as this would suggest that they are more robust than the others. Although I would REALLY like a better way to evaluate the robustness of parameters.
By this point I can usually identify 2-3 passes that stand out to me as superior and I will run these in the portfolio for that month.
3. Repeat the process above across a number of currency pairs and other instruments. In addition to diversifying through the asset/instrument, I will also run optimisations that identify different types of strategies (the most simple example being testing for settings that only trade during Asian session hours), such that once the whole portfolio is up and running, which might be 5 assets, and 1-2 instances of the EA trading a different strategy on each of them, it would be effectively impossible for all of those individual strategies to take a loss on the same day. Diversifying like this across assets and across strategies seems to be the secret to success.
Other notes:
Statistical significance. This EA trades in 'sequences', meaning each individual trade is not independant to the next, since often 3 or 4 trades will open, but they are all part of the same overall sequence of trades (and will close together at the same time). So assuming each sequence has an average of 2.5 trades, based on my understanding, I think that the below is a reasonable minimum amount of trades for 3 month backtest and 1 month forward test.
Minimum amount of trades (In sample): 650
Minimum amount of trades (Out of sample): 125
My main question:
In instances similar to the screenshot above, where there are a number of potentially profitable settings, how can I run a robustness analysis (either in MT5 or excel)? I remember MT4 used to have a grid/matrix thing you could look at to see which 'areas' of your backtest are generally profitable. I would love to be able to do this on excel (similar to what they guy explains in the first link below) but I'm not sure how to given that I am often optimising 6-7 parameters. (?)
Other threads I've considered:
https://www.mql5.com/en/forum/ 451696
https://www.forexfactory.com/ thread/167720-systematic- trading
https://www.tradelikeamachine.com/blog/over-optimization/part-1-have-you-over-fitted-your-trading-system
Thanks for reading and I hope it's useful. I'm open to any discussion on robustness and optimisation. Any other tips you might've picked up along your journeys.