Why constantly optimizing fails to work. Proof.
http://gifmaker.me/PlayFrameAnimation.php?folder=20140803173NE4ZBodV7NHtAUIIdUoVA
In the link posted I created a gif image of 3d optimization graphs. This is an EA trading EURUSD. there are 25 optimization passes/ settings combinations. Starting from january 2012 for every month each graph frame in the gif represents a 3 month test. You can clearly see the problem of always preparing EA settings for the next month setting based on the last few months.
The setting combination that performs the best had a very small chance to be the best and is very unlikely to be the best again next month.
This problem keeps traders always chasing, and never catching the best profit.
My advice: Use many symbols with the same settings. Run an "all symbols in market watch test" making sure only the symbols you'd want to trade are in the market watch window. If the cumulative profit/loss on all the symbols is positive, use those symbols settings and stick with it for years. As favorable behaviors on some symbols disappears, they may appear in other symbols. The point is not to overfit a system to any specific symbol, timeframe, or period in the past, it should generally work across all markets.
What are your thoughts?
My advice: Use many symbols with the same settings. Run an "all symbols in market watch test"...
What are your thoughts?
I agree with you.
Like I've said in a previous post (https://www.mql5.com/en/forum/34731)
the main problem is that the history has a too little variability to check the limits of an EA.
Instead, test and discover the limits of an EA is a high priority, and history is unsuitable and long-term misleading to achieve this.
An ideal EA should be able to not lose in a totally chaotic market condition (= white noise).
With the tools that we have, testing a single configuration with all the symbols is the best stress test available for an EA.
![[PROPOSAL] Use of a Quotes Synthetizer for deeper testing of EAs](/i/community/logo_mql5_small.png "[PROPOSAL] Use of a Quotes Synthetizer for deeper testing of EAs"){kind=logo}
- www.mql5.com
I agree with you.
Like I've said in a previous post (https://www.mql5.com/en/forum/34731)
the main problem is that the history has a too little variability to check the limits of an EA.
Instead, test and discover the limits of an EA is a high priority, and history is unsuitable and long-term misleading to achieve this.
An ideal EA should be able to not lose in a totally chaotic market condition (= white noise).
With the tools that we have, testing a single configuration with all the symbols is the best stress test available for an EA.
Hello
Interesting idea. But the idea already exists between statistician and econometrician. Also the limitation of the method is well known too.
One thing you should be careful is that why do you think sine wave or other Synthetic data represents the property or characteristics of historical price or future price. If they don't, it is almost no point to go though this kind of extended simulation because they won't reveal anything.
One thing we know for sure is historical variation. You might tackle this using Monte Carlos simulation approach. Variation is only one property of our price series among many other properties. So it is far from being perfection and satisfaction in my view.
I hope this comment is useful for your future development work and good luck with them.
Kind regards.
It is difficult to generalize that constant optimization fails because simply many profitable system is based on constant optimization. Even manual traders, somehow they will constantly optimize their trading strategy.
This will be done in the manner they look at historical performance of their trading strategy either monthly or quarterly.
Constant optimization is good when you do it to avoid the same mistake you made in the past. However, constant optimization is bad if it tricks you to see only optimistic future which is not really existing.
So use constant optimization for right context.
Kind regards.
Sorry to anyone wanting to view the GIF at this time, apparently the file is only stored on their server for a few days.
I believe every symbol got their own movement range. So EA might work on this and not that. The design of the EA plays a part too.
The problem is that every symbols movement range/ oscillation / forces / behaviors, whatever you want to call it, changes. For example December's EURUSD 2015 behavior may be like AUDCAD's March 2001 behavior (theoretical not actual, just making a point) or what not- the only way I would trust an EA to work in the future is if it can work on over 50% of the symbols I want to use with the same settings. Because you are right, each symbol has a different behavior- but their behavior is not exclusive even to it's own symbol, month to month the variation can be crazy, year to year also. We can never trust the "best setting" from an optimization to work tomorrow. We must subject an EA, trying one pair of settings on all symbols because over any unit of time all symbols exhibit different behaviors.
I agree with you.
Like I've said in a previous post (https://www.mql5.com/en/forum/34731)
the main problem is that the history has a too little variability to check the limits of an EA.
Instead, test and discover the limits of an EA is a high priority, and history is unsuitable and long-term misleading to achieve this.
An ideal EA should be able to not lose in a totally chaotic market condition (= white noise).
With the tools that we have, testing a single configuration with all the symbols is the best stress test available for an EA.
Exactly, one single configuration, all symbols really is the beset stress test, thanks for agreeing with me ;) and i think a white noise test with random small medium large and massive price swings is a great stress test as well. I actually have a fantastic idea about how to implement this type of test, but right now I'm spending all my time working and studying in a summer session class at school and have no time for coding for the rest of the month. If anyone wants to try to code the idea private message me.
It is difficult to generalize that constant optimization fails because simply many profitable system is based on constant optimization. Even manual traders, somehow they will constantly optimize their trading strategy.
This will be done in the manner they look at historical performance of their trading strategy either monthly or quarterly.
Constant optimization is good when you do it to avoid the same mistake you made in the past. However, constant optimization is bad if it tricks you to see only optimistic future which is not really existing.
So use constant optimization for right context.
Kind regards.
The highlighted part: Absolutely agree with this. And it can be hard for traders to understand the difference between a bad trading plan and a good trading plan's bad month or two.
Say if a breakout strategy worked on a symbol last year, and all of a sudden it becomes fake-out season on the instrument, when do (or should) we decide to adjust the algorithm to trade against fake-outs? It is possible that by the time we notice and adjust the algorithm, that fake-out season will end and normal breakouts occur again- this is all theoretical and overly simplified problems, but even so are real problems none-the-less.
When using EA's I wouldn't ever pick the one that says "use only on XY symbol with ABC settings", it either means 1) it does not work on many symbols or 2) the programmer didn't care to test it enough so why should I trust the code in the EA? Anyway, that's why I learned to program in the first place :) I was just a ballroom dance teacher when I first got my feet wet trading ~10 years ago, sometime later I bought Fap Turbo and was thoroughly disappointed when it lost thousands of my dollars- and my hatred for those guys increased every day, they never answered my emails, not once, I tried other EAs too with the same bad results until I broke and decided to learn to program.
Before I ramble on with my whole life's story I'll just finish now and say thanks for your replies guys, I'm sorry the GIF isn't there anymore... I'll try to post it here as an inserted image, if it works you'll see a gif with changing images, if not it will just be a picture of the first frame.
Ok before "Add coment" i can see the gif changing images, lets see if it works after posting...
Nice. that's better.
If someone want's to do this it would be awesome to see ~260 tests starting at almost every day of the week (M-F) for one year, each of which are one year tests. I don't have time for a while. I'm guessing the green islands might gradually instead of dramatically shift as in the last image which was starting every month for test periods of 3 months.
You can optimize GIF animations, also online.
Try this, for example: http://ezgif.com/optimize
this is reduced to 577 Kb:
- ezgif.com
I believe every symbol got their own movement range. So EA might work on this and not that. The design of the EA plays a part too.
Sorry to anyone wanting to view the GIF at this time, apparently the file is only stored on their server for a few days.
The problem is that every symbols movement range/ oscillation / forces / behaviors, whatever you want to call it, changes. For example December's EURUSD 2015 behavior may be like AUDCAD's March 2001 behavior (theoretical not actual, just making a point) or what not- the only way I would trust an EA to work in the future is if it can work on over 50% of the symbols I want to use with the same settings. Because you are right, each symbol has a different behavior- but their behavior is not exclusive even to it's own symbol, month to month the variation can be crazy, year to year also. We can never trust the "best setting" from an optimization to work tomorrow. We must subject an EA, trying one pair of settings on all symbols because over any unit of time all symbols exhibit different behaviors.
Exactly, one single configuration, all symbols really is the beset stress test, thanks for agreeing with me ;) and i think a white noise test with random small medium large and massive price swings is a great stress test as well. I actually have a fantastic idea about how to implement this type of test, but right now I'm spending all my time working and studying in a summer session class at school and have no time for coding for the rest of the month. If anyone wants to try to code the idea private message me.
The highlighted part: Absolutely agree with this. And it can be hard for traders to understand the difference between a bad trading plan and a good trading plan's bad month or two.
Say if a breakout strategy worked on a symbol last year, and all of a sudden it becomes fake-out season on the instrument, when do (or should) we decide to adjust the algorithm to trade against fake-outs? It is possible that by the time we notice and adjust the algorithm, that fake-out season will end and normal breakouts occur again- this is all theoretical and overly simplified problems, but even so are real problems none-the-less.
When using EA's I wouldn't ever pick the one that says "use only on XY symbol with ABC settings", it either means 1) it does not work on many symbols or 2) the programmer didn't care to test it enough so why should I trust the code in the EA? Anyway, that's why I learned to program in the first place :) I was just a ballroom dance teacher when I first got my feet wet trading ~10 years ago, sometime later I bought Fap Turbo and was thoroughly disappointed when it lost thousands of my dollars- and my hatred for those guys increased every day, they never answered my emails, not once, I tried other EAs too with the same bad results until I broke and decided to learn to program.
Before I ramble on with my whole life's story I'll just finish now and say thanks for your replies guys, I'm sorry the GIF isn't there anymore... I'll try to post it here as an inserted image, if it works you'll see a gif with changing images, if not it will just be a picture of the first frame.
Ok before "Add coment" i can see the gif changing images, lets see if it works after posting...
Simply speaking, model building for trading is as highly uncertain as trading itself.
At every single step, your best bet is to stick with most sound statistical decision. It is because you don't want to under or over estimate your models.
In my view, trading model is very honest, as long as you know what you are building, they will do exactly how you specified no more or no less.
To be honest, constant optimization is not too much issue for model builder. More important thing to discuss is what is the data you work with. (i.e. Stationarity or non statationarity to start with, and many more to consider).
In my view, econometrics and time series forecasting books are the bible for this. They will tell you most of answer what model builder want to know.
Good book by reputable Professors in the area.
Forecasting Economic Time Series By Michael Clements, David Hendry
Plus I like your opintion on fab turbo. I really support on it. :) :)
Kind regards.
- Free trading apps
- Over 8,000 signals for copying
- Economic news for exploring financial markets
You agree to website policy and terms of use
http://gifmaker.me/PlayFrameAnimation.php?folder=20140803173NE4ZBodV7NHtAUIIdUoVA
In the link posted I created a gif image of 3d optimization graphs. This is an EA trading EURUSD. there are 25 optimization passes/ settings combinations. Starting from january 2012 for every month each graph frame in the gif represents a 3 month test. You can clearly see the problem of always preparing EA settings for the next month setting based on the last few months.
The setting combination that performs the best had a very small chance to be the best and is very unlikely to be the best again next month.
This problem keeps traders always chasing, and never catching the best profit.
My advice: Use many symbols with the same settings. Run an "all symbols in market watch test" making sure only the symbols you'd want to trade are in the market watch window. If the cumulative profit/loss on all the symbols is positive, use those symbols settings and stick with it for years. As favorable behaviors on some symbols disappears, they may appear in other symbols. The point is not to overfit a system to any specific symbol, timeframe, or period in the past, it should generally work across all markets.
What are your thoughts?