Ok. First let me explain linear optimization before I get to my question.
Say you're testing a basic EMA crossover system from 2000.01.01 - 2015.07.04.
You run an MT4 optimization for one of the EMA's. The following values result (excluding drawdown and others for simplicity's sake):
Initial Equity: $5,000.00
Period Net Profit
After running this test, which value would you select to use for your system going forward? Which would you assume has the highest statistical chance for having a relatively similar level of profit in the future?
If you selected "80," there's an extremely strong chance you would end up with returns much lower than you'd be expecting, and much lower than those you could possibly achieve using this exact same system. In this example, "52" would in fact be the wisest value to use since it yielded relatively strong returns while also having the strongest statistical probability of continuing to produce those returns in the future. When running an optimization, neighboring parameter values should yield roughly the same level of returns to offer the tester confidence in that his system truly captured the movement it was meant to, and that the optimized value was not due to over-fitting.
My question is this; When you have a larger number of parameters, or anything 2+, how would you not succumb to this pitfall? Say you're attempting to optimize both EMA's as well as all three aspects of the MACD and another factor. How would you select the value more similar to "52" above rather than "80"? Would you select one parameter at a time using the above method? Would you run an optimization of every parameter and use those optimized values?
Interested to see what people here think of this topic.
Have a great one.
Drawdowns, despite your phrasing, DO happen, and have a potential effect on your potential income, under any system. Slippage does as well, and that needs to be considered as well. I know you did it to keep it simple, but I'm not totally sure doing that is a good idea because doing that does not give you the whole picture.
You can have the most kick ass system, but if you do not have some method (a generic stop loss, not the specific one that people seem to go hunting for) to stop the market from taking back all (and more) of the money you make on a trade, then the system, no matter how much money it makes you on each trade, can't really be all that good. To me, if it were a choice between using a system that would possibly get me huge returns every month, but had a large possibility to wipe out my account on very few of those trades each month, or a system that made me less per month, but was more stable, and had a lot lower chance of wiping out my account, even if it was a slightly larger loss than my best trade, but overall I made a significantly larger number of successful trades, and overall, was making enough money to cover the losses (due to drawdowns, slippage, or just bad trades occasionally) and extra to grow my account balance, I would go with the second system.
That is my opinion anyway. But that all depends on what level of risk someone is willing to take with their money as well. That might work fine for other people.
The method that I am going to try next was based out of a book I read. (the only thing useful in it.)
At the end of the optimization (OnTester) append to a file every trade, profit, all the parameters, TF, and Symbol. Optimize multiple TFs and multiple symbols.
Move the data to a spread sheet, and compute for each parameter, TF, and symbol, the average profit / standard deviation.
It should be a rounded curve. Choose each parameter value with the highest ratio; parameter value with the most consistent profit.
If you have separate peeks, that indicates to me that your parameters are not all independent.
I understand the rounded curve, but where would the individual peaks be located?
The parameters can all work synergistcally. This obviously could lead to even more wonderful results, but we must be careful when applying complexity through additional parameters since their optimization could more and more easily simply tell a perfect story of the past.
My best intuition would be a bit out of the scope of MT4. Using a graphical optimization analysis, a new dimension could be added with each additional parameter. To the extent of my knowledge, the truthfully optimal values would be those which are located in the absolute center of it all. This would force you to be more creative and manual with your optimizations, which, with theory behind them, would be much more sound and beneficial than computerized optimization. Using this method, say the lower extent of the object is 0% annual returns, while the highest is 80% but with 25% of the weight. You could select the basket of parameter values which yielded ~ 50% year and be pretty comfortable in the system's ability since there are reasonably equal returns from the neighboring values in every direction.
To me, that would be the *cough* easiest way to select the "52" as opposed to the "80" in my first example.