Which test has killed the most “great” EAs for you? - page 2
You are missing trading opportunities:
- Free trading apps
- Over 8,000 signals for copying
- Economic news for exploring financial markets
Registration
Log in
You agree to website policy and terms of use
If you do not have an account, please register
Two that aren't on the list yet, both cheap.
1. Shift the start date
Test your strategy three times, moving the start date forward by two or three weeks each time. If the top-performing configurations change rankings between tests, it may indicate that the good result was merely a calendar coincidence. Standard backtesting misses this because the initial date always stays fixed.
2. Compare the level of data detail
Test the best results using two levels of precision: first with 1-minute data, and then with maximum detail (tick-by-tick). Compare whether the top configurations remain in the same order, regardless of profit. If the ranking changes, your strategy's edge relies on microscopic movements within each candle (such as the exact point where a stop loss triggers), which typically fails in practice.
Both tests detect flaws immediately and usually do not require additional tools.
Good additions. I especially like the idea of looking at ranking stability between parameter sets, rather than simply asking whether the final profit remains positive.
Shifting the start date can expose a strategy that is accidentally dependent on a very specific historical sequence, while the data-detail comparison can reveal whether the edge survives once intrabar execution becomes more realistic.
The second test is particularly useful because a strategy can look robust at candle level while actually depending on very precise stop/entry ordering inside the bar.
I would still treat both as diagnostic tests rather than absolute pass/fail rules, but they are exactly the kind of inexpensive checks that can eliminate weak candidates before spending time on heavier robustness testing.
Good additions to the list.
Well said. No single trade nor small window of trades should be focused on. Emotionally responding to the outcome of a single trade or small window of trades is a frequent pitfall of manual trading.
Stated a bit differently, focus on the large sample, the whole large sample, and nothing but the large sample. Personally, I prefer a sample of 4000 round trades or more. When the dust settles, I have statistics that don't need to be changed.
Completely agree on the importance of the large sample.
My only nuance would be that I would not make 4,000 trades a universal requirement, because a lower-frequency strategy may never produce that many trades without spanning so many years that we start mixing very different market regimes.
For me, the real objective is to have a sample large enough for the expectancy, variance and drawdown statistics to become reasonably stable — and then verify that this stability survives different periods and conditions.
A high-frequency EA may need thousands of trades. A slower system may need fewer, but stronger evidence across independent market regimes.
Either way, the principle is the same: judge the distribution, not the last trade.
I would not use a fixed number of unseen windows as the main acceptance criterion.
Five weakly independent windows can give less evidence than three windows covering genuinely different regimes.
I would look at:
The important question for me is not “how many windows passed?” but “how much independent evidence did those windows actually add?”
After that I would still keep one final untouched holdout that was never used to redesign either the strategy or the validation procedure.
Also worth trying: perturb the entry timing instead of the parameters. Add a small random delay to each entry (1-5 min) and keep the logic as-is. If the results fall apart, the edge was probably riding on hitting an exact instant, not a real structural edge.
Mostly kills scalping/short-term stuff though - swing or trend systems usually don't care much either way. Complements playgold's point above about trades that look too similar in timing/setup.
Two that aren't on the list yet, both cheap.
1. Shift the start date
Test your strategy three times, moving the start date forward by two or three weeks each time. If the top-performing configurations change rankings between tests, it may indicate that the good result was merely a calendar coincidence. Standard backtesting misses this because the initial date always stays fixed.
2. Compare the level of data detail
Test the best results using two levels of precision: first with 1-minute data, and then with maximum detail (tick-by-tick). Compare whether the top configurations remain in the same order, regardless of profit. If the ranking changes, your strategy's edge relies on microscopic movements within each candle (such as the exact point where a stop loss triggers), which typically fails in practice.
Both tests detect flaws immediately and usually do not require additional tools.
Excellent additions, Carlos.
Shifting the start date is especially useful because it exposes strategies that are unintentionally dependent on a very specific market window. If a configuration remains among the strongest when the test period is displaced several times, confidence in the underlying structure increases considerably.
I also agree on comparing modelling precision. If a strategy only preserves its ranking under extremely specific intrabar assumptions, that is something worth investigating before treating the result as robust.
Both are simple tests, but they can reveal weaknesses that a conventional single backtest may completely hide. Thanks for adding them.
I would not use a fixed number of unseen windows as the main acceptance criterion.
Five weakly independent windows can give less evidence than three windows covering genuinely different regimes.
I would look at:
The important question for me is not “how many windows passed?” but “how much independent evidence did those windows actually add?”
After that I would still keep one final untouched holdout that was never used to redesign either the strategy or the validation procedure.
I agree. A fixed number of OOS windows is useful as a procedural rule, but not as a substitute for independence.
Three genuinely different regimes can indeed tell us more than five windows that are essentially variations of the same environment.
I especially agree with looking at IS → OOS degradation, parameter neighbourhood stability and execution-cost sensitivity rather than simply asking whether OOS remained profitable.
For me, the strongest validation is when the strategy survives several independent attacks without changing its parameters or validation rules along the way.
And yes — keeping one final untouched holdout is important. Otherwise the validation process itself can gradually become another form of optimization.
Good distinction between “number of windows passed” and “amount of independent evidence accumulated.”
Also worth trying: perturb the entry timing instead of the parameters. Add a small random delay to each entry (1-5 min) and keep the logic as-is. If the results fall apart, the edge was probably riding on hitting an exact instant, not a real structural edge.
Mostly kills scalping/short-term stuff though - swing or trend systems usually don't care much either way. Complements playgold's point above about trades that look too similar in timing/setup.
Very good point. Entry-time perturbation is a simple stress test that can reveal a surprisingly fragile edge.
If shifting execution by only a few minutes destroys the result, the strategy may be exploiting an extremely narrow timing coincidence rather than a broader market structure.
I also agree that the interpretation depends heavily on the strategy type. A scalping system can legitimately be much more timing-sensitive than a swing or trend-following system, so I would not use the same tolerance for every architecture.
What I like about this test is that it perturbs execution, not the strategy logic itself. If the edge remains recognizable after small timing disturbances, that adds another useful piece of robustness evidence.
Excellent complement to the other stress tests discussed above.