Which test has killed the most “great” EAs for you?

New comment
We all know how easy it is to produce an impressive-looking backtest.

The interesting part is what happens when you actively try to break it.

I’m curious about real experience here, not textbook answers:

Which test has exposed the greatest number of apparently excellent but actually fragile EAs for you?

For example:

• Out-of-sample testing
• Parameter sensitivity / nearby parameter values
• Walk-forward analysis
• Monte Carlo simulation
• Higher spreads and slippage
• Different brokers / execution conditions
• Different market regimes
• Forward testing with frozen parameters

Personally, I find one of the biggest warning signs is when a strategy looks exceptional at one exact parameter combination but deteriorates sharply with only small changes around it.

A beautiful optimum is nice.

A stable neighborhood around that optimum is much more convincing.

But I’m interested in what other developers have actually seen in practice.

Have you ever had an EA that looked almost perfect in the Strategy Tester, only for one specific validation test to completely destroy the idea?

What test exposed it?

Totally agree on parameter stability! For me, Higher Spreads & Variable Slippage is the ultimate EA killer—especially for short-term/scalping strategies. A system can show a beautiful equity curve with fixed 10-point spreads, but once you stress-test it with 20–30 point spreads and realistic execution delays, 80% of the edge completely vanishes.

Frozen-parameter walk-forward is usually the fastest killer for me. I keep the settings unchanged across several unseen windows and raise spread, commission and execution delay together. If it survives only one window or one exact parameter set, the edge is probably fitted rather than robust.
Nattanan Ekeda #:

Totally agree on parameter stability! For me, Higher Spreads & Variable Slippage is the ultimate EA killer—especially for short-term/scalping strategies. A system can show a beautiful equity curve with fixed 10-point spreads, but once you stress-test it with 20–30 point spreads and realistic execution delays, 80% of the edge completely vanishes.

That’s a very good point, especially for short-term systems.

Execution-cost stress can completely change the picture because a strategy may look robust statistically while depending on an execution environment that is simply too “clean”.

I also like testing spread, slippage and delay together rather than independently, because that is closer to what actually happens under difficult conditions.

Do you usually stress spreads as a fixed number of points, or relative to the symbol’s normal spread?
Guad Bibar #:
Frozen-parameter walk-forward is usually the fastest killer for me. I keep the settings unchanged across several unseen windows and raise spread, commission and execution delay together. If it survives only one window or one exact parameter set, the edge is probably fitted rather than robust.
I like that approach a lot.

Keeping the parameters completely frozen across several unseen windows is a very clean way to separate robustness from optimization luck.

For me, the interesting part is not whether every window is profitable, but whether the behavior remains structurally consistent without the strategy needing constant re-tuning.

How many unseen windows do you normally want to see before you start trusting the result?

I agree with all that was said above, but usually there's lower hanging fruit: EAs that trade very frequently and/or have low expectancy per trade tend to fail backtests on real ticks when logged in to a live account. Many (maybe most) of them in the Market here have their published backtests (if any) made in "Every tick" mode (fake ticks) or/and using demo account quotes, or worse in 1-minute OHLC mode. Delays and extra slippage (beyond available real tick quotes) aren't even necessary to expose this.

If this doesn't fail, I try on a related instrument that likely wasn't a direct optimization target, such as XAUEUR instead of XAUUSD. Unfortunately, interpretation of the result of such stress-test requires a deeper look into the strategy and actual trades. Some are genuinely very specialized to the original instrument and that's fine. Some others may pass test with the other instrument yet rely on prior knowledge of historical market behavior, such as the metals and stock markets mostly going up that year. Yet it's a fine quick test to run that doesn't require a forward testing period, and it provides an extra data point for consideration.

Good point — especially regarding low expectancy per trade. Those systems can look excellent in a simplified tester environment and then collapse as soon as real tick structure, execution conditions and realistic spreads are introduced.

I also like the idea of testing a related instrument as a diagnostic stress test, not necessarily as a pass/fail requirement. A strategy can be legitimately specialized for one market, but if a small change in instrument completely destroys the behaviour, that tells us something useful about how dependent the edge may be on the original optimization target.

For me, the important distinction is exactly what you mention: the result always needs interpretation. Robustness testing is not about forcing every EA to work everywhere — it is about discovering what assumptions the strategy is actually relying on.

That is often more valuable than the headline profit factor itself.

Here's a new one: test on unrelated and/or negatively correlated instruments, and look not at the trading result but at the trades themselves - are they maybe too similar to trades that are made for the intended instrument? It's sufficient to look at a brief period like a month. This reveals recorded or seeded pseudo-random trade times or setups. This would have exposed a certain EA here in the Market that sold 200+ copies in August. Would not kill "the most" "great" EAs, so not directly an answer to your original question, but is a good addition to our arsenal.

That said, for some EAs having deterministic trades is fine and expected - it just needs to be disclosed as (part of) the strategy used (time-based patterns or such). The problem is when the EA is marketed as something else entirely and uses overfitted historical trade times rather than repeating patterns.
playgold #:

Here's a new one: test on unrelated and/or negatively correlated instruments, and look not at the trading result but at the trades themselves - are they maybe too similar to trades that are made for the intended instrument? It's sufficient to look at a brief period like a month. This reveals recorded or seeded pseudo-random trade times or setups. This would have exposed a certain EA here in the Market that sold 200+ copies in August. Would not kill "the most" "great" EAs, so not directly an answer to your original question, but is a good addition to our arsenal.

That said, for some EAs having deterministic trades is fine and expected - it just needs to be disclosed as (part of) the strategy used (time-based patterns or such). The problem is when the EA is marketed as something else entirely and uses overfitted historical trade times rather than repeating patterns.

That’s a very interesting distinction — look at the trades, not the P/L.

Using an unrelated or negatively correlated instrument as a diagnostic test could reveal something that a normal robustness test may miss: whether the EA is reacting to actual market structure or simply reproducing highly specific timing/setup patterns inherited from the development sample.

I also agree that deterministic behaviour itself is not necessarily a problem. If time-based logic is genuinely part of the strategy and is disclosed as such, that can be perfectly legitimate.

The problem starts when apparent market intelligence is actually historical timing memorization dressed up as an adaptive strategy.

I wouldn’t use this as a universal pass/fail test either, but as another forensic tool in the robustness toolbox. Good addition.

Two that aren't on the list yet, both cheap.

1. Shift the start date
Test your strategy three times, moving the start date forward by two or three weeks each time. If the top-performing configurations change rankings between tests, it may indicate that the good result was merely a calendar coincidence. Standard backtesting misses this because the initial date always stays fixed.

2. Compare the level of data detail
Test the best results using two levels of precision: first with 1-minute data, and then with maximum detail (tick-by-tick). Compare whether the top configurations remain in the same order, regardless of profit. If the ranking changes, your strategy's edge relies on microscopic movements within each candle (such as the exact point where a stop loss triggers), which typically fails in practice.

Both tests detect flaws immediately and usually do not require additional tools.

Juan Miguel Alzugaray Galarregui #:
For me, the interesting part is not whether every window is profitable, but whether the behavior remains structurally consistent without the strategy needing constant re-tuning.

Well said. No single trade nor small window of trades should be focused on. Emotionally responding to the outcome of a single trade or small window of trades is a frequent pitfall of manual trading.

Stated a bit differently, focus on the large sample, the whole large sample, and nothing but the large sample. Personally, I prefer a sample of 4000 round trades or more. When the dust settles, I have statistics that don't need to be changed.

12
New comment