Which test has killed the most “great” EAs for you?

New comment
 
We all know how easy it is to produce an impressive-looking backtest.

The interesting part is what happens when you actively try to break it.

I’m curious about real experience here, not textbook answers:

Which test has exposed the greatest number of apparently excellent but actually fragile EAs for you?

For example:

• Out-of-sample testing
• Parameter sensitivity / nearby parameter values
• Walk-forward analysis
• Monte Carlo simulation
• Higher spreads and slippage
• Different brokers / execution conditions
• Different market regimes
• Forward testing with frozen parameters

Personally, I find one of the biggest warning signs is when a strategy looks exceptional at one exact parameter combination but deteriorates sharply with only small changes around it.

A beautiful optimum is nice.

A stable neighborhood around that optimum is much more convincing.

But I’m interested in what other developers have actually seen in practice.

Have you ever had an EA that looked almost perfect in the Strategy Tester, only for one specific validation test to completely destroy the idea?

What test exposed it?
 

Totally agree on parameter stability! For me, Higher Spreads & Variable Slippage is the ultimate EA killer—especially for short-term/scalping strategies. A system can show a beautiful equity curve with fixed 10-point spreads, but once you stress-test it with 20–30 point spreads and realistic execution delays, 80% of the edge completely vanishes.

 
Frozen-parameter walk-forward is usually the fastest killer for me. I keep the settings unchanged across several unseen windows and raise spread, commission and execution delay together. If it survives only one window or one exact parameter set, the edge is probably fitted rather than robust.
 
Nattanan Ekeda #:

Totally agree on parameter stability! For me, Higher Spreads & Variable Slippage is the ultimate EA killer—especially for short-term/scalping strategies. A system can show a beautiful equity curve with fixed 10-point spreads, but once you stress-test it with 20–30 point spreads and realistic execution delays, 80% of the edge completely vanishes.

That’s a very good point, especially for short-term systems.

Execution-cost stress can completely change the picture because a strategy may look robust statistically while depending on an execution environment that is simply too “clean”.

I also like testing spread, slippage and delay together rather than independently, because that is closer to what actually happens under difficult conditions.

Do you usually stress spreads as a fixed number of points, or relative to the symbol’s normal spread?
 
Guad Bibar #:
Frozen-parameter walk-forward is usually the fastest killer for me. I keep the settings unchanged across several unseen windows and raise spread, commission and execution delay together. If it survives only one window or one exact parameter set, the edge is probably fitted rather than robust.
I like that approach a lot.

Keeping the parameters completely frozen across several unseen windows is a very clean way to separate robustness from optimization luck.

For me, the interesting part is not whether every window is profitable, but whether the behavior remains structurally consistent without the strategy needing constant re-tuning.

How many unseen windows do you normally want to see before you start trusting the result?
New comment