EA Validation: From Backtesting to Real Trading

New comment

For me, EA validation should be a multi-stage process rather than relying on a single backtest.

I would normally look at:

  1. Backtesting over different market conditions, not only the period where the strategy performs best.

  2. Out-of-sample and forward testing to check whether the logic remains stable outside the optimization period.

  3. Sensitivity testing. Small changes in parameters should not completely destroy the strategy. If they do, overfitting may be a problem.

  4. Drawdown and exposure analysis. I consider equity drawdown, maximum exposure and position sizing at least as important as total profit.

  5. Execution testing. Spread, slippage and broker conditions can make real results different from backtests.

  6. Real-account validation with conservative risk. Ultimately, I think a real track record is one of the most valuable stages because it tests the strategy under conditions that cannot be perfectly reproduced in the Strategy Tester.

I also prefer evaluating robustness over a sufficiently large sample rather than focusing on win rate alone.

A strategy with excellent historical statistics but poor robustness is much less interesting to me than a system with controlled risk and stable behavior across different conditions.

One check I’d add is trade-distribution stability. Compare trade count, average holding time and profit contribution by month or session between the backtest and forward period. An EA can keep a decent overall result while one market regime or a small group of trades is carrying the entire edge.
One stage I would add is an execution-state test matrix, not only a price-series test. Run the same logic with wider spreads, delayed fills, rejected orders, partial closes, terminal restarts, and two events arriving in a different order. Then verify invariants: only one intended position, stop remains attached, recovery reconstructs state, and no duplicate order is sent.

I also separate parameter robustness from execution robustness. A smooth parameter surface can still fail live if the EA assumes perfect event order. If a small timing or fill change breaks state, that is a software defect before it is a strategy result.

Thank you both for the comments. I think you have raised two aspects that complement the idea of robustness I was trying to describe.

Guad Bibar, I particularly agree with your point about the distribution of results. It is not enough for the aggregate statistics to look good; we also need to understand where the profits are actually coming from. Looking at the stability of expectancy, number of trades, profit concentration, and performance across different periods and market regimes can reveal weaknesses that a simple PF or drawdown figure may not show.

That said, I would not necessarily expect every month or every segment to be profitable, since that could itself become another form of overfitting. What matters to me is whether the overall edge is excessively dependent on a particular period, regime, or a small number of trades.

Arnold Holm, I also think your comment highlights a fundamental distinction: strategy robustness and implementation robustness are not the same thing.

A strategy can be statistically robust and still have a fragile EA implementation. Wider spreads, slippage, execution delays, rejected requests, partial fills, terminal restarts, or unexpected transaction sequences can expose problems that are completely invisible in a conventional backtest.

So I would treat these as separate but complementary validation layers. First, we need to establish that the strategy has a genuine edge and that it survives different data, periods, parameter perturbations, and market regimes. Then we need to verify that the EA can actually execute and maintain that edge under imperfect execution conditions.

In that sense, I increasingly see robustness as multidimensional:

data → parameters → trade distribution → market regimes → execution → EA state management.

A strong backtest only answers part of these questions. The real test is whether the system continues to behave coherently when we deliberately perturb those dimensions without having optimized for them beforehand.

Ultimately, an EA that performs well in backtests but cannot reliably maintain its internal state under unexpected execution events is not truly robust. It may contain a good strategy, but it is still sitting inside a potentially fragile implementation.

[Deleted]

Really thorough breakdown. One thing I'd add that's separate from the execution/robustness layers already covered here: raw sample size, before anything else is even meaningful.


Below roughly 30 closed trades I treat every other stat (PF, drawdown, expectancy) as provisional almost regardless of how good it looks, because a strategy with zero real edge can still string together a good-looking run through pure variance (same reason a fair coin landing 8/10 heads in a small sample isn't suspicious, it's just what small samples do). Above 100+ trades the numbers start meaning closer to what they claim to.


On Jorge's "aggregate stats can look fine" point: I'd promote Sharpe ratio above profit factor or raw return specifically, because it's harder to fake with one lucky stretch. Two backtests can show the same total return where one earned it steadily and the other got most of it from one or two outsized trades against an otherwise flat/losing base, PF and total return look similar, Sharpe usually doesn't.

One thing this thread treats as a reported statistic rather than what it actually is: drawdown is the binding constraint, not a score.

Every other number here is descriptive. Profit factor, expectancy, Sharpe — they tell you what happened. Drawdown tells you what you are allowed to do next. It sets your position size, it sets whether you can survive the bad stretch long enough to reach the good one, and on a funded or prop account it is the only number that can end the account outright. A system with half the return and a third of the drawdown is not a worse system; it is a system you can run at larger size.

There is a second reason to weight it, and I think it is the more important one. Almost every metric in a backtest is optimistic — fills are perfect, spread is average, slippage is absent, the order never gets rejected. Drawdown is the rare one that usually understates rather than overstates, because the live conditions that hurt you (widened spread, a stop filled well past its level, an execution failure like the one FXprism described) all deepen it and none of them appear in the tester. So when a backtest and live run diverge, return tends to fall and drawdown tends to rise. Treating the tester's drawdown as a floor rather than an estimate is the safer reading.

This is also why I would put it ahead of sample size in the ordering someone suggested above. You need enough trades before the statistics mean anything, yes — but you need the drawdown to be survivable before you are allowed to collect those trades at all. The strategies that never get their hundred-trade sample are the ones that got stopped out of the account at trade forty.

The suggestions here are very good and entirely valid. I would add that backtesting should be done to analyze the MFE/MAE route of operations to verify maximum profits and losses.

I would also add that it is quite interesting to segregate the inputs first and then the outputs so that the outputs do not compensate for weak inputs.

Jorge Vazquez Suarez:

For me, EA validation should be a multi-stage process rather than relying on a single backtest.

I would normally look at:

  1. Backtesting over different market conditions, not only the period where the strategy performs best.

  2. Out-of-sample and forward testing to check whether the logic remains stable outside the optimization period.

  3. Sensitivity testing. Small changes in parameters should not completely destroy the strategy. If they do, overfitting may be a problem.

  4. Drawdown and exposure analysis. I consider equity drawdown, maximum exposure and position sizing at least as important as total profit.

  5. Execution testing. Spread, slippage and broker conditions can make real results different from backtests.

  6. Real-account validation with conservative risk. Ultimately, I think a real track record is one of the most valuable stages because it tests the strategy under conditions that cannot be perfectly reproduced in the Strategy Tester.

I also prefer evaluating robustness over a sufficiently large sample rather than focusing on win rate alone.

A strategy with excellent historical statistics but poor robustness is much less interesting to me than a system with controlled risk and stable behavior across different conditions.

Compare backtest and live on the same metrics, over the same kind of sample, instead of eyeballing the equity curves. Export the tester's deal list and your live deal history for that EA (filter live by magic number), then compare per-trade average profit, win rate, average win/loss, trade count per week, average holding time and average spread or slippage at entry. Run the tester on real ticks with the broker's spread and commission settings, because a gap in average cost per trade is the most common reason live results fall short. To judge whether the gap is more than luck, bootstrap the backtest trades: resample them at your live trade count many times and see where your live total profit falls in that distribution. If it sits in the bottom 5%, something structural differs (costs, fills, regime); if it is well inside the range, you are probably just seeing normal variance. Do this only once you have roughly 100 or more live trades, and track drawdown on the live account against the tester's worst case, since live drawdown usually comes out deeper.
One stage I would add before forward testing: run the backtest from every start date, not just one.

On my current five strategy portfolio, the single run from 5 January showed a 14.75% maximum drawdown against the deposit. Checked across all 194 start dates, the worst start went 15.90% below its deposit. Close in this case, but I only trusted the first figure once I had seen the second.

On an older four strategy set, the same check from every start date since 2021 gave an 88% pass rate on prop challenge rules, with a median of 50 days. A single backtest cannot give you either of those numbers.

hello..@Jorge Vazquez Suarez

I agree with most of this. One thing I would add is Monte Carlo and stress testing. A strategy can look robust in backtests and still depend heavily on one particular trade sequence or execution condition.
I’d also test the strategy with worse-than-normal spread, slippage and commission assumptions, not just the historical values.
For me, the goal isn’t to find the most profitable backtest. It’s to find out how much performance can deteriorate before the strategy becomes unacceptable.

So my process would be roughly:
Backtest → out-of-sample → walk-forward → parameter sensitivity → Monte Carlo/stress testing → forward/live validation.

If the strategy remains reasonably stable through those stages, I have much more confidence in its robustness than from a high return or win rate alone.

Lots of good stuff here already. @Abdullah Uygar Tuna bootstrap idea is the cleanest way I've seen to put a number on "is the live gap just bad luck".

One thing I'd add, because I learned it the expensive way on prop accounts: write down the stop rule before the EA goes live, not after it starts hurting. And not only for depth. Most people set a max drawdown, but almost nobody sets a limit for how long it can stay flat.
If the longest flat stretch in the backtest was 3 months and live is now at 5, that's a signal too, even if the drawdown still looks fine.

Once it's losing money it's very easy to find reasons to give it one more week. That's exactly why I want the rule written down first.

best regards,
Chris
New comment