How do you test an adaptive EA properly?

New comment
 

i am working on an adaptive EA and I’ve started wondering about something that is not easy to solve.

With a normal EA, we can optimize it on historical data and then test it on data that the EA has never seen before. That part is quite clear.

But an adaptive EA is different.

The EA can change its behaviour depending on what is happening in the market. For example, it may detect a trend, a ranging market, high or low volatility, and then use different settings or trading rules. It may also adjust itself after getting more data.

This makes me wonder where we should draw the line between adapting to the market and overfitting the historical data.

There is another problem as well.

We can start changing the testing method itself.

For example, we might try different training periods, walk-forward periods, recalculation frequencies, regime settings, etc. If we keep changing these things after seeing the results, we could eventually end up fitting the testing process to the same historical data.

So even if the EA passes an out-of-sample test, I’m not sure that automatically means the result is reliable.

At the moment, I’m thinking about using things like:

  • walk-forward testing;
  • a final out-of-sample period that is never touched during development;
  • Monte Carlo testing;
  • testing the EA on different symbols;
  • testing different market conditions;
  • realistic spread and slippage;
  • and fixing the testing procedure before looking at the final results.

But I’d like to hear from people who have actually dealt with this problem.

If you have developed an adaptive or regime-based EA, how do you test it without slowly tuning the whole testing process to historical data?

Do you keep the adaptation rules fixed from the beginning?

Or do you allow them to change during development and then lock everything before the final test?

I’m interested in what has worked for you in real testing, especially if you have taken an EA from backtesting to forward testing or live trading.

 

I think the main risk is validation overfitting, not only parameter overfitting.

Before looking at the final OOS result, I would freeze the entire evaluation procedure: adaptation rules, parameter ranges, walk-forward windows, objective function, execution costs, regime/symbol set, and Monte Carlo rules.

Then use one completely untouched final OOS period only once.

I would also compare the adaptive version against the same strategy with fixed parameters. If adaptation only looks better after repeatedly changing the walk-forward procedure, that may be evidence of overfitting the validation process rather than genuine robustness.

For me, the strongest evidence would be: predefined rules, realistic costs, several regimes/symbols, parameter perturbation, Monte Carlo, and one untouched final OOS run with no further tuning.


 
You stick it on a demo and bypass back test and see what it can do ! . 
 
One thing nobody's brought up yet: the leakage isn't just in the data, it's in our own head too.

Even if you lock the walk-forward windows and regime rules before the final OOS run, you've probably already got a feel for that period just from living through it, or from months of iterating while that data existed. That's a softer kind of overfitting and I don't think a single untouched final test really fixes it — your sense of what counts as a reasonable rule got shaped by already knowing how things played out.

One thing that's helped: write down the adaptation logic before you ever touch the OOS block. What regimes you're defining, how you detect them, why. If the only justification you can give for a rule is pointing at a backtest curve, that's worth questioning. I've also had some luck running the adaptation logic on synthetic data with fake regime switches first, before it sees any real history. If it can't pick up on regime changes in data that has zero connection to any real market, it's probably just fitting noise, not actually adapting to anything. And it doesn't cost you any real OOS data to check.
 
Could you please tell us more about how you use the Monte Carlo method?
 

The part that cost me most was not the walk forward, it was counting how many variants I had looked at. I once tested a time cycle hypothesis across 121 candidate lags. Measured against the base rate the best cell came out at p 0.036, which reads like a finding. A permutation test on the maximum gave 0.947, so 95 percent of pure noise runs produce a lag somewhere that is at least as strong. With 121 candidates the expected maximum z from noise alone is around 2.7, and an adaptive EA is exactly that kind of search over a family of parameters.

The second thing I would check is whether the adaptation is aimed at something structural. In my data the structural part of an edge carries from one half of the sample to the other with a beta of 0.90, while a cell that has just turned significant keeps about 11 percent of its jump in the next period. So I shrink a fresh anomaly by roughly 90 percent before I believe it, and I select on the whole history instead of on the last window.

 

On the Monte Carlo question - the version I get the most out of for an adaptive EA isn't resampling the equity curve, it's resampling the adaptation itself. Keep the entry and exit logic fixed, then re-run the same sample with the regime labels randomly permuted and with the switch dates shifted by a random offset; if the shuffled runs produce roughly the same spread of results as the real one, the adaptation isn't carrying information and you are mostly measuring the base edge. Alongside that I would bootstrap the trade sequence with replacement (a few thousand paths) to look at the drawdown distribution instead of the single historical path, and perturb each adaptive parameter to check the surface is a plateau rather than a spike. The important part is that these Monte Carlo rules get frozen together with everything else before the final OOS block is touched, otherwise the randomness quietly becomes one more thing you tune.

 
There is a cheaper test than any of the Monte Carlo work, and it catches the embarrassing case.

Does the adaptive branch change the trade list at all?

On a client EA I had four configurations that differed by one input. They produced two results, not four. The input everyone believed in was on the identical side, byte for byte. The summary lines differed by rounding, which is exactly why it had looked like four different outcomes.

The way to see it: run the same period for each setting you care about, sort every trade list, and hash it. Two configurations with the same hash mean that switch did nothing on that data, and everything measured after it is really about the base system.

The other half is counting. Make the EA print which branch it took on every entry, then count the entries per branch. A regime label that fires nine times in three years was never tested, whatever the optimiser reports about its parameters.
 
Yogesh Kaushik:


You have a trading strategy that produces good results after optimization on one specific section of historical data. You then optimize the same strategy on another, different section of the historical data, and you obtain a different set of optimal parameters. You repeat this process across several different historical sections — for example, five or more independent periods — and each period produces its own optimal parameter set.

The objective is to identify the relationship and correlation between the different parameter values and the market conditions represented by each historical period.

New comment