How do you test an adaptive EA properly?

New comment
 

i am working on an adaptive EA and I’ve started wondering about something that is not easy to solve.

With a normal EA, we can optimize it on historical data and then test it on data that the EA has never seen before. That part is quite clear.

But an adaptive EA is different.

The EA can change its behaviour depending on what is happening in the market. For example, it may detect a trend, a ranging market, high or low volatility, and then use different settings or trading rules. It may also adjust itself after getting more data.

This makes me wonder where we should draw the line between adapting to the market and overfitting the historical data.

There is another problem as well.

We can start changing the testing method itself.

For example, we might try different training periods, walk-forward periods, recalculation frequencies, regime settings, etc. If we keep changing these things after seeing the results, we could eventually end up fitting the testing process to the same historical data.

So even if the EA passes an out-of-sample test, I’m not sure that automatically means the result is reliable.

At the moment, I’m thinking about using things like:

  • walk-forward testing;
  • a final out-of-sample period that is never touched during development;
  • Monte Carlo testing;
  • testing the EA on different symbols;
  • testing different market conditions;
  • realistic spread and slippage;
  • and fixing the testing procedure before looking at the final results.

But I’d like to hear from people who have actually dealt with this problem.

If you have developed an adaptive or regime-based EA, how do you test it without slowly tuning the whole testing process to historical data?

Do you keep the adaptation rules fixed from the beginning?

Or do you allow them to change during development and then lock everything before the final test?

I’m interested in what has worked for you in real testing, especially if you have taken an EA from backtesting to forward testing or live trading.

New comment