Backtest Depth - Long-term Robustness (10 years) vs. Recent Adaptation (5 years)?

New comment
 

Hello community. I'm evaluating the reliability of my EAs and a question has come up regarding the optimal historical depth for backtesting.

My dilemma is this: a 10-year backtest theoretically proves that the EA can survive multiple market regimes. However, the current market doesn't behave the same way it did a decade ago (different volatility, institutional algorithms, etc.). Do you prefer to optimize over the last 3-5 years to adapt to current market conditions, or do you require your systems to be profitable over 10+ year periods?

I'd appreciate hearing about your experiences.

 
Isaac Uriel Arenas Caldera:

Hello community. I'm evaluating the reliability of my EAs and a question has come up regarding the optimal historical depth for backtesting.

My dilemma is this: a 10-year backtest theoretically proves that the EA can survive multiple market regimes. However, the current market doesn't behave the same way it did a decade ago (different volatility, institutional algorithms, etc.). Do you prefer to optimize over the last 3-5 years to adapt to current market conditions, or do you require your systems to be profitable over 10+ year periods?

I'd appreciate hearing about your experiences.

Personally, I don't adapt didley squat. If I get any extended ugly periods of time in my testing, I go back to the drawing board. 10 years is my self-imposed minimum for say, a swing strategy that trades 2 or 3 times per week. Basically, the length of my backtest is determined by the trading frequency of my given strategy. For a scalping strategy that trades over a hundred times per month, I find that several months of testing will suffice.

I should note that I've only recently come around to scalping. Interestingly, changing market conditions seem to affect swing trading more than scalping. The market always seems to be moving just a little bit which is enough for scalping in almost any regime.

 

Thanks for the reply!

By the way, you mention:

If I get any extended ugly periods of time in my testing, I go back to the drawing board.



Curious about this, so firstly you start with an idea from scratch and test it, if the graph isn't ok you move into another strat?

Or is there a period of optimizing for certaing market conditions, etc...

 

Isaac Uriel Arenas Caldera #:
Curious about this, so firstly you start with an idea from scratch and test it, if the graph isn't ok you move into another strat?

Or is there a period of optimizing for certaing market conditions, etc...

Yes. My workflow generally goes as follows:

  1. Select a market (FX, futures, etc.).
  2. Select an symbol/contract based on its attributes─predominantly ranging or trending?
  3. Select a chart structure/timeframe─the "Uniformity Factor" script in the CodeBase will often answer item #2 (above) when run on each chart structure/timeframe. By "structure," I mean custom Renko bricks, range bars, equal volume bars, etc.
  4. For a "rangey" symbol, focus on reversion logic. For a "trendy" symbol, focus on breakout logic.
  5. Anecdotally/visually evaluate and/or code experimental indicators, tools, and/or utilities on chart history. If I don't see potential "profit leads" formed on the chart (this is inherently more often the case than not), clear the slate and start over.
  6. Assuming that item #6 was cleared, refine/tweak and insert/remove indicators, tools, and/or utilities.
  7. Code the EA.
  8. Run a backtest. I prefer to code my EA's to trade on OHLC data, so I have no issues testing on OHLC prices only. I should note that this requires knowing/researching your broker-dealer average spread, slippage, and your trade execution latency─which should be entered into the Tester. There are utilities in the CodeBase for collecting such data.
  9. At this point, I will know whether the EA is worth its salt. I will not attempt to "resurrect" a junk EA by optimizing. Instead, clear the slate and start over.
  10. If I have an unknown setting for a specific input to explore, I may optimize that one setting. I'm really not a fan of "blanket" optimizing nor optimizing in general.
If this sounds like a grind, that's because it is a grind.
     
    Isaac Uriel Arenas Caldera:

    Hello community. I'm evaluating the reliability of my EAs and a question has come up regarding the optimal historical depth for backtesting.

    My dilemma is this: a 10-year backtest theoretically proves that the EA can survive multiple market regimes. However, the current market doesn't behave the same way it did a decade ago (different volatility, institutional algorithms, etc.). Do you prefer to optimize over the last 3-5 years to adapt to current market conditions, or do you require your systems to be profitable over 10+ year periods?

    I'd appreciate hearing about your experiences.

    Hello. Pleased to meet you.


    Wow 10 year backtest. I tend to backtest for 2 years, for example from 2024 to today. I feel that the backtest doesn't need to be long nor can it be too short. I once made an EA, then I backtested it for a long period, the EA survived and made a profit, but it very rarely traded, there could even be no trading for 1 month. Sometimes we set EA to survive in various conditions, but the result is that the filter becomes too tight so it is difficult to achieve this in the current conditions. My standard is 2 years with the "Every Tick base on real tick" option. What I remember is "Past performance does not guarantee future results" so there is no guarantee that EA will produce results like the backtest.
     
    Is there any broker providing 10yrs of data in tick quality ?
     
    Icham Aidibe #:
    Is there any broker providing 10yrs of data in tick quality ?
    Discussion/referencing brokers is prohibited on the forum. Do your own search
     
    Oleksandr Medviediev #:
    Discussion/referencing brokers is prohibited on the forum. Do your own search
    How many RAM needed to load 10y of tick data in the tester ?
     

    This is the wrong framing, and that's why it's hard to answer. The right question isn't "5 or 10 years", it's "how does your strategy's regime sensitivity interact with the timeframe you're trading?"

    A few things worth considering:

    1. The horizon depends on your asset class and timeframe, not on a personal preference.

    The empirical research (Pesaran & Timmermann 2007 on optimal window selection, plus the broader literature on regime detection in financial time series) shows that the right backtest window scales roughly with the timeframe's typical regime persistence:

    • M15 forex majors: meaningful regimes shift every ~1.5-2 years
    • H1 forex majors: every ~4-6 years
    • D1 forex majors: every ~7-10 years
    • Forex minors / exotics: typically 1.5-2x longer due to lower liquidity-driven autocorrelation

    So if you're running an H1 EA, neither 5 nor 10 years is "right." 5 years probably covers 1-2 regime cycles. 10 years covers 2-3. Both are defensible. What matters is whether your backtest spans enough regimes, not whether it's exactly 5 or 10.

    2. The volatility ratio between your backtest and the present is what actually matters.

    If you're in a high-volatility regime now (and we are, post-2022), a 10-year backtest with 70% of its data in calmer markets is going to overstate your strategy's stability. Volatility-adjust the window: shorten it in high-vol regimes, lengthen it in low-vol ones.

    3. "Robust across 10 years" can mean two very different things.

    Either:

    • The strategy genuinely adapts to multiple regimes (rare, valuable)
    • The strategy happens to break even across the bad years and shine in the good ones — masquerading as "robust" when it's actually averaging

    The second is the trap. A strategy that's profitable on 10 years but loses money in 4 of those 10 years is not robust — it's lucky in the aggregate. Test the consistency window-by-window, not just the cumulative.

    4. Practical answer for most retail traders:

    • For an EA with <500 trades total: lean toward the longer window (10y) — you need the sample size more than you need recent representativeness
    • For an EA with >2000 trades and active trading in the last 12-24 months: 5 years is fine, supplement with sample adequacy tests
    • For everything in between: split-sample test. Optimize on 5 years, walk-forward on a held-out 2-year recent slice. If parameters are stable across the split, the choice between 5 and 10 doesn't matter much. If they're wildly different, you have an adaptation problem regardless of horizon.

    5. The thing nobody talks about: re-optimization frequency is the real question.

    If you re-optimize quarterly, you barely need 5 years of history — you only need enough to fit current parameters well. If you set-and-forget, you need 10+ years AND structural robustness tests. The horizon question only makes sense paired with the question of how often you'll re-tune.

    References worth reading:

    • Bailey & López de Prado (2014): The Deflated Sharpe Ratio — relevant for understanding why long backtests with positive Sharpe might still be overfit
    • Pesaran & Timmermann (2007): Selection of estimation window in the presence of breaks — directly addresses your question
    • Politis & Romano (1994): The stationary bootstrap — foundation for proper backtest validation
    • Hall, Horowitz & Jing (1995): On blocking rules for the bootstrap with dependent data — Biometrika

    Hope this helps.

     
    Bambang Christianto #:

    Hello. Pleased to meet you.


    Wow 10 year backtest. I tend to backtest for 2 years, for example from 2024 to today. I feel that the backtest doesn't need to be long nor can it be too short. I once made an EA, then I backtested it for a long period, the EA survived and made a profit, but it very rarely traded, there could even be no trading for 1 month. Sometimes we set EA to survive in various conditions, but the result is that the filter becomes too tight so it is difficult to achieve this in the current conditions. My standard is 2 years with the "Every Tick base on real tick" option. What I remember is "Past performance does not guarantee future results" so there is no guarantee that EA will produce results like the backtest.
    For wich timeframe?
     
    Botond Ratonyi #:

    The empirical research (Pesaran & Timmermann 2007 on optimal window selection, plus the broader literature on regime detection in financial time series) shows that the right backtest window scales roughly with the timeframe's typical regime persistence:

    • M15 forex majors: meaningful regimes shift every ~1.5-2 years
    • H1 forex majors: every ~4-6 years
    • D1 forex majors: every ~7-10 years
    • Forex minors / exotics: typically 1.5-2x longer due to lower liquidity-driven autocorrelation

    I find this quite interesting, tbh!

    Thanks for sharing.

    123456
    New comment