All Blogs / Analytics & Forecasts

Why We Split Backtest Data Into Three Periods

15 September 2026, 11:00

Dan Mishima

A long backtest is useful, but its length is not the whole story.

If the same historical period is used to build the EA, adjust its rules, select the best candidate, and judge the final result, the backtest can become increasingly polished without giving us a clean answer to a more important question:

How does the finished EA behave on data that did not help create it?

That is why we separate historical data by role.

In our current framework, the basic structure is:

2014–2022: Development
2023–2024: Validation
2025–2026 H1: Final OOS

The dates are not the main point.

The important part is deciding in advance which data may influence development, which data is used to check the work, and which data is kept out of the decision process until the end.

Why not use the entire history for development?

Using all available history can feel efficient.

If a weak period appears, you can add a filter.
If losses cluster in one condition, you can adjust the entry.
If a market phase performs poorly, you can tighten the rules.

Every new test gives immediate feedback.

The problem is that the same dataset gradually starts doing two jobs:

helping you build the EA, and
helping you judge whether the EA is good.

Once enough decisions have been made while looking at the full history, that history is no longer a clean test of the finished system.

The issue is not limited to parameter tuning.

Historical results can influence logic changes, trading hours, filters, candidate selection, and which versions are rejected or kept.

So instead of spending all the data at once, we deliberately reserve later periods for different purposes.

The EA looked good on the data used to build it.
What happened when it was taken outside that development period?

That is a more useful question than simply making the original curve smoother.

Each period has a different job

Development — 2014 to 2022

Development is where the system is built.

This is the period for:

forming hypotheses,
generating and testing candidates,
changing rules,
rejecting weak ideas,
and refining the structures that survive.

Iteration belongs here.

Validation — 2023 to 2024

Validation changes the question.

Instead of asking how to improve the candidate, we ask whether the candidate still behaves reasonably on a separate period.

A disappointing validation result is not automatically a problem to repair.

Sometimes it is useful evidence that the original edge was more fragile than expected.

If Validation becomes another unrestricted tuning period, the distinction between Development and Validation starts to disappear.

Final OOS — 2025 to 2026 H1

Final OOS should serve a different purpose again.

By this point, the candidate and the relevant decision rules should already be fixed.

The final period is not another place to optimize.

It is a place to observe what happens when previously made decisions are carried into separated data.

In simple terms:

Development: build
Validation: check
Final OOS: final untouched review

Keeping those roles separate makes the result easier to interpret.

What happens if Final OOS changes the decision?

Suppose the Final OOS result is weaker than expected.

It is tempting to make one more adjustment:

change a filter,
modify the entry,
keep a different candidate,
change the portfolio mix,
or relax an acceptance rule.

Then the Final OOS result improves.

But the role of that period has changed.

Once the result is used to change the system or the selection decision, the period is no longer untouched.

It has become part of development.

This is why the useful question for Final OOS is not:

How can we fix the system after seeing this result?

It is:

What happened when we took our fixed decisions and exposed them to data we had not been developing against?

Good or bad, the result has value because it was not part of the tuning loop.

Data splitting does not make the future predictable

Separating Development, Validation, and Final OOS does not prove future profitability.

Final OOS is still historical data.

Future market regimes can differ. Trade counts may be limited. Execution conditions may change.

Other checks are still useful, including:

Monte Carlo analysis,
cost stress,
parameter stability,
harsher execution assumptions,
and portfolio-level dependency checks.

Data splitting is not a guarantee.

Its value is more practical:

it creates a point where the system stops receiving more answers from the same history.

That distinction helps reduce the risk of mistaking repeated historical adjustment for genuine robustness.

What to ask when you see a long backtest

“Backtested for more than 10 years” sounds reassuring.

But length alone is not enough.

A better set of questions is:

Was the entire period used during development?
Was a separate period reserved for validation?
Was there a final period that remained outside the decision process?
Were rules, candidates, or portfolio choices changed after those later results were seen?

Two EAs can both show ten years of history and still provide very different evidence depending on how that history was used.

So when you evaluate a backtest, do not ask only:

How long is it?

Also ask:

Which part was used to build the EA, and which part was kept aside to challenge it?