All Blogs / Analytics & Forecasts

How OOS Is Used Matters More Than the Label

16 September 2026, 11:00

Dan Mishima

“OOS tested” sounds reassuring.

And in principle, it should.

Out-of-Sample testing matters because it gives us a way to check whether an EA still behaves reasonably on data that did not directly help build it. If an EA is developed on one historical period and then tested unchanged on another period, that later period can provide useful evidence.

But a label alone is not enough.

When reading an OOS result, a more important question is:

What had already been decided before the OOS was viewed, and what changed afterward?

That single question can change the meaning of the result.

A separate date range is not automatically a clean OOS test

Imagine an EA is built on 2014–2022 data.

Then it is tested on 2023–2024.

So far, that is a normal OOS test.

But if the developer sees that result and then changes the parameter, changes the candidate, changes the gate, or changes the portfolio structure — and reruns that same OOS period — the role of that period starts to change.

It is no longer acting only as a reserved check.

It is now feeding information back into the next development decision.

That is why two OOS results with the same calendar dates can still mean very different things.

One may be a cleaner confirmation step.

The other may already be part of a selection-and-adjustment loop.

The same date range does not guarantee the same evidential value.

OOS can influence selection even when no parameter changes

This point is easy to miss.

Suppose you create 100 EA candidates.

All 100 survive Development.

Then you run all 100 on the same OOS period.

Their results are different.

Some look strong. Some are average. Some collapse.

If you keep only the single EA with the best OOS result, you did not change any parameter. But the OOS result still determined which EA survived.

That means OOS was not used only to confirm an already-chosen EA.

It also helped choose the EA.

This is one reason that a very good OOS result should not be read in isolation.

Another important question is:

Was this EA already fixed before OOS, or was it selected because of OOS?

With enough candidates, some will look unusually strong on that reserved period just by chance. That possibility becomes more important as the candidate pool grows.

Clean OOS becomes stronger when more decisions are frozen first

If you want OOS to function mainly as confirmation, it helps to lock down as much as possible before you look at the result.

That may include:

which candidates will be tested,
the PASS / FAIL gate,
portfolio membership,
weighting and risk allocation,
and the acceptance rule.

The sequence matters.

A cleaner process looks like this:

Define the candidate or portfolio.
Define the rule for passing or failing.
Freeze the relevant choices.
View OOS.
Accept or reject.

That does not make the result perfect or future-proof.

It simply means the OOS period is doing a clearer job: checking a pre-defined decision rather than helping create the decision after the fact.

If you change the candidate, gate, weight, or composition after seeing OOS, that OOS result has now influenced the next version. It becomes part of development for what comes next.

The key question is not “Was it OOS?” It is “What was fixed before anyone saw it?”

The same issue applies to portfolios, not just single EAs

Suppose several candidates survive Development.

You then use OOS to decide:

which strategies stay in the portfolio,
how many strategies to keep,
how much weight each one gets,
or which risk allocation to adopt.

In that case, OOS is being used to help construct the portfolio.

So there is an important difference between:

“This portfolio looked good on OOS.”

and

“This portfolio was already fixed before OOS, and it also looked good there.”

The dates may be identical.

The decision process is not.

OOS is still useful — but only if you ask the right follow-up questions

None of this means OOS testing is bad.

Quite the opposite.

OOS is useful precisely because it can challenge what was built earlier.

But when you read “OOS tested,” it helps to ask two more questions:

Before OOS was viewed, how much had already been fixed?
After OOS was viewed, were any candidates, parameters, gates, weights, or portfolio choices changed?

Those answers tell you much more than the OOS label alone.

The value of OOS does not come only from using a different period.

It comes from protecting the order of decisions.