Two developers can report a strong OOS result for the same dates while using that period in different ways. One freezes an EA and its acceptance criteria before opening the period. The other tests several candidates there, adjusts the selection and publishes the best result. Both used a separate date range. The first tested an already-chosen EA; the second used OOS to help choose it.
Out-of-sample (OOS) testing is useful, but the label cannot tell you how the result entered the development process. For an EA buyer, the important distinction is between a test that checks a decision already made and one that helps make that decision.
A separate period can become development data
Consider an EA built on 2014–2022 and assessed on 2023–2024. If the later result prompts a parameter change, a new filter or a different candidate, it has supplied information to the next version. Repeating the test on those dates does not restore the original separation.
The same applies to moving a pass threshold after seeing a near miss. The calendar boundary remains unchanged, but the acceptance decision is now partly shaped by the supposed check.
That does not make the later test worthless. It means its role should be described accurately: it contributed to selection or adjustment. A reader should not be asked to interpret it as confirmation of decisions that had already been fixed.
AI-generated illustration of the decision process, not a record of a product test.
Selecting a different candidate also uses the result
Suppose 100 candidates pass Development and are all tested on the same OOS interval. Keeping the one with the highest OOS PF changes no parameters in that EA. It still lets OOS decide which EA will be offered.
This is selection, not merely confirmation. With more candidates, there are more opportunities for one to fit the interval unusually well. The selected EA may have a real edge, but the winning OOS result alone does not resolve that uncertainty.
So “no optimization after OOS” needs a follow-up question: was the eventual EA already chosen, or was it chosen because of that OOS result? A fixed candidate list is helpful, but it does not remove selection effects if the final winner is ranked using the same period later presented as its independent check.
Freeze the decisions that the test is meant to assess
For a clearer confirmation stage, define the candidate or portfolio, pass/fail criteria, membership, weights and risk allocation before viewing the result. Then apply those criteria as written. A failure under a predefined rule is a legitimate outcome, not a reason to redesign the rule until the candidate passes.
That is different from choosing a new threshold, replacing components or changing allocations after inspecting OOS. Those changes may be useful research, but the resulting version has now benefited from the period. It needs to be described as such.
This distinction avoids a common confusion: an OOS test is supposed to inform an acceptance decision. What weakens its independence is using its details to improve the candidate, select the best of many outcomes, or revise the criteria after the fact, then presenting the result as though none of that occurred.
AI-generated illustration of the decision process, not a record of a product test.
Portfolio selection has its own data history
The issue does not end when the individual EAs stop changing. OOS can also determine which strategies enter a portfolio, how many are included and how much risk each receives. A portfolio assembled using that period has a different evidence history from a fixed portfolio that was first exposed to it afterward.
It is therefore worth asking about both levels: how the component strategies were selected and how the combination was selected. “The portfolio was frozen” is informative only when the description makes clear which decisions had stopped changing and which data had already been seen.
A more useful question than “Did it pass OOS?”
Ask what was fixed before OOS and what changed afterward. Those two answers reveal whether the result primarily confirmed a predefined candidate, helped choose one, or guided another round of development.
None of these distinctions guarantees future profitability. They do let you judge the evidence on the terms under which it was obtained, rather than granting extra confidence to a date range simply because it carries the OOS label.
Product details and testing conditions
When reviewing our products, distinguish the research history of individual strategies from the selection of the portfolio.
Specifications, published historical results and operating limits: EdgeDriven Gold Portfolio — XAUUSD · EdgeDriven Dollar Yen Portfolio — USDJPY.
The examples in this article explain evaluation methods; they are not test results for those products. Historical simulations do not guarantee future results. Leveraged trading can cause substantial losses.
