What Is In-Sample and Out-Of-Sample and Why Is It So Important?

What Is In-Sample and Out-Of-Sample and Why Is It So Important?

18 November 2026, 09:18
Margarita Florencia Del Aguila
0
7

1. The Exam Example and How It Works in Trading

Imagine a teacher wants to know if a student has truly learned how to solve math problems or simply has a good memory.

Throughout the school year, the student practices with a workbook containing 100 problems with step-by-step solutions. If on the day of the final exam the teacher gives the student those exact same 100 problems from the workbook, the student will score 100%. However, that grade does not prove the student understands math — it only proves they remember the answers they already saw.

To find out the truth, the teacher keeps 20 brand-new problems locked away that the student has never seen and hands them out on exam day. The score on those 20 new questions is the only one that proves their real ability.

What does this mean when training a Trading Bot or an Artificial Intelligence?

When developing trading systems and Artificial Intelligence, we apply that exact same method by splitting market history into two parts separated by a cutoff date:

  • In-Sample (Within the Sample): This is the historical period we feed into the computer and the Artificial Intelligence so they can study the market and learn which patterns work. Because the system has access to everything that happened during this period, getting good results here is just like answering the 100 questions from the study workbook: it is necessary for learning, but on its own it does not guarantee future success.
  • The Freeze Day: This is the moment we stop the learning process. We do not feed the entire history to the AI; instead, we reserve the most recent portion untouched. On that cutoff date, training is closed and the AI's brain is locked so nothing else can be modified.
  • Out-Of-Sample / OOS (Outside the Sample): These are the "20 brand-new exam questions." We take the bot, with its brain already frozen, and let it trade on the historical period we kept under lock and key — data it never saw during training.
  • Why it is so important: This is the ultimate test. If the bot maintains the exact same win rate and stability in this unseen territory as it did during training, it proves that it has learned real market logic rather than relying on memorizing the past.

Why Is Achieving a Competitive Out-Of-Sample So Difficult, and What Variation Margins Are Considered Valid?

Most traders are unaware that producing a perfect chart of the past (In-Sample) takes only a few minutes, whereas achieving a competitive and stable Out-Of-Sample can require months of rigorous research.

1. Optimizing the past takes minutes; beating the future takes months

Any modern computer can test millions of combinations in the optimizer in just a few minutes until it finds a mathematical configuration that won in the past (In-Sample). However, in real quantitative development, more than 95% of models that look brilliant during training fail or stagnate the moment they enter the blind test (Out-Of-Sample).

2. The golden rule: In Out-Of-Sample, "fixing" is strictly forbidden

The strictest rule of Out-Of-Sample testing is that the developer cannot touch anything at all. If you freeze the Artificial Intelligence on a cutoff date, run the test over the following year, and see that it loses money in a specific month, going back to change a parameter to dodge that loss is forbidden. If you do that, you are no longer blind, you break statistical rigor, and you turn that period into In-Sample. If a model fails in Out-Of-Sample, it must be discarded completely and rebuilt from scratch using only old data.

3. The market changes: What variation margins are considered valid?

From one year to the next, interest rates, structural volatility, spreads, and reactions to macroeconomic news change. Because of this, when a system transitions from the training environment (In-Sample) to facing a brand-new market (Out-Of-Sample) across hundreds of trades, the quantitative industry applies these standard tolerance margins:

  • Win Rate Variation: It is considered completely valid and healthy for the win rate to vary within a range of ±2% to ±5% compared to its training phase. This is the natural adaptation to live market noise. However, a sharp drop greater than 10% – 15% is mathematical confirmation that the model memorized the past (Overfitting).
  • Profit Factor Variation: When facing new and unseen conditions, a decay or variation of up to 15% – 20% in Profit Factor compared to In-Sample is considered valid (meaning it retains at least 80% to 100% of its original efficiency). If, on the contrary, the Profit Factor remains identical or is even higher after hundreds of new trades in Out-Of-Sample, the statistical robustness of the model is maximal.

2. Real Multi-Year Backtest Benchmark: In-Sample vs. Blind Out-Of-Sample (2021 – 2026)

[INSERT REAL CHART HERE: real_backtest_is_vs_oos_en.png — then delete this yellow line]

In a systematic dual-pair ONNX benchmark, the Artificial Intelligence model was trained exclusively on historical data from January 2021 to September 30, 2025 (In-Sample).

On that exact day (September 30, 2025), the AI's brain was frozen and tested blind from October 1, 2025 to October 2026 (Out-Of-Sample):

  • Phase 1 — In-Sample (January 2021 – September 30, 2025): During its learning phase on 100% real ticks with real ECN commissions deducted, it executed 1,715 trades, achieving a 63.5% Win Rate, a 1.59 Profit Factor, and +62.10% return.
  • Phase 2 — Blind Out-Of-Sample (October 1, 2025 – October 2026): With its ONNX models frozen on September 30, 2025 and facing 100% unseen market data, it executed 312 new trades, maintaining a 60.3% Win Rate (within the natural ±5% band), a 1.33 Profit Factor (>83% efficiency retention), and adding +6.05% to reach +68.15% ($1,681.48).
  • Visual result: After crossing the dashed September 30, 2025 freeze line, the green Out-Of-Sample curve continues rising smoothly with the same structural stability as the blue In-Sample curve, confirming genuine predictive generalization on unseen data.

3. How Can I Know Whether Out-Of-Sample (OOS) Was Respected in an EA?

Honestly, just by looking at a normal backtest chart, you cannot know.

In a standard MetaTrader 5 screenshot, the entire curve is drawn in a single color, making it impossible to tell with the naked eye whether the developer trained the bot using 100% of the history up to yesterday (all In-Sample) or genuinely left an untouched period for a blind test (Out-Of-Sample).

However, there are two things you can do and evaluate:

1. Check whether it is highlighted and marked on the chart or in the description

Building a system that passes a blind test (Out-Of-Sample) with its statistics intact is the hardest technical challenge in algorithmic trading. Because of that, anyone who genuinely applies it will always explain it and mark the exact cutoff date. The mere fact that an EA's description or chart does not even mention or indicate which part is In-Sample and which part is Out-Of-Sample is already a major point to keep in mind.

2. Request or verify the Digital Fingerprint (Hash) of the ONNX models

When a bot is powered by Artificial Intelligence (ONNX), the trained brain is stored inside the program as a sealed file. The moment that file finishes training and is frozen, it automatically generates a unique digital fingerprint code (SHA-256):

  • EURUSD ONNX Model Code (model_eurusd_frozen.onnx): 9B2A0485BC1A82A49B2C8DEA223BAFCC56D7B07EAE60F4B3C2CAABBFDE3EE21F
  • GBPUSD ONNX Model Code (model_gbpusd_frozen.onnx): 39AABC53536746CA1327A7B54C945278B978B839118CF93A8E056B0641FF9592

What is this code and why does it prove the truth?

It works just like a notary's digital seal on a document. That code belongs solely and exclusively to the brain frozen on the cutoff date. If anyone tried to cheat by retraining the Artificial Intelligence with newer data (even if only a single decimal changed), the resulting code would change completely. Publicly displaying the fingerprint of both ONNX models guarantees that the AI's brain has not been retrained after its freeze date.

In summary: If you acquire an EA where it is demonstrated with mathematical proof (such as the ONNX model fingerprint) that the Out-Of-Sample (OOS) period was strictly respected, you ensure you are dealing with an Artificial Intelligence with true predictive capability rather than an algorithm that simply memorized the past.