Discussing the article: "Unified Validation Pipeline Against Backtest Overfitting"

New comment
 

Check out the new article: Unified Validation Pipeline Against Backtest Overfitting.

This article explains why standard walkforward and k-fold CV inflate results on financial data, then shows how to fix it. V-in-V enforces strict data partitions and anchored walkforward across windows, CPCV purges and embargoes leakage while aggregating path-wise performance, and CSCV measures the Probability of Backtest Overfitting. Practitioners gain a coherent framework to assess regime robustness and selection reliability.

Every algorithmic trader eventually encounters a backtest that looks too good to be true. The equity curve is a near-perfect staircase climbing to the upper-right corner of the chart. The Sharpe ratio is exceptional. Drawdowns are shallow and brief.

The strategy then fails immediately upon going live.

This outcome is so common that it has earned its own vernacular in the quantitative research community. The culprit is almost always some form of overfitting: the algorithm has learned the historical noise of a specific dataset rather than any durable, forward-applicable market structure. What is less commonly understood is that overfitting is not a single phenomenon. It arrives through several distinct channels, each requiring a different countermeasure. A practitioner who deploys only one safeguard — the most common of which is a simple train/test split — remains exposed to the others.

This article examines three of the most rigorous tools available for combating overfitting in algorithmic strategy development: Validation-within-Validation (V-in-V), as articulated by Timothy Masters; Combinatorially Purged Cross-Validation (CPCV), developed by Marcos Lopez de Prado; and Combinatorially Symmetric Cross-Validation (CSCV), introduced by Bailey and Lopez de Prado. Each addresses a distinct failure mode. Together, they form a comprehensive defence against the most consequential forms of statistical self-deception in quantitative research.

Author: Patrick Murimi Njoroge

 
While the technical depth of this article is commendable—especially the focus on CPCV and Purging/Embargoing—there is a significant "Implementation Gap" that most retail traders will fall into. The core issue with a "Unified Validation Pipeline" implemented manually is that it introduces a new layer of Researcher Overfitting. When a trader is responsible for configuring the data partitions, purging windows, and embargo lengths, they often (unconsciously) tweak these parameters until the "validated" results look favorable. This is just overfitting at a higher level of abstraction. Furthermore, implementing these complex statistical methods directly in MQL5 is highly prone to error. A single mistake in the purging logic or a slight overlap in the anchored walk forward windows can lead to catastrophic data leakage, giving a false sense of security. In the institutional world, we are moving away from "manual pipelines" and toward Automated Testing Studios. The goal isn't just to have a pipeline; it's to have a standardized, battle-tested environment where the human element is removed from the validation process entirely. Without that automation and standardization, even the most "unified" pipeline is just another tool for p-hacking.
 

Oh, I'm late with my thoughts, Warren Giddings already expressed good points. ;-)

Specifically, I would like to mention that very important meta-optimization is left behind the scene in the article - namely the adjustment of in-sample window and forward step sizes. Because walkforward is not limited by rolling and anchored, but there is also the cluster walkforward optimization.

So all the methods described should be re-invoked "so to speak" in another perpendicular dimension of IS/OOS combinations of sizes and verified on test period.

 
Greate Job! Thank you!
 
Warren Giddings #:
While the technical depth of this article is commendable—especially the focus on CPCV and Purging/Embargoing—there is a significant "Implementation Gap" that most retail traders will fall into. The core issue with a "Unified Validation Pipeline" implemented manually is that it introduces a new layer of Researcher Overfitting. When a trader is responsible for configuring the data partitions, purging windows, and embargo lengths, they often (unconsciously) tweak these parameters until the "validated" results look favorable. This is just overfitting at a higher level of abstraction. Furthermore, implementing these complex statistical methods directly in MQL5 is highly prone to error. A single mistake in the purging logic or a slight overlap in the anchored walk forward windows can lead to catastrophic data leakage, giving a false sense of security. In the institutional world, we are moving away from "manual pipelines" and toward Automated Testing Studios. The goal isn't just to have a pipeline; it's to have a standardized, battle-tested environment where the human element is removed from the validation process entirely. Without that automation and standardization, even the most "unified" pipeline is just another tool for p-hacking.
Hello Warren,

Thank you for your feedback. I completely agree with all points you have raised, and the construction of such an automated pipeline is the point of my series 

MetaTrader 5 Machine Learning Blueprint. This article was meant to be an eye-opener for readers who have never given much thought to the kind of overfitting I used this article to address, which is why I actually made it separate from the other articles in the ML Blueprint series.

 
Stanislav Korotky #:

Oh, I'm late with my thoughts, Warren Giddings already expressed good points. ;-)

Specifically, I would like to mention that very important meta-optimization is left behind the scene in the article - namely the adjustment of in-sample window and forward step sizes. Because walkforward is not limited by rolling and anchored, but there is also the cluster walkforward optimization.

So all the methods described should be re-invoked "so to speak" in another perpendicular dimension of IS/OOS combinations of sizes and verified on test period.

Hello Stanislav,

Absolutely correct. As I was telling Warren, this article is meant to be an introduction to the concepts we can use to mitigate overfitting. They will be better addressed in my MetaTrader 5 Machine Learning Blueprint series.
 
Vasiliy Sokolov #:
Greate Job! Thank you!
You're welcome! Thank you for taking the time to read it.
New comment