Discussing the article: "MetaTrader 5 Machine Learning Blueprint (Part 16): Nested CV for Unbiased Evaluation"

[MetaQuotes]

Check out the new article: MetaTrader 5 Machine Learning Blueprint (Part 16): Nested CV for Unbiased Evaluation.

The article presents a V-in-V nested cross-validation pipeline for financial data that breaks leakage at three decision points: hyperparameter search, calibration, and final evaluation. A temporal three‑zone split isolates an inner walk‑forward search with the 1‑SE rule from an outer walk‑forward or CPCV evaluation, while OOF isotonic calibration is fitted independently. The resulting UnifiedValidationCalibrator delivers unbiased out‑of‑sample scores and well‑calibrated probabilities for deployment.

The outermost structural decision is the temporal split of the full dataset into three zones. This architecture comes from Masters (1993), who argued that a two-way train/test split is insufficient when the practitioner has the opportunity to iterate: seeing the test result and adjusting the model turns the test set into a second training set, invalidating the evaluation. Masters' solution is a three-zone partition where the final zone can be opened exactly once.

• Outer Training (~60%): All nested CV operations, including hyperparameter search, model fitting, OOF prediction generation, and calibration map fitting, occur within this zone. The practitioner can iterate freely here without contaminating downstream evaluations.
• Inner Validation (~20%): A shortlisting checkpoint. After the nested CV loop completes and the final model is assembled, its predictions on this zone are scored. The practitioner inspects the result but must not retune. If the inner validation Brier score is substantially worse than the outer OOF scores, something has gone wrong (typically overfitting to the outer training folds or a regime shift at the boundary). The model is either accepted or rejected at this stage.
• Final Test (~20%): Opened exactly once. The DataPartition class enforces this with a _final_opened flag that raises RuntimeError on any subsequent call to open_final_test(). Once this result is seen, the evaluation is committed. Any model adjustment after this point invalidates the entire procedure.

Author: Patrick Murimi Njoroge