MetaTrader 5 Machine Learning Blueprint (Part 16): Nested CV for Unbiased Evaluation

Table of Contents

1. The Three-Zone Partition
2. The Inner Loop: Walk-Forward Search with the 1-SE Rule
3. The Outer Loop: Evaluating the Selected Configuration
4. OOF Calibration Within the Nested Framework
5. Post-Loop Assembly: Consensus, Calibration, and Checkpoints
6. The Final Test Gate
7. Implementation: UnifiedValidationCalibrator
8. Practical Considerations
9. Conclusion
10. References
11. Attached Files

Introduction

A standard ML pipeline trains a model on one partition, tunes hyperparameters on a second, and evaluates on a third. In academic machine learning, this procedure works because the three partitions are drawn independently from a stationary distribution. Financial data violates both assumptions. The observations are not independent (triple-barrier labels overlap in time), and the distribution is not stationary (regimes shift). Every boundary between partitions is a potential information leak, and every leak inflates the performance estimate by some amount the practitioner cannot quantify after the fact.

The problem compounds across the pipeline's three decision points. The hyperparameter search selects the configuration that scores best on its validation folds. This is a form of validation-set overfitting. The calibrator learns a mapping from raw to calibrated probabilities using model predictions. If those predictions come from observations used for training, the calibration curve becomes optimistically biased. The final evaluation can be reopened after inspecting the result, turning the "test" set into a second tuning set. Each of these individually biases the estimate upward. Together they compound into a Sharpe ratio or accuracy figure that can be substantially more optimistic than the model's true out-of-sample performance.

A V-in-V (validation-in-validation) nested cross-validation design blocks all three leakage paths. The data is partitioned into three temporal zones, each protecting a different decision boundary. Within the largest zone, two concentric CV loops separate hyperparameter selection from performance estimation. A calibration layer sits between the loops, using out-of-fold predictions that neither the model nor the hyperparameter search has seen. The result is a performance estimate that is uncontaminated by any of the three decision processes it reports on.

This article develops the theory behind each layer and explains the necessity of each design decision. It also walks through the production implementation in UnifiedValidationCalibrator. The code integrates with the AFML pipeline components from earlier articles: PurgedWalkForwardCV from the Unified Validation Pipeline, CombinatorialPurgedCV from the same article, and the isotonic calibration machinery from Part 12.

1. The Three-Zone Partition

The outermost structural decision is the temporal split of the full dataset into three zones. This architecture comes from Masters (1993), who argued that a two-way train/test split is insufficient when the practitioner has the opportunity to iterate: seeing the test result and adjusting the model turns the test set into a second training set, invalidating the evaluation. Masters' solution is a three-zone partition where the final zone can be opened exactly once.

Figure 1. V-in-V three-zone data partition

• Outer Training (~60%): All nested CV operations, including hyperparameter search, model fitting, OOF prediction generation, and calibration map fitting, occur within this zone. The practitioner can iterate freely here without contaminating downstream evaluations.
• Inner Validation (~20%): A shortlisting checkpoint. After the nested CV loop completes and the final model is assembled, its predictions on this zone are scored. The practitioner inspects the result but must not retune. If the inner validation Brier score is substantially worse than the outer OOF scores, something has gone wrong (typically overfitting to the outer training folds or a regime shift at the boundary). The model is either accepted or rejected at this stage.
• Final Test (~20%): Opened exactly once. The DataPartition class enforces this with a _final_opened flag that raises RuntimeError on any subsequent call to open_final_test(). Once this result is seen, the evaluation is committed. Any model adjustment after this point invalidates the entire procedure.

The split is temporal: 60% outer training, 20% inner validation, and 20% final test (most recent). Randomization would violate temporal ordering, allowing the model to train on future data and test on past data. The partition_data function uses iloc throughout to preserve the DataFrame indices and datetime alignment, which is critical because PurgedWalkForwardCV validates that the t1 Series shares the same index as the feature matrix.

def partition_data(
    X: pd.DataFrame,
    y: pd.Series,
    t1: pd.Series,
    sample_weight: pd.Series,
    inner_val_pct: float = 0.20,
    final_test_pct: float = 0.20,
) -> DataPartition:
    n = len(X)
    outer_end = int(n * (1.0 - inner_val_pct - final_test_pct))
    val_end = int(n * (1.0 - final_test_pct))

    return DataPartition(
        X_outer=X.iloc[:outer_end],
        y_outer=y.iloc[:outer_end],
        sw_outer=sample_weight.iloc[:outer_end],
        t1_outer=t1.iloc[:outer_end],
        X_inner_val=X.iloc[outer_end:val_end],
        y_inner_val=y.iloc[outer_end:val_end],
        sw_inner_val=sample_weight.iloc[outer_end:val_end],
        X_final=X.iloc[val_end:],
        y_final=y.iloc[val_end:],
        sw_final=sample_weight.iloc[val_end:],
    )

2. The Inner Loop: Walk-Forward Search with the 1-SE Rule

Within the outer training zone, the first task is hyperparameter selection. The naive approach is to grid-search over parameter combinations using k-fold CV and pick the configuration with the highest mean score. This has two problems in financial data. First, k-fold CV violates temporal ordering. Second, picking the absolute best-scoring configuration overfits to the validation folds; the selected configuration is the one that happened to exploit the specific temporal arrangement of those folds most effectively.

The inner loop addresses both problems. It uses PurgedWalkForwardCV with an anchored expanding window, which respects temporal ordering and applies purging and embargo to prevent label leakage. For the selection rule, it implements Masters' 1-SE (one standard error) rule: among all configurations whose mean score is within one standard error of the best, it selects the simplest.

Figure 2. Masters' 1-SE rule for hyperparameter selection

• Blue line: Mean score of the best configuration across inner CV folds.
• Red dotted line: Threshold at one standard error below the best mean. All configurations above this line are statistically indistinguishable from the best, given the estimation noise.
• Green band: The set of configurations within 1 SE of the best. The teal marker shows the selected configuration: the simplest (first in ParameterGrid order) that falls within this band.

The logic is straightforward. The best-scoring configuration may have achieved its top rank by exploiting noise in the inner validation folds. A simpler configuration with a mean score within one standard error is statistically indistinguishable from the best but less likely to have overfit. By always preferring simplicity within the band, the rule acts as a regularizer on the model selection process itself.

# 1-SE rule implementation (from inner_cv_search)
best_entry = max(all_scores, key=lambda x: x['mean_score'])
threshold = best_entry['mean_score'] - best_entry['std_score']
within_1se = [s for s in all_scores if s['mean_score'] >= threshold]

# ParameterGrid is ordered; within_1se[0] is the simplest
best_params = within_1se[0]['params']

One caveat: the "simplest" designation relies on ParameterGrid producing configurations in a consistent order where simpler configurations come first. This holds if you order your parameter lists from simple to complex (e.g., n_estimators: [50, 200, 500]). If the lists are in arbitrary order, the first configuration within the band is not necessarily the simplest in any meaningful sense. A more robust approach would assign an explicit complexity score to each configuration and minimize it within the 1-SE band.

3. The Outer Loop: Evaluating the Selected Configuration

The inner loop selects hyperparameters. The outer loop evaluates those selections on data the inner loop never saw. This separation is the core mechanism of nested CV: the performance scores computed on the outer test folds are unbiased by the hyperparameter selection process because the inner loop had no access to the outer test data.

The outer loop supports two CV strategies, switched by the outer_cv_type parameter. walk-forward (PurgedWalkForwardCV) produces a sequence of expanding training sets with temporally ordered test sets; each fold's training set includes all data up to some cutoff, and the test set is the next temporal block. CPCV (CombinatorialPurgedCV) produces φ[N, k] combinatorial paths, providing a richer distribution of performance estimates at higher computational cost.

One integration detail is critical. CombinatorialPurgedCV.split() yields (train_idx, test_idx_list) where test_idx_list is a list of arrays (one per test fold), not a flat array. walk-forward CV yields (train_idx, test_idx) as flat arrays. The outer loop normalizes this by calling np.concatenate(test_idx_list) for CPCV. Missing this distinction causes index errors or silently incorrect fold assignments.

Figure 3. Nested cross-validation architecture

• Outer loop (blue dashed): Iterates over walk-forward or CPCV splits of the outer training zone. Each iteration produces a train/test partition.
• Inner loop (orange dashed): For each outer fold, runs the walk-forward grid search over the hyperparameter space, applies the 1-SE rule, generates OOF predictions for calibration, and fits the fold-level isotonic calibrator.
• Post-inner steps (within outer, below orange box): Refit the classifier on the full outer training fold with the selected parameters. Apply the fold-level calibrator to the outer test predictions. Score both raw and calibrated probabilities.
• Post-loop (green box): Aggregate across folds to determine consensus parameters, fit the final calibrator on all OOF predictions, and run the inner validation checkpoint.

4. OOF Calibration Within the Nested Framework

Calibration within nested CV requires careful attention to which predictions the calibrator is allowed to see. Part 12 established that calibrating on predictions the model was trained on produces an optimistically biased mapping. The same principle applies here, but with an additional constraint: the calibrator must also be independent of the hyperparameter selection process.

The solution is a second pass of walk-forward CV within each outer fold. After the inner search selects the best hyperparameters, _oof_for_fold() runs a fresh PurgedWalkForwardCV using those parameters. It clones the estimator for each inner split, fits it on the inner training set, and collects predictions on the inner test set. The result is a full-length OOF prediction array for the outer training fold. Each prediction is produced by a model that was not trained on that observation.

def _oof_for_fold(self, X_tr, y_tr, sw_tr, t1_tr, params):
    inner_cv = PurgedWalkForwardCV(
        n_splits=self.n_inner_splits,
        t1=t1_tr,
        pct_embargo=self.pct_embargo,
        expanding_window=True,
        min_train_size=self.min_train_size,
    )
    inner_oof = pd.Series(np.nan, index=X_tr.index)

    for in_tr, in_val in inner_cv.split(X_tr, y_tr):
        clf_oof = clone(self.estimator)
        clf_oof.set_params(**params)
        try:
            clf_oof.fit(
                X_tr.iloc[in_tr], y_tr.iloc[in_tr],
                sample_weight=sw_tr.iloc[in_tr].values,
            )
        except TypeError:
            clf_oof.fit(X_tr.iloc[in_tr], y_tr.iloc[in_tr])

        inner_oof.iloc[in_val] = clf_oof.predict_proba(
            X_tr.iloc[in_val]
        )[:, 1]

    return inner_oof

The isotonic calibrator is then fit on these OOF predictions against the true labels, weighted by the same sample weights used for model training. The fold-level calibrator transforms the raw probabilities on the outer test set. This ensures that the calibrated outer test scores reflect what the calibrator would produce on genuinely unseen data, not on data it participated in generating.

Figure 4. Data flow through one outer fold

• Steps 1–3 (left column): Inner search selects hyperparameters; inner OOF predictions are generated; the isotonic calibrator is fit on those predictions. All three steps operate exclusively on the outer training fold.
• Step 4: The classifier is refitted on the full outer training fold using the selected parameters. This model has seen all training data for this fold but has not seen the outer test data.
• Steps 5–6 (right column): The refitted model predicts on the outer test fold; the fold-level calibrator transforms those predictions; both raw and calibrated scores are recorded.

5. Post-Loop Assembly: Consensus, Calibration, and Checkpoints

After all outer folds complete, each fold has selected its own best hyperparameters via the 1-SE rule. These per-fold selections may agree or disagree. The pipeline resolves the disagreement by majority vote: it counts which parameter set was selected most frequently across folds and designates that as the consensus.

# Consensus: most frequently selected params across folds
param_strs = [str(sorted(p.items())) for p in best_params_per_fold]
consensus_str = Counter(param_strs).most_common(1)[0][0]
consensus_params = best_params_per_fold[param_strs.index(consensus_str)]

The consensus model is fitted on the full outer training zone. In parallel, a final isotonic calibrator is fitted on the full OOF prediction array collected across all outer folds. This calibrator has seen more data than any individual fold-level calibrator; its step function is therefore more stable and better estimated.

The inner validation checkpoint then applies the consensus model and the final calibrator to the inner validation zone. This is the practitioner's last chance to detect problems before opening the final test set. If the inner validation Brier score is substantially worse than the mean outer OOF Brier score, possible explanations include overfitting to the outer training folds, a regime shift at the 60/20 boundary, or insufficient data in the inner validation zone for stable scoring. The practitioner inspects this result, but the critical discipline is: do not retune. Any adjustment based on this result converts the inner validation zone into a second training set, collapsing the three-zone architecture into a two-zone one.

After the checkpoint, the final model is refitted on the combined outer training and inner validation zones (80% of the data), using the consensus parameters. This model is the one that will be deployed or evaluated on the final test set.

6. The Final Test Gate

The evaluate_final_test() method opens the final test zone exactly once. The DataPartition dataclass enforces this with a boolean flag:

def open_final_test(self):
    if self._final_opened:
        raise RuntimeError(
            "The Final Test Set has already been opened. "
            "Any further model adjustment invalidates the evaluation."
        )
    self._final_opened = True
    return self.X_final, self.y_final, self.sw_final

This is not a suggestion; it is an architectural constraint. The RuntimeError fires unconditionally on any second call, regardless of whether the practitioner intended to "just look" or to make a "small adjustment." The design makes the evaluation protocol machine-enforced rather than discipline-dependent. Masters (1993) argued that human discipline is insufficient because the temptation to peek is always present; a programmatic gate removes the temptation entirely.

The evaluation computes four metrics: raw and calibrated Brier score, and raw and calibrated log loss. It also returns the raw and calibrated probability arrays for downstream analysis (reliability diagrams, position size distributions, equity simulations). These arrays are the deployment probabilities. They come from a model trained on 80% of the data, calibrated on OOF predictions from the first 60%, and evaluated on a test set unseen by any pipeline component.

7. Implementation: UnifiedValidationCalibrator

The full pipeline is encapsulated in a single sklearn-compatible class. It implements fit(), predict(), predict_proba(), and evaluate_final_test(). The fit() method runs the entire nested CV procedure; after fitting, predict_proba() applies the consensus model and the final calibrator to new data.

The class inherits ClassifierMixin and BaseEstimator and satisfies the full sklearn constructor contract: all __init__ parameters are simple scalars or estimators, so clone(), get_params(), set_params(), and Pipeline threading all work without modification. Data-carrying arguments — t1, sample_weight, close_prices, and primary_sides — belong to fit(), not to __init__(). Passing a pd.Series as a constructor parameter would break clone()'s equality checks and is the pattern this design avoids. The class also validates outer_cv_type at construction time, so a typo such as 'WalkForward' raises a ValueError immediately rather than silently falling back to walk-forward behavior.

check_estimator() tests that call fit(X, y) without temporal metadata are skipped via __sklearn_tags__. The t1 argument cannot be synthesized from X and y alone; it encodes the label end-times that PurgedWalkForwardCV requires to apply purging and embargo correctly. Omitting it is not a degraded mode — it is an invalid state. All other sklearn contracts are satisfied.

Figure 5. UnifiedValidationCalibrator complete pipeline

• Top: partition_data() splits the full dataset into 60/20/20 zones.
• Blue dashed box: The outer loop iterates over walk-forward or CPCV splits. For each outer fold, the inner search selects hyperparameters, OOF predictions are generated, and the fold-level calibrator is fitted.
• Green boxes: Post-loop assembly — consensus parameters by majority vote, final calibrator on all OOF predictions.
• Yellow box: Inner validation checkpoint. Inspect, but do not retune.
• Red box: Final model refitted on Outer + Inner Val; evaluate_final_test() opens the final test zone exactly once.

Usage

from sklearn.ensemble import RandomForestClassifier
from afml.cross_validation.nested_cv import UnifiedValidationCalibrator

uvc = UnifiedValidationCalibrator(
    estimator=RandomForestClassifier(random_state=42),
    param_grid={
        'n_estimators': [100, 200, 500],
        'max_depth': [3, 5, 8],
    },
    outer_cv_type='walkforward',
    n_outer_splits=5,
    n_inner_splits=3,
    pct_embargo=0.01,
    min_train_size=0.1,
    scoring='neg_brier',
)

# Run the full nested CV pipeline
uvc.fit(X, y, t1=events['t1'], sample_weight=events['tW'])

# Inspect outer fold scores
print(uvc.outer_scores_summary())

# Open the final test set — EXACTLY ONCE
result = uvc.evaluate_final_test()
print(f"Final Brier (cal): {result['brier_cal']:.4f}")

# Inference on new data uses consensus model + final calibrator
probs = uvc.predict_proba(X_new)[:, 1]

CPCV Mode

When outer_cv_type='cpcv', the outer loop uses CombinatorialPurgedCV instead of walk-forward. This produces φ[N, k] combinatorial paths, each providing an independent out-of-sample performance estimate. After all outer folds complete, the pipeline runs CPCVAnalyzer to compute the path Sharpe distribution and PBO audit. CPCV mode requires two additional data arguments: close_prices (for mark-to-market Sharpe computation) and primary_sides (for meta-labelling direction signals). Both are passed to fit(), not to __init__(); this keeps the constructor free of data arrays and preserves clone() compatibility.

In CPCV, each observation can appear in multiple test paths. If you write predictions position-by-position, you get last-write-wins behavior: only the final path's prediction is retained. The correct approach is to accumulate all predictions per position across all test paths and average them before fitting the final calibrator. This is what the production implementation does via an accumulator dictionary, and the difference matters: the last-write approach makes the calibrator's training data dependent on the arbitrary order in which CPCV paths are iterated.

uvc_cpcv = UnifiedValidationCalibrator(
    estimator=RandomForestClassifier(random_state=42),
    param_grid={'n_estimators': [100, 200], 'max_depth': [3, 5]},
    outer_cv_type='cpcv',
    cpcv_n_folds=6,
    cpcv_n_test_folds=2,
    scoring='neg_brier',
)
# close_prices and primary_sides go to fit(), not __init__()
uvc_cpcv.fit(
    X, y,
    t1=events['t1'],
    sample_weight=events['tW'],
    close_prices=close_series,
    primary_sides=side_predictions,
)

# Path Sharpe distribution is available after fit
print(uvc_cpcv.cpcv_distribution_metrics_)

8. Practical Considerations

Computational Cost

Nested CV is expensive. For each outer fold, the inner loop runs a full grid search with walk-forward CV, then a second pass for OOF prediction generation. With 5 outer folds, 3 inner splits, and a grid of 9 parameter combinations, the total number of model fits is 5 × (9 × 3 + 3) = 150. For CPCV with N=6 and k=2, there are 15 outer splits, bringing the total to 15 × (9 × 3 + 3) = 450 fits. On financial datasets with thousands of observations and a Random Forest with 200 trees, each fit takes seconds; the full pipeline completes in minutes. On datasets with hundreds of thousands of observations or with gradient boosted trees, the cost can reach hours.

The inner OOF pass (_oof_for_fold) doubles the number of model fits beyond what the grid search alone requires. This cost is unavoidable: the calibrator needs predictions that neither the model nor the hyperparameter search has contaminated, and the only way to generate those predictions is to refit the model on each inner fold with the selected parameters and collect test-fold predictions.

Parameter Grid Ordering

The 1-SE rule's "select the simplest" behavior depends on ParameterGrid producing configurations in an order where simpler configurations come first. ParameterGrid iterates over the Cartesian product in the order the values are specified. To ensure the simplest configuration is first, order each parameter's value list from least complex to most complex. For tree-based models, this means fewer trees before more trees, shallower depths before deeper, and higher regularization before lower.

Minimum Data Requirements

Three data-size constraints interact. The outer training zone (60% of the data) must be large enough for the outer CV to produce meaningful splits. The inner splits within each outer fold must contain enough observations for the walk-forward training windows to train a reasonable model. The OOF predictions must number at least several hundred for the isotonic calibrator to produce a stable step function. As a rough guideline, the full dataset should contain at least 2,000 independent observations (after accounting for label concurrency reducing the effective sample size). Below this threshold, consider using Platt scaling instead of isotonic regression, reducing the number of inner splits, or widening the outer training zone at the expense of the inner validation and final test zones.

When Not to Use Nested CV

Nested CV is the correct procedure when the model will be deployed with the hyperparameters and calibrator it selects. It is unnecessary when hyperparameters are fixed externally (e.g., inherited from a published strategy or mandated by policy). In that case, a single-loop OOF calibration via CalibratorCV (Part 12) is sufficient. Nested CV is also unnecessary when the practitioner is conducting exploratory research and does not need unbiased performance estimates; the additional computational cost slows iteration without corresponding benefit during the exploration phase.

Sklearn Compatibility

UnifiedValidationCalibrator is designed as a top-level validation orchestrator, not as the base estimator inside another meta-estimator. Wrapping it in GridSearchCV or CalibratedClassifierCV is a conceptual error: it would nest one CV procedure inside another with no principled separation of what each layer evaluates.

Within that constraint, the class satisfies the full sklearn constructor contract. Three design rules enforce this. First, __init__ stores only scalars and estimators; all data arrays (t1, sample_weight, close_prices, primary_sides) are accepted by fit() only. Second, outer_cv_type is validated at construction time against the set {'walkforward', 'cpcv'}, so an invalid value raises ValueError immediately rather than producing a silent behavioral change deep inside fit(). Third, check_estimator() requires the mixin order ClassifierMixin, BaseEstimator (more specialized before more general). The consequence is that clone(), get_params(), set_params(), and Pipeline threading all work without modification. check_estimator() tests that call fit(X, y) without temporal metadata are skipped via __sklearn_tags__; the reason is documented in the class docstring.

Conclusion

The UnifiedValidationCalibrator integrates three layers of protection against information leakage. The three-zone partition prevents the practitioner from re-tuning after seeing test results. Nested cross-validation separates hyperparameter selection from performance estimation. OOF calibration within the inner loop ensures the probability mapping is learned from predictions that neither the model nor the hyperparameter search has contaminated.

Key takeaways:

• The three-zone partition is enforced programmatically. DataPartition.open_final_test() raises RuntimeError on any second call. This converts the Masters (1993) single-opening discipline from a convention into an architectural constraint.
• The 1-SE rule regularizes model selection. Among configurations within one standard error of the best, the simplest is selected. Order your param_grid value lists from least complex to most complex so that ParameterGrid places simpler configurations first.
• OOF calibration within nested CV requires a second inner pass. The _oof_for_fold method generates predictions that are independent of both the hyperparameter search and the model's training data. This is the correct — and the only correct — source of calibration data within the nested framework.
• Consensus parameters use majority vote. Each outer fold selects its own best configuration; the most frequently selected set becomes the consensus. If no configuration wins a majority, the pipeline still selects the most common, but the practitioner should interpret disagreement as a signal that the model is sensitive to the training window.
• The inner validation checkpoint is for inspection, not adjustment. If the inner validation score is materially worse than the outer OOF scores, the model should be rejected at this stage, not re-tuned. Adjusting based on the inner validation result converts it into a second training set.
• Nested CV belongs in Python, not MQL5. It is a model selection and validation procedure. Its output is a trained model, a calibrator, and a set of consensus hyperparameters. MQL5 consumes these artifacts; it does not need to reproduce the selection process.

References

1. López de Prado, M. (2018). Advances in Financial Machine Learning. Wiley. Chapters 4, 7, and 12.
2. Masters, T. (1993). Practical Neural Network Recipes in C++. Wiley. Chapter on validation-in-validation and the 1-SE rule.
3. Varma, S., & Simon, R. (2006). "Bias in error estimation when using cross-validation for model selection." BMC Bioinformatics, 7(1), 91.
4. Cawley, G. C., & Talbot, N. L. C. (2010). "On over-fitting in model selection and subsequent selection bias in performance evaluation." JMLR, 11, 2079–2107.

Attached Files

	File	Description
1.	nested_cv.py	DataPartition, partition_data, inner_cv_search, and UnifiedValidationCalibrator. Implements the V-in-V three-zone partition, anchored walk-forward inner search with the 1-SE rule, OOF isotonic calibration, and the single-open final test gate.
2.	cross_validation.py	Required dependency. Provides PurgedKFold and PurgedWalkForwardCV, which nested_cv.py uses for the inner hyperparameter search loop, the OOF prediction pass, and the outer walk-forward splits.
3.	combinatorial.py	Required dependency. Provides CombinatorialPurgedCV, CPCVAnalyzer, and optimal_folds_number, which nested_cv.py uses when outer_cv_type='cpcv' to generate the φ[N, k] path distribution and PBO audit.