Discussing the article: "MetaTrader 5 Machine Learning Blueprint (Part 26): Process Permutation"

[MetaQuotes]

Check out the new article: MetaTrader 5 Machine Learning Blueprint (Part 26): Process Permutation.

We introduce the permutation process for backtest validation: permute bars, rerun the complete development process, and compare the observed Sharpe to the null distribution it produces. Outputs include one-sided p-values, robust z, and quantile confidence intervals, plus a spurious performance curve that quantifies selection versus trial budget. A cache-isolation check prevents false passes. The EURUSD H1 case shows how to set a tolerance and assess whether performance is distinguishable from selection.

Run a hundred trials of a parameter search, keep the best one, and the backtest reports a positive Sharpe ratio. How much of that number came from the market, and how much from keeping the best of a hundred?

This article answers that question for one concrete search. A moving-average crossover is searched over 100 pairs of window lengths on EURUSD H1 data from 2018 through 2024, and the best pair reports an annualized Sharpe of 0.17. The bars are then reshuffled so that no bar carries information about the next, and the same search runs again, 500 times. On the reshuffled data, the search reports a median Sharpe of 0.25.

The real result is lower than what the search typically finds in noise: it sits at the 41st percentile of the reshuffled runs. Repeating the experiment with 20 different sets of parameter pairs does not rescue it. The real result lands between the 31st and 57th percentile each time, and no p-value falls below 0.43. The search found nothing that reshuffled bars could not also produce. Section 6 measures this in full; the sections before it build the tool that does the measuring and the safeguards it depends on.

Part 23 asked whether a feature carries information. Part 24 asked whether a backtest's risk figures survive resequencing. Part 25 asked whether a deployed strategy is still doing what it did in the backtest. None of the three addresses this article's question: how much apparent performance the full development process reports (feature generation, labeling, hyperparameter search, model fitting, and backtest) after inter-bar predictability has been removed from the input. This is a test of the process, not of any one strategy it outputs.

Author: Patrick Murimi Njoroge