DataSnooping: SPA test, Reality Check, StepM and model confidence set - library for MetaTrader 5

DataSnooping.mqh answers the question that follows every optimization: the best of many strategies (or of many parameter sets of one strategy) beat the benchmark, but was it better, or only the luckiest of the lot? It has Hansen's test for superior predictive ability (SPA), White's Reality Check, the Romano-Wolf StepM procedure that finds every strategy better than the benchmark, and the model confidence set of Hansen, Lunde and Nason. The bootstrap keeps the dependence of daily results (stationary bootstrap) and takes a seed. No DLL.

Usage

#include <DataSnooping\DataSnooping.mqh>

matrix returns;                            // one column per strategy (or parameter set), one row per day
vector benchmark;                          // returns of the benchmark on the same days (zeros: no position)
// ... fill returns and benchmark ...
CSuperiorAbility spa;
spa.SetData(returns, benchmark, true);     // true: higher is better
spa.Options(2000, 0.0, true, 1);           // draws, mean block (0: square root of the days), studentized, seed
if(spa.Compute())
  {
   double p = spa.PValue(1);               // consistent SPA p-value: is the best really better?
   int better[];
   int n = spa.StepM(0.05, better);        // all the strategies better than the benchmark at 5%
  }
CModelConfidenceSet mcs;
if(mcs.Compute(returns, true, 2000))
  {
   bool kept = mcs.Included(0, 0.1);       // is strategy 0 in the 90% set of the best?
  }

What is in it

• CSuperiorAbility: the lower, consistent and upper p-values of the SPA test, the Reality Check (the upper p-value without studentization), critical values, the strategies above them, and StepM. Studentized (Hansen 2005: the best t statistic, with the long-run variances of the stationary bootstrap) or not.
• CModelConfidenceSet: the p-value of every strategy and the set of the best at any size, with the 'R' statistic (pairs of strategies) or the 'max' statistic. The elimination keeps the largest pair of each draw in memory, so 200 strategies take 0.2 s.
• CStationaryBootstrap: indices of the stationary bootstrap of Politis and Romano with a seed, and the means of the resamples from running sums (a whole block at a time).

A note on arch

The tests follow the Python package arch, and give the same p-values as arch when the same bootstrap draws are used. arch's SPA class has a studentize option, but in version 8.0 its computation does not divide by the standard deviations, so it returns the test without studentization whatever the option. Here, Options(..., studentize = false, ...) gives the same numbers as arch, and studentize = true gives Hansen's statistic, where a strategy with a high mean but very noisy results counts less than a steadier one.

Example

The script SnoopingExample.mq5 builds 100 moving average rules on a symbol (long while the fast average is above the slow one, flat otherwise, 10 fast and 10 slow lengths), or reads the returns of your own strategies from a CSV file, and compares the p-value of the best rule tested alone with the SPA, Reality Check, StepM and model confidence set results (picture 1). On US30 D1 the best rule has a t statistic of 1.72: alone, a p-value of 0.043; as the best of 100, an SPA p-value of 0.19 (picture 2).

Picture 3: the t statistic of each rule, by fast and slow average. Neighbouring rules give similar numbers, because they are in the market on mostly the same days. The bootstrap of the SPA test and of the Reality Check keeps this dependence; a Bonferroni correction would ignore it and be far too strict.

Picture 4: the same best rule, SMA 5/160. Tested alone, its p-value is 0.043, below 5%. Taken as the best of the 100 rules that were tried, it is 0.244 (Reality Check) and 0.187 (SPA): nothing left to show that it beats staying out of the market.

Checked

With the bootstrap indices drawn by arch 8.0 (stationary bootstrap, 1 000 draws) on three data sets (50 strategies of which 3 have a real edge, 200 correlated parameter sets without edge, the daily results of 9 sample Expert Advisors):

• SPA without studentization: the lower, consistent and upper p-values identical to arch's, critical values within 2e-15, long-run variances identical, the same better strategies and the same StepM set;
• SPA studentized: p-values identical to an independent numpy version of Hansen's test;
• model confidence set, 'R' and 'max': the p-value of every strategy identical to arch's.

Files

DataSnooping.mqh goes to MQL5\Include\DataSnooping, SnoopingExample.mq5 to MQL5\Scripts\DataSnooping.