RegimeHMM: market regimes with a hidden Markov model - library for MetaTrader 5

RegimeHMM.mqh finds market regimes with a hidden Markov model: the returns (or any features) come from one of a few states, each with its own normal distribution, and the state follows a Markov chain. The class fits the model by Baum-Welch, gives the probability of each regime at every bar, the most likely path, the expected duration of each regime and the probabilities of the regimes in the coming periods. One or several features, full covariance, no DLL.

Usage

#include <RegimeHMM\RegimeHMM.mqh>

double returns[];                          // for example 100 * log(close[i] / close[i-1])
// ... fill returns ...
CGaussianHMM hmm;
if(hmm.SetData(returns) && hmm.Fit(2))     // 2 regimes, 10 starts; regime 0 is the calmer one
  {
   const int last = hmm.Observations() - 1;
   double pCalm = hmm.Filtered(last, 0);      // probability of the calm regime now, from the data up to now
   double vol1 = MathSqrt(hmm.Covariance(1)); // volatility in the other regime
   double days = hmm.ExpectedDuration(1);     // mean length of a stay in it
   vector next;
   hmm.Predict(5, next);                      // probabilities of the regimes in 5 periods
  }

For several features, pass a matrix with one row per bar and one column per feature to SetData.

Filtered or smoothed

Filtered(t, k) uses only the data up to bar t: it is what was known at that time, and the one to use in a backtest or an Expert Advisor. Posterior(t, k) uses all the data, before and after t: it draws cleaner regimes on a chart, but a test that trades on it sees the future. The same holds for the Viterbi path. The parameters themselves come from all the data: for a fair test, fit on the past and apply to the bars after it.

How it works

• Forward-backward with scaling (no underflow on long series), Baum-Welch updates of the start, transition, mean and covariance parameters; a small floor (1e-6 of the variance of the data) keeps the covariances positive.
• Starts by k-means++ with a seed on scaled features, several starts, the best likelihood kept. The states are then sorted by the variance of the first feature, so regime 0 is always the calmest.
• Viterbi path in logarithms, forecasts of the state by powers of the transition matrix, long-run shares of time, AIC and BIC to choose the number of regimes.

Example

The script RegimeExample.mq5 fits 1 to 4 regimes to the returns of a symbol, keeps the best by BIC, and prints for each regime the return and volatility a year, the mean duration and the share of time, then the transition matrix, the probability of each regime now and in 1, 5 and 20 periods (picture 1). It can write the regime of every bar to a CSV file; picture 2 shows US30 with the regime known on each day.

Checked

• On a model of 3 states, 2 features and 7 bars, every one of the 2 187 paths of states was enumerated: the likelihood, the smoothed and filtered probabilities, the Viterbi path and its probability, and the forecast were equal within 3e-15.
• From the same start, 30 Baum-Welch steps were compared with an independent Python version computed in logarithms, on 2 simulated and 3 market series with 2 and 3 states: log-likelihoods within 1e-12 and parameters within 3e-11 after 30 steps, and the likelihood never decreased.
• Every full fit ends at a fixed point of the algorithm: one more step adds less than 1e-5 to the log-likelihood. On simulated data with volatilities of 0.6 and 2.0 the fit found 0.595 and 2.009.

A fit of 2 000 bars with 10 starts takes 20 to 160 ms with 2 regimes, up to about 1.2 s with 3.

Files

RegimeHMM.mqh goes to MQL5\Include\RegimeHMM, RegimeExample.mq5 to MQL5\Scripts\RegimeHMM.