Kalman: exact diffuse Kalman filter, local level and time-varying hedge ratio - libreria per MetaTrader 5

Kalman.mqh is a Kalman filter and smoother for linear Gaussian state space models with one observation per bar. It starts with the exact diffuse initialization of Durbin and Koopman, so no first state and no large "starting variance" has to be guessed, it accepts missing values, and it estimates the variances by maximum likelihood for three ready models: a local level (a level that drifts, seen with noise), a local linear trend, and a regression whose coefficients drift, for example the hedge ratio of a pair. No DLL.

Usage

#include <Kalman\Kalman.mqh>

vector y, x;                               // for example log prices of two symbols on the same bars
// ... fill y and x ...
matrix X(x.Size(), 1);
X.Col(x, 0);
CKalman k;
if(k.FitDynamicRegression(y, X, true))     // intercept and hedge ratio follow random walks
  {
   const int last = k.Observations() - 1;
   double beta = k.Filtered(last, 1);       // hedge ratio now, from the data up to now
   double sd = k.FilteredStd(last, 1);      // its standard deviation
   double z = k.Innovation(last) / MathSqrt(k.InnovationVariance(last));   // last forecast error in sd
  }

CKalman ll;
if(ll.FitLocalLevel(y))                    // y = level + noise
  {
   double now = ll.Filtered(ll.Observations() - 1, 0);
   vector mean, var;
   ll.Forecast(10, mean, var);
  }

Any other model y(t) = Z(t) a(t) + e(t), a(t+1) = T a(t) + w(t) can be set with SetModel(Z, T, Q, H), then Filter(y) and Smooth(). SetDynamicRegression() runs the regression with variances you choose instead of fitting them.

What it gives

• Filtered states (from the data up to each bar: what was known then, the one to use in a test) and smoothed states (from all the data), each with its standard deviation; one step forecast errors and their variances; forecasts of y with their variances.
• The log-likelihood (exact diffuse, as defined by Durbin and Koopman), AIC and BIC, the number of diffuse bars.
• Variances by maximum likelihood on their logarithms: a grid of 20 starts, BFGS from the 3 best, a Nelder-Mead polish. The likelihood of the regression can have two maxima, one of them with no observation noise; the grid finds the higher one.

Numerical care

• The diffuse period is ended by counting: each bar that removes a diffuse direction lowers its rank by one. A test on the size of the diffuse variance fails when two prices are almost equal.
• In the regression the regressors are centered inside, an exact change of coordinates of the state that the results are converted back from. With a raw log price of gold (always between 7.5 and 8.4) next to the intercept, the covariances of the state are badly conditioned: the standard deviations of the smoothed hedge ratio were 7% wrong without it, they are within 6e-6 of a 40-digit computation with it.
• The optimizer uses a separate likelihood routine on plain arrays: a fit of 2 000 bars takes 10 to 400 ms.

Example

The script KalmanHedge.mq5 fits the regression of one symbol on another (log prices by default), prints the fitted variances, the hedge ratio now and 20, 60, 250 and 500 bars ago as it was known then, with its standard deviation and its smoothed value, the z-scores of the last forecast errors (the signal of a Kalman pair trade) and the share of errors beyond 2 standard deviations (picture 1). It can write the hedge ratio of every bar to a CSV file (picture 2).

Checked

On six series (simulated local level, the same with 10% missing values, simulated local linear trend, the log price of gold, a simulated regression and silver on gold, H4), against the state space models of statsmodels 0.15 with exact diffuse initialization and against a 40-digit computation of the same recursions:

• the same number of diffuse bars; log-likelihoods and smoothed states within 2e-11 of the 40-digit values, smoothed standard deviations within 2e-9 (6e-6 on silver on gold). statsmodels differs from the 40-digit values by up to 7e-9 in the log-likelihood and 2e-2 in the standard deviations on the same data;
• maximum likelihood: the same variances and likelihood as statsmodels (within 3e-7) on every series.

Files

Kalman.mqh goes to MQL5\Include\Kalman, KalmanHedge.mq5 to MQL5\Scripts\Kalman.