Offline Reinforcement Learning in MQL5: Conservative Q-Learning from Historical Trade Logs Without Live Exploration

Introduction

Most reinforcement learning approaches to trading assume the agent can interact with the market during training — it tries an action, observes a reward, and updates. That's reasonable in a simulator; dangerous on a live account, since "trying an action" means a real order with real slippage while the policy is still bad at its job. Training offline removes that risk, but not the risk of deploying a bad policy once training is done: a frozen policy can still trade badly, it just can't make things worse by continuing to experiment.

This article works through a different setup: training a trading policy entirely from a fixed batch of historical transitions, with no further market interaction during training. The dataset is whatever a rule-based strategy, a manual trader, or a prior EA already logged — states, the action taken, the reward realized, and the state that followed. The policy learns from that log alone, then deploys frozen: no exploration, no live trial-and-error, no chance for learning itself to cause money loss.

Offline training also matters for auditability: a fixed dataset can be scrutinized and replayed before any order is placed, with every transition inspectable in a spreadsheet — worth the trade for a desk that has to explain its systems to a risk committee, even though an online agent's exploration policy keeps changing. This article doesn't demonstrate formal dataset versioning, a seed/parameter log, or an explicit train/test split, though — call it replayable rather than fully audited.

The catch: naive Q-learning on a fixed dataset doesn't work well out of the box. If the logged data never tried going long during a certain volatility regime, a standard Q-network will still happily assign that action a high value, because nothing in the data ever pushed the estimate back down. The bootstrapped Bellman update propagates these overestimates to state-action pairs never observed, and at deployment the policy may exploit them — extrapolation error, the central problem offline RL has to solve.

Conservative Q-Learning (CQL) adds a penalty term that pushes down the Q-values of actions the current policy would pick broadly, while leaving room for actions actually present in the data to fit the true return — the network stays conservative where evidence is thin, trusting its own estimates only where data backs them up. CQL's literature frames this as a lower bound on the true value. With a single network and 838 transitions, treat "conservative" as accurate and "lower bound" as motivating theory, not guarantee.

Scope note: this article implements CQL natively in MQL5, with no ONNX model and no external ML framework — the training script, network, and deployment EA are all plain .mq5/.mqh files. A Python companion script only converts MQL5's own CSV exports into the transition format the training script reads; it never touches the model itself.

Contents

1. Why offline RL needs a conservative correction
2. The CQL objective, in the form the code actually computes
3. Building the offline dataset from MQL5 trade logs
4. The native Q-network: forward pass, loss, and backprop
5. Deploying a frozen policy: the EA side
6. Edge cases and pitfalls
7. Testing in the Strategy Tester
8. Conclusion

Why offline RL needs a conservative correction

Picture a dataset from a trend-following strategy that almost never shorted during low-volatility mornings, since its own rules kept it flat then. Standard Q-learning still computes a Q-value for "short" in that regime — the network must output something for every action, even ones barely touched — and because training bootstraps from the reward plus the best next action's discounted value, an accidental overestimate there gets treated as ground truth, propagating backward through every earlier state that could reach it. Nothing proves it wrong offline.

Q-learning's update target is r + gamma * max_a' Q(s', a') — the network bootstraps from its own estimate of the best next action, not a true observed outcome. Online, a wrong estimate gets corrected next episode, once the agent tries it and sees the real reward. Offline there is no next episode: the overestimate becomes the target for the prior state's update too, compounding backward through the trajectory until a small overestimate becomes a policy confidently preferring an action nobody ever demonstrated.

Fixes split roughly into two camps: constrain the policy to stay close to what the data demonstrated (behavior-cloning-regularized methods, or support-matching approaches like BCQ), or constrain the value function to stay pessimistic on anything under-supported (CQL and related methods). This article uses the second family because it's simpler with a hand-written backprop loop — the correction adds straight into the loss, no second network needed, keeping the script under two hundred lines.

The architecture in this article is deliberately linear so the mechanism stays visible in the code rather than getting buried in framework abstractions:

Offline CQL pipeline, end to end

Fig. 1. The full offline pipeline: trade log and bar history become a transition dataset, the dataset trains a Q-network entirely offline, and only the resulting frozen weights ever reach the deployment EA.

Two things are worth noticing. First, no arrow feeds live ticks or Tester activity back into the training script — CQL_OfflineTrain.mq5 is a Script, not an EA or indicator, and runs once, off-chart, against a static file. Second, the weights file is the only thing crossing from training to deployment: the EA never sees the raw dataset, only calling Forward() and SelectAction() on weights fixed before it started.

The CQL objective, in the form the code actually computes

The standard temporal-difference loss for a discrete-action Q-network is nothing unusual: fit Q(s,a) for the logged action toward the observed reward plus the discounted value of the best next action. What CQL adds is a second term, computed from the network's current output for every action at state s, not just the logged one:

L_TD(θ)    = 0.5 * ( Q_θ(s,a) - y )^2
y          = r + gamma * max_{a'} Q_θ(s', a') * (1 - done)

L_CQL(θ)   = alpha * ( logsumexp_{a} Q_θ(s,a) - Q_θ(s,a_data) )

L_total(θ) = L_TD(θ) + L_CQL(θ)

The logsumexp_a term is a smooth stand-in for "the value the current policy would extract by picking greedily" — dominated by whichever action has the highest Q-value, softened by the others in proportion to closeness. Subtracting Q(s,a_data) makes the penalty small when the logged action already dominates, large when an undemonstrated action is winning. alpha controls the strength: too small and extrapolation error creeps back, too large and the network grows too pessimistic to prefer any action.

Differentiating the CQL term with respect to each output Q(s,a) gives a clean, cheap gradient: exactly softmax(Q(s,.))[a], with an extra -1 subtracted at the logged action. That's the whole reason this is tractable to implement by hand in MQL5 — no behavior-cloning network, no importance sampling over actions, just a softmax over three numbers the forward pass already computed. The full TrainStep() implementation below computes this directly:

//+------------------------------------------------------------------+
//| CQLQNetwork::TrainStep                                           |
//+------------------------------------------------------------------+
double CQLQNetwork::TrainStep(const CQLTransition &batch[], const int batch_size,
                               const double alpha, const double gamma, const double lr)
  {
//--- gradient accumulators, zeroed once per mini-batch (batch SGD).
//--- as in the constructor, fixed-size 2D arrays cannot be zeroed a row at
//--- a time via ArrayInitialize(), so each element is set explicitly
   double gW1[CQL_STATE_FEATURES][CQL_HIDDEN_UNITS];
   double gb1[CQL_HIDDEN_UNITS];
   double gW2[CQL_HIDDEN_UNITS][CQL_ACTION_COUNT];
   double gb2[CQL_ACTION_COUNT];
   for(int i = 0; i < CQL_STATE_FEATURES; i++)
      for(int j = 0; j < CQL_HIDDEN_UNITS; j++)
         gW1[i][j] = 0.0;
   for(int j = 0; j < CQL_HIDDEN_UNITS; j++)
      for(int a = 0; a < CQL_ACTION_COUNT; a++)
         gW2[j][a] = 0.0;
   ArrayInitialize(gb1, 0.0);
   ArrayInitialize(gb2, 0.0);

   double total_loss = 0.0;
   double q_s[], q_next[];

   for(int n = 0; n < batch_size; n++)
     {
      //--- forward pass on s; m_pre/m_post are cached as a side effect and
      //--- reused below for backprop through this same sample
      Forward(batch[n].state, q_s);
      double hidden_cache[CQL_HIDDEN_UNITS];
      double pre_cache[CQL_HIDDEN_UNITS];
      ArrayCopy(hidden_cache, m_post);
      //--- pre_cache must be copied now, before the second Forward() call
      //--- below overwrites m_pre with next_state's pre-activations
      ArrayCopy(pre_cache, m_pre);

      //--- bootstrapped TD target: r + gamma * max_a' Q(s',a') * (1-done).
      //--- This is semi-gradient (no gradient flows through the target network,
      //--- which here is just the same weights evaluated at s') - standard
      //--- practice to keep training stable
      Forward(batch[n].next_state, q_next);
      double max_next = q_next[0];
      for(int a = 1; a < CQL_ACTION_COUNT; a++)
         if(q_next[a] > max_next)
            max_next = q_next[a];
      double target = batch[n].reward + (batch[n].done ? 0.0 : gamma * max_next);

      int    a_logged = batch[n].action;
      double td_error  = q_s[a_logged] - target;

      //--- CQL conservative term: softmax over the network's own current
      //--- Q(s,.) approximates E_{a~mu}[Q(s,a)] via log-sum-exp; its gradient
      //--- w.r.t. q_s[a] is softmax(q_s)[a], minus 1 at the logged action.
      //--- This pushes down Q for actions the data never demonstrated while
      //--- leaving the logged action's Q free to fit the TD target
      double max_q = q_s[0];
      for(int a = 1; a < CQL_ACTION_COUNT; a++)
         if(q_s[a] > max_q)
            max_q = q_s[a];
      double exp_sum = 0.0, softmax[CQL_ACTION_COUNT];
      for(int a = 0; a < CQL_ACTION_COUNT; a++)
        {
         softmax[a] = MathExp(q_s[a] - max_q); // subtract max_q for numerical stability
         exp_sum   += softmax[a];
        }
      for(int a = 0; a < CQL_ACTION_COUNT; a++)
         softmax[a] /= exp_sum;

      double log_sum_exp = max_q + MathLog(exp_sum);
      total_loss += 0.5 * td_error * td_error + alpha * (log_sum_exp - q_s[a_logged]);

      //--- combine TD gradient (only at the logged action) with the CQL
      //--- penalty gradient (spread across every action via the softmax)
      double dLoss_dq[CQL_ACTION_COUNT];
      for(int a = 0; a < CQL_ACTION_COUNT; a++)
         dLoss_dq[a] = alpha * softmax[a];
      dLoss_dq[a_logged] += td_error - alpha; // TD term + CQL's -1 at the logged action

      //--- backprop through the linear output layer: dL/dW2, dL/db2
      double dHidden[CQL_HIDDEN_UNITS];
      ArrayInitialize(dHidden, 0.0);
      for(int a = 0; a < CQL_ACTION_COUNT; a++)
        {
         gb2[a] += dLoss_dq[a];
         for(int j = 0; j < CQL_HIDDEN_UNITS; j++)
           {
            gW2[j][a] += dLoss_dq[a] * hidden_cache[j];
            dHidden[j] += dLoss_dq[a] * m_W2[j][a]; // accumulate for layer 1
           }
        }

      //--- backprop through ReLU: gradient is zero wherever the pre-activation
      //--- was <= 0 (the unit did not fire, so it contributed nothing forward)
      for(int j = 0; j < CQL_HIDDEN_UNITS; j++)
        {
         //--- pre_cache (this sample's s), not m_pre - by this point m_pre
         //--- holds next_state's pre-activations from the Forward() call above
         double relu_grad = (pre_cache[j] > 0.0) ? dHidden[j] : 0.0;
         gb1[j] += relu_grad;
         for(int i = 0; i < CQL_STATE_FEATURES; i++)
            gW1[i][j] += relu_grad * batch[n].state[i];
        }
     }

//--- apply averaged SGD update: mean gradient over the mini-batch keeps
//--- the effective learning rate independent of batch_size
   double inv_n = 1.0 / batch_size;
   for(int i = 0; i < CQL_STATE_FEATURES; i++)
      for(int j = 0; j < CQL_HIDDEN_UNITS; j++)
         m_W1[i][j] -= lr * gW1[i][j] * inv_n;
   for(int j = 0; j < CQL_HIDDEN_UNITS; j++)
      m_b1[j] -= lr * gb1[j] * inv_n;
   for(int j = 0; j < CQL_HIDDEN_UNITS; j++)
      for(int a = 0; a < CQL_ACTION_COUNT; a++)
         m_W2[j][a] -= lr * gW2[j][a] * inv_n;
   for(int a = 0; a < CQL_ACTION_COUNT; a++)
      m_b2[a] -= lr * gb2[a] * inv_n;

   return(total_loss * inv_n);
  }

Reading it action by action: the TD half only touches the logged action's Q-value, exactly like ordinary Q-learning; the CQL half touches every action through the softmax, so dLoss_dq[a_logged] gets both the TD error and the CQL correction while every other action only gets the CQL push. From there it's standard backprop through a ReLU hidden layer, zeroed wherever pre_cache[j] <= 0. pre_cache matters because Forward() runs twice per sample (state, then next_state) and both overwrite the same m_pre/m_post buffers — reading m_pre afterward would mask the gradient with next_state's pre-activations; — pre_cache, copied right after the first call, keeps it correct.

One detail easy to miss: the CQL penalty here is computed against the network's own current Q-values at s, not a separate target network — unlike published CQL variants that keep one for the Bellman half, similar to DQN's trick. This keeps one weights file and no live/target synchronization, trading a little stability for simplicity; if a future dataset proves unstable, a target network is the first thing to add back, and the class is structured so a second, periodically-copied instance could be introduced without touching Forward().

Training two networks on the same 838-transition dataset — one with alpha=0 (CQL disabled), one with alpha=1 (this article's default) — and comparing their Q-values where the rare action was never logged turns up an extreme case of CQL's failure mode: FLAT never appears once across all 838 transitions, since a trade log only records opens, making "do nothing" about as pure a case of an under-visited action as exists. This measures how far the CQL penalty moves a Q-value, not decision quality — a real improvement still needs evaluation on outcomes:

Naive Q-learning vs. CQL, measured on the actual dataset

Fig. 2. Mean Q-value for FLAT (0 of 838 transitions logged) and LONG (523 of 838 logged), averaged over the 315 states, drawn from the same training dataset, where the rare action was not what actually happened, comparing a network trained with alpha=0 against this article's alpha=1. FLAT's naive estimate sits near zero simply from having no gradient signal at all; CQL's penalty drives it sharply negative instead, actively discouraging an action the data gives no basis to trust. LONG, which is well-represented, also drops under CQL but by proportionally less, since its estimate is still anchored by real TD targets on every epoch.

The comparison in Fig. 2 was produced by CQL_CompareAlpha.mq5, a small standalone diagnostic script that is not part of the deployed system — it exists purely to measure what the CQL penalty is doing on this specific dataset, by loading two independently-trained networks and evaluating both on the same states:

//+------------------------------------------------------------------+
//| OnStart                                                          |
//+------------------------------------------------------------------+
void OnStart(void)
  {
   if(!g_net_alpha0.LoadWeights(InpWeightsAlpha0))
     {
      Print("CQL_CompareAlpha: failed to load ", InpWeightsAlpha0);
      return;
     }
   if(!g_net_alpha1.LoadWeights(InpWeightsAlpha1))
     {
      Print("CQL_CompareAlpha: failed to load ", InpWeightsAlpha1);
      return;
     }

   int loaded = LoadDatasetForComparison(InpDatasetFile, g_dataset);
   if(loaded <= 0)
     {
      Print("CQL_CompareAlpha: could not load dataset");
      return;
     }

//--- count how often each action appears in the logged data - the action
//--- with the fewest examples is the one naive Q-learning is most likely
//--- to overestimate, and the one CQL's penalty should visibly pull down
   int counts[CQL_ACTION_COUNT];
   ArrayInitialize(counts, 0);
   for(int i = 0; i < loaded; i++)
      counts[g_dataset[i].action]++;

   int rare_action = 0;
   for(int a = 1; a < CQL_ACTION_COUNT; a++)
      if(counts[a] < counts[rare_action])
         rare_action = a;

   PrintFormat("CQL_CompareAlpha: action counts - short=%d flat=%d long=%d (rarest = %d)",
               counts[CQL_ACTION_SHORT], counts[CQL_ACTION_FLAT], counts[CQL_ACTION_LONG], rare_action);

//--- average both networks' Q-value for the rare action, and separately for
//--- the most-logged action, ONLY over states where the rare action was NOT
//--- what actually happened - this is exactly the extrapolation scenario
//--- CQL is meant to guard against: judging an action the data barely tried
   int common_action = 0;
   for(int a = 1; a < CQL_ACTION_COUNT; a++)
      if(counts[a] > counts[common_action])
         common_action = a;

   double sum_rare0 = 0, sum_rare1 = 0, sum_common0 = 0, sum_common1 = 0;
   int n_eval = 0;
   double q0[], q1[];
   for(int i = 0; i < loaded; i++)
     {
      if(g_dataset[i].action == rare_action)
         continue; // only evaluate on states where this action wasn't logged
      g_net_alpha0.Forward(g_dataset[i].state, q0);
      g_net_alpha1.Forward(g_dataset[i].state, q1);
      sum_rare0   += q0[rare_action];
      sum_rare1   += q1[rare_action];
      sum_common0 += q0[common_action];
      sum_common1 += q1[common_action];
      n_eval++;
     }

   PrintFormat("CQL_CompareAlpha: over %d states (rare action=%d NOT logged):", n_eval, rare_action);
   PrintFormat("  rare action %d   -> alpha=0: %.4f   alpha=1: %.4f", rare_action, sum_rare0/n_eval, sum_rare1/n_eval);
   PrintFormat("  common action %d -> alpha=0: %.4f   alpha=1: %.4f", common_action, sum_common0/n_eval, sum_common1/n_eval);
  }

The script counts how often each action appears in the logged data, identifies the rarest and most common, then averages both networks' Q-value estimate for each — restricted to states where the rare action was not what actually happened, since that's precisely the extrapolation scenario CQL guards against. Running it is a one-time diagnostic: train a second set of weights with alpha=0 under a different filename, run this script once, then discard that alpha=0 file — it never deploys.

Building the offline dataset from MQL5 trade logs

The dataset is nothing more exotic than rows of (state, action, reward, next_state, done), but getting there from a real trade log takes care. Export_TradeLog.mq5 reconstructs trades into trade_log.csv, and Export_BarHistory.mq5 exports matching OHLC/ATR/RSI into bar_history.csv — both UTF-16, with a regional delimiter and a stray BOM. build_offline_dataset.py handles all three and walks both exports together to reconstruct the same feature vector the EA computes live, so training and deployment never see two definitions of "state":

def read_mql5_csv(path: str) -> pd.DataFrame:
    df = pd.read_csv(path, encoding="utf-16", sep=None, engine="python")
    df.columns = df.columns.str.replace("", "", regex=False)
    return df

On the MQL5 side, LoadDataset() reads back the plain UTF-8 CSV the Python script produces (deliberately not FILE_CSV's UTF-16, since training never round-trips through MetaTrader's export path). One wrinkle: it opens with FILE_ANSI, technically a different mode from UTF-8 — harmless here since every column is plain ASCII, but unsafe the moment a non-ASCII value enters one. It also grows its buffer geometrically instead of reading the file twice (once to count, once to load), which matters once a trade log spans years:

//+------------------------------------------------------------------+
//| LoadDataset                                                      |
//+------------------------------------------------------------------+
int LoadDataset(const string filename, CQLTransition &dataset[])
  {
//--- offline_dataset.csv is written by build_offline_dataset.py using
//--- pandas' default UTF-8 (not FILE_CSV's native UTF-16), so this reads
//--- as a plain UTF-8 text file with a one-line header to skip
   int handle = FileOpen(filename, FILE_READ | FILE_CSV | FILE_ANSI | FILE_COMMON, ',');
   if(handle == INVALID_HANDLE)
     {
      Print("LoadDataset: cannot open ", filename, ", err=", GetLastError());
      return(-1);
     }

//--- header row: state_0..state_8,action,reward,next_state_0..next_state_8,done
   for(int c = 0; c < 2 * CQL_STATE_FEATURES + 3; c++)
      FileReadString(handle);

   int count = 0;
   ArrayResize(dataset, 4096); // grown geometrically below as needed
   while(!FileIsEnding(handle))
     {
      CQLTransition row;
      for(int i = 0; i < CQL_STATE_FEATURES; i++)
         row.state[i] = StringToDouble(FileReadString(handle));
      row.action = (int)StringToInteger(FileReadString(handle));
      row.reward = StringToDouble(FileReadString(handle));
      for(int i = 0; i < CQL_STATE_FEATURES; i++)
         row.next_state[i] = StringToDouble(FileReadString(handle));
      row.done = (StringToInteger(FileReadString(handle)) != 0);

      //--- all fields for this row are read regardless, so the file cursor
      //--- stays aligned; only a malformed or out-of-range action is then
      //--- dropped, rather than silently indexing Q-value arrays with it
      if(row.action < 0 || row.action >= CQL_ACTION_COUNT)
         continue;

      if(count >= ArraySize(dataset))
         ArrayResize(dataset, ArraySize(dataset) * 2);
      dataset[count++] = row;
     }
   FileClose(handle);
   ArrayResize(dataset, count);
   return(count);
  }

Two design choices in build_offline_dataset.py deserve a word each. First, reward is taken directly from the trade log's realized profit column, with no shaping applied — a Sharpe-like or drawdown-penalized reward would encourage smoother equity curves, but changes what the Bellman target represents, and the CQL penalty's job only makes sense relative to whatever reward definition produced the data in the first place. Changing it later means rebuilding the dataset, not just retraining.

Worth flagging: the trade log and the deployment EA don't quite speak the same language. The log records completed trades — opened, closed — while deployment asks for a target exposure on every closed bar, so the dataset samples only the moments the logging strategy acted while the EA evaluates every bar. Reward attaches to a closed trade's P/L while the action in that row is the opening decision, making the chain a reasonable proxy for a bar-by-bar MDP, not an exact one.

The second is the is_last_of_session flag feeding the done column. Marking a transition terminal tells TrainStep() to drop the bootstrapped continuation term entirely (batch[n].done? 0.0: gamma * max_next), which matters at genuine boundaries — overnight gaps, a strategy change, an end-of-week flattening. treating every transition as non-terminal would inject a continuity assumption the market never promised.

The training loop, in CQL_OfflineTrain.mq5's OnStart(), is a script that never opens a chart connection to the live market. It loads the dataset once, computes the feature statistics the OOD guard will need later, shuffles before every epoch (otherwise a chronologically-ordered dataset biases each mini-batch toward whichever regime was active then), and calls TrainStep() per mini-batch:

//+------------------------------------------------------------------+
//| OnStart                                                          |
//+------------------------------------------------------------------+
void OnStart(void)
  {
//--- Step 1: load the fixed offline dataset - this script never places
//--- a trade or reads a live tick; every transition below comes from a
//--- CSV a strategy or trade log already produced
   int loaded = LoadDataset(InpDatasetFile, g_dataset);
   if(loaded <= 0)
     {
      Print("CQL_OfflineTrain: could not load dataset, aborting");
      return;
     }
   Print("CQL_OfflineTrain: loaded ", loaded, " transitions from ", InpDatasetFile);

//--- Step 2: compute per-feature mean/stdev up front so the OOD guard
//--- ships baked into the same weights file as the policy it protects
   ComputeFeatureStats(g_dataset, loaded, g_stats);
   g_network.SetFeatureStats(g_stats);

//--- Step 3: train
   g_network.Init(InpRandomSeed);
   MathSrand(InpRandomSeed);
   for(int epoch = 0; epoch < InpEpochs; epoch++)
     {
      ShuffleDataset(g_dataset, loaded); // i.i.d. mini-batches, not chronological blocks
      double epoch_loss = 0.0;
      int    batches     = 0;
      //--- rows left over once loaded is not an exact multiple of InpBatchSize
      //--- are simply not visited this epoch, rather than padded into a short
      //--- final batch - a small, known gap in how much of the 838 rows each
      //--- epoch actually uses
      for(int start = 0; start + InpBatchSize <= loaded; start += InpBatchSize)
        {
         CQLTransition mini_batch[];
         ArrayResize(mini_batch, InpBatchSize);
         for(int k = 0; k < InpBatchSize; k++)
            mini_batch[k] = g_dataset[start + k];
         epoch_loss += g_network.TrainStep(mini_batch, InpBatchSize, InpAlpha, InpGamma, InpLearningRate);
         batches++;
        }
      if(batches > 0 && (epoch % 20 == 0 || epoch == InpEpochs - 1))
         PrintFormat("CQL_OfflineTrain: epoch %d/%d  mean loss = %.6f", epoch + 1, InpEpochs,
                     epoch_loss / batches);
     }

//--- Step 4: persist - the deployment EA never trains, it only loads this file
   if(g_network.SaveWeights(InpWeightsFile))
      Print("CQL_OfflineTrain: weights saved to Common\\Files\\", InpWeightsFile);
   else
      Print("CQL_OfflineTrain: failed to save weights");
  }

Notice the feature statistics are computed before training starts and saved into the same weights file as the network — so the OOD guard and the policy it protects can never drift apart.

The native Q-network: forward pass, loss, and backprop

The shared type definitions live in CQLDefs.mqh: a small state width, three discrete actions, and the two structs that carry a transition and a feature-statistics summary between files:

//+------------------------------------------------------------------+
//| Discrete actions                                                 |
//+------------------------------------------------------------------+
enum ENUM_CQL_ACTION
  {
   CQL_ACTION_SHORT = 0, // reduce / open short exposure
   CQL_ACTION_FLAT  = 1, // flatten to zero exposure
   CQL_ACTION_LONG  = 2  // increase / open long exposure
  };

//+------------------------------------------------------------------+
//| One offline transition row                                       |
//+------------------------------------------------------------------+
struct CQLTransition
  {
   double state[CQL_STATE_FEATURES];      // s  : engineered feature vector
   int    action;                         // a  : ENUM_CQL_ACTION taken by the logging strategy
   double reward;                         // r  : realized reward for the transition
   double next_state[CQL_STATE_FEATURES]; // s' : feature vector after the transition
   bool   done;                           // terminal flag (position closed / session end)
  };

Twenty-four hidden units is a deliberately small choice, reached by starting smaller and increasing only until training loss stopped improving. A wider network isn't automatically better offline the way it might be online, where enough interaction fills capacity's blind spots — every added parameter is one more thing the fixed dataset must constrain, and an under-determined large network only worsens extrapolation error. Two layers is enough to fit the nine engineered features nonlinearly, with no library beyond MQL5's native arrays.

CQLQNetwork is a plain two-layer feedforward net — nine inputs, twenty-four ReLU hidden units, three linear outputs. Forward() computes identically whether called from training or the deployment EA, removing one risk: the Q-value formula can't quietly diverge between the two. That guarantee stops at the feature vector feeding it, though — build_offline_dataset.py and BuildStateVector() are separate code computing the same state in two languages, kept aligned only by manual review; see "Feature drift" below.

//+------------------------------------------------------------------+
//| CQLQNetwork::Forward                                             |
//+------------------------------------------------------------------+
void CQLQNetwork::Forward(const double &state[], double &q_out[])
  {
   ArrayResize(q_out, CQL_ACTION_COUNT);
//--- hidden layer: pre-activation = W1^T * state + b1, then ReLU
   for(int j = 0; j < CQL_HIDDEN_UNITS; j++)
     {
      double sum = m_b1[j];
      for(int i = 0; i < CQL_STATE_FEATURES; i++)
         sum += state[i] * m_W1[i][j];
      m_pre[j]  = sum;
      m_post[j] = (sum > 0.0) ? sum : 0.0; // ReLU, cached for TrainStep()
     }
//--- output layer is linear: Q(s,a) = W2^T * hidden + b2
   for(int a = 0; a < CQL_ACTION_COUNT; a++)
     {
      double sum = m_b2[a];
      for(int j = 0; j < CQL_HIDDEN_UNITS; j++)
         sum += m_post[j] * m_W2[j][a];
      q_out[a] = sum;
     }
  }

Persisting the trained network is a binary file with a small dimension header written first, so a stale or mismatched weights file fails loudly the moment it is loaded rather than silently misaligning every subsequent read:

//+------------------------------------------------------------------+
//| CQLQNetwork::SaveWeights                                         |
//+------------------------------------------------------------------+
bool CQLQNetwork::SaveWeights(const string filename)
  {
   int handle = FileOpen(filename, FILE_WRITE | FILE_BIN | FILE_COMMON);
   if(handle == INVALID_HANDLE)
     {
      Print("CQLQNetwork::SaveWeights failed to open ", filename, ", err=", GetLastError());
      return(false);
     }
//--- write the dimension header first so LoadWeights() can verify the
//--- three-way contract (this file / #define constants / any consumer)
//--- before trusting a single weight value
   FileWriteLong(handle, CQL_STATE_FEATURES);
   FileWriteLong(handle, CQL_HIDDEN_UNITS);
   FileWriteLong(handle, CQL_ACTION_COUNT);
   for(int i = 0; i < CQL_STATE_FEATURES; i++)
      for(int j = 0; j < CQL_HIDDEN_UNITS; j++)
         FileWriteDouble(handle, m_W1[i][j]);
   for(int j = 0; j < CQL_HIDDEN_UNITS; j++)
      FileWriteDouble(handle, m_b1[j]);
   for(int j = 0; j < CQL_HIDDEN_UNITS; j++)
      for(int a = 0; a < CQL_ACTION_COUNT; a++)
         FileWriteDouble(handle, m_W2[j][a]);
   for(int a = 0; a < CQL_ACTION_COUNT; a++)
      FileWriteDouble(handle, m_b2[a]);
//--- OOD reference stats travel in the same file as the weights so the
//--- two can never drift apart or be loaded from mismatched runs
   for(int i = 0; i < CQL_STATE_FEATURES; i++)
      FileWriteDouble(handle, m_stats.mean[i]);
   for(int i = 0; i < CQL_STATE_FEATURES; i++)
      FileWriteDouble(handle, m_stats.stdev[i]);
   FileClose(handle);
   return(true);
  }

Training against the actual 838-transition offline dataset produced a loss curve that's honest rather than tidy: mostly flat in the 59-61 range for most of the run, with one dip around epoch 161 that partially recovers by epoch 200. That flatness is consistent with the dataset's limits — with only 838 trades and raw (unshaped) dollar P/L as reward, much of the loss reflects irreducible reward variance, and more epochs or a different alpha may not improve it visibly.

Training loss on the actual 838-transition dataset

Fig. 3. Mean batch loss (TD term plus the CQL penalty), logged every 20 epochs during the actual training run behind this article's Strategy Tester result. The loss stays close to flat throughout, with a one-off dip near epoch 161 that does not persist — consistent with a reward signal that is mostly noise relative to the nine engineered features.

Deploying a frozen policy: the EA side

CQL_OfflineRL_EA.mq5 is intentionally the simplest file in the project. It loads the weights once in OnInit() and refuses to run at all if that load fails — there is no fallback behavior, because a fallback would mean trading on a policy nobody trained:

//+------------------------------------------------------------------+
//| OnInit                                                           |
//+------------------------------------------------------------------+
int OnInit(void)
  {
   g_ready = false;

//--- this EA is a pure consumer of an offline-trained policy: it never
//--- calls TrainStep(), so a missing weights file is a hard failure
//--- rather than something to fall back and improvise around
   if(!g_network.LoadWeights(InpWeightsFile))
     {
      Print("OnInit: failed to load ", InpWeightsFile, " - refusing to trade");
      return(INIT_FAILED);
     }

   g_h_atr = iATR(_Symbol, PERIOD_CURRENT, InpATRPeriod);
   g_h_rsi = iRSI(_Symbol, PERIOD_CURRENT, InpRSIPeriod, PRICE_CLOSE);
   if(g_h_atr == INVALID_HANDLE || g_h_rsi == INVALID_HANDLE)
     {
      Print("OnInit: indicator handle creation failed, err=", GetLastError());
      return(INIT_FAILED);
     }

   g_ready = true;
   return(INIT_SUCCEEDED);
  }

OnTick() acts at most once per new bar, not every tick — the policy was trained on bar-sampled transitions, so intra-bar noise would put it in states it's never seen. One caveat: the fragment below triggers on the bar-time change and reads buffer index 0 for ATR, RSI, and close, which on most feeds is the bar that just opened, not the one that just finished. Treat this as evaluating the newest available bar, not a guaranteed closed-bar read — shift to index 1 if your broker updates index 0 before the bar completes.

//+------------------------------------------------------------------+
//| OnTick                                                           |
//+------------------------------------------------------------------+
void OnTick(void)
  {
//--- act once per bar transition, not per tick - the policy was trained on
//--- bar-sampled transitions, so evaluating it intra-bar would feed it
//--- states it never saw during offline training
   static datetime last_bar_time = 0;
   datetime         bar_time      = iTime(_Symbol, PERIOD_CURRENT, 0);
   if(bar_time == last_bar_time || !g_ready)
      return;
   last_bar_time = bar_time;

   double state[CQL_STATE_FEATURES];
   if(!BuildStateVector(state))
      return; // indicators not warmed up yet

   double q_values[];
   int    action = g_network.SelectAction(state, q_values);
   ApplyAction((ENUM_CQL_ACTION)action);
  }

BuildStateVector() is where the training/deployment contract actually lives. Every feature here must appear in the same order, with the same normalization, as the corresponding column in build_offline_dataset.py's output. The shared CQL_STATE_FEATURES constant and the dimension check inside LoadWeights() are the only enforcement:

//+------------------------------------------------------------------+
//| BuildStateVector                                                 |
//+------------------------------------------------------------------+
bool BuildStateVector(double &state[])
  {
//--- feature layout must exactly match the columns build_offline_dataset.py
//--- wrote when the offline dataset was constructed - the three-way
//--- contract (this function / CSV columns / CQL_STATE_FEATURES) is what
//--- OnInit()'s dimension check in LoadWeights() ultimately protects
   double atr_buf[], rsi_buf[], close_buf[];
   if(CopyBuffer(g_h_atr, 0, 0, 3, atr_buf) < 3)
      return(false);
   if(CopyBuffer(g_h_rsi, 0, 0, 3, rsi_buf) < 3)
      return(false);
   if(CopyClose(_Symbol, PERIOD_CURRENT, 0, 6, close_buf) < 6)
      return(false);

   double ret1 = (close_buf[5] - close_buf[4]) / close_buf[4];  // 1-bar return
   double ret3 = (close_buf[5] - close_buf[2]) / close_buf[2];  // 3-bar return
   double norm_atr = atr_buf[0] / close_buf[5];                 // volatility, price-normalized
   double norm_rsi = (rsi_buf[0] - 50.0) / 50.0;                // centered to [-1, 1]

   long   volumes[];
   if(CopyTickVolume(_Symbol, PERIOD_CURRENT, 0, 3, volumes) < 3) // long[], never double[]
      return(false);
   double vol_ratio = (volumes[2] > 0) ? (double)volumes[0] / (double)volumes[2] : 1.0;

   double pos_exposure = 0.0;
   if(PositionSelect(_Symbol))
      pos_exposure = (PositionGetInteger(POSITION_TYPE) == POSITION_TYPE_BUY) ? 1.0 : -1.0;

//--- MQL5 has no TimeHour()/TimeDayOfWeek() (those are MQL4-only); the
//--- portable way to pull calendar fields is to unpack TimeCurrent() into
//--- an MqlDateTime struct and read its named members
   MqlDateTime dt;
   TimeToStruct(TimeCurrent(), dt);

   int i = 0;
   state[i++] = ret1;
   state[i++] = ret3;
   state[i++] = norm_atr;
   state[i++] = norm_rsi;
   state[i++] = vol_ratio;
   state[i++] = pos_exposure;
   state[i++] = (double)dt.hour / 24.0;       // session position
   state[i++] = (double)dt.day_of_week / 6.0; // weekday seasonality
   state[i++] = norm_atr * norm_atr;          // volatility-of-volatility proxy
   return(true);
  }

Selecting an action layers the out-of-distribution guard directly in front of the argmax, rather than trusting the raw Q-values everywhere. Worth being honest: FLAT is zero of 838 logged transitions, exactly the under-supported action CQL's penalty drives down hardest elsewhere. Using it as the OOD fallback is a design choice — flattening is the least committal thing the EA can do — not a claim it's validated by the data; it's not passive either, since routing an open position to FLAT makes ApplyAction() actively close it.

//+------------------------------------------------------------------+
//| CQLQNetwork::SelectAction                                        |
//+------------------------------------------------------------------+
int CQLQNetwork::SelectAction(const double &state[], double &q_out[])
  {
//--- the policy never explores here - this is a frozen, offline-trained
//--- network, so "safety" means refusing to act outside its training
//--- distribution rather than trying something new on live capital
   if(IsOutOfDistribution(state))
     {
      ArrayResize(q_out, CQL_ACTION_COUNT);
      ArrayInitialize(q_out, 0.0);
      return(CQL_ACTION_FLAT); // fallback when uncertain, not a data-validated "safe" action
     }

   Forward(state, q_out);
   int best = 0;
   for(int a = 1; a < CQL_ACTION_COUNT; a++)
      if(q_out[a] > q_out[best])
         best = a;
   return(best);
  }

The guard itself is a simple average absolute z-score against the per-feature mean and standard deviation captured at training time:

//+------------------------------------------------------------------+
//| CQLQNetwork::IsOutOfDistribution                                 |
//+------------------------------------------------------------------+
bool CQLQNetwork::IsOutOfDistribution(const double &state[])
  {
//--- without reference stats we cannot judge distance. This fails OPEN -
//--- treating the state as in-distribution rather than blocking every
//--- trade - which trades availability for protection; see the Edge
//--- cases section below before relying on this path
   if(!m_stats_ready)
      return(false);

   double sum_abs_z = 0.0;
   for(int i = 0; i < CQL_STATE_FEATURES; i++)
     {
      double sd = (m_stats.stdev[i] > 1e-8) ? m_stats.stdev[i] : 1e-8; // guard div-by-zero
      sum_abs_z += MathAbs((state[i] - m_stats.mean[i]) / sd);
     }
   double avg_abs_z = sum_abs_z / CQL_STATE_FEATURES;
   return(avg_abs_z > CQL_OOD_MAX_ZSCORE);
  }

With CQL_OOD_MAX_ZSCORE set to 3.5, a live state has to sit unusually far from anything the training data contained before the EA falls back to flat. Running it against three thousand simulated in-distribution states shows how rarely it should trip, and how sharply the threshold separates ordinary variation from a genuine untrained regime:

Distribution of live-state distance from the training set

Fig. 4. Mean absolute z-score of simulated live states against the training distribution, with the OOD guard's rejection threshold marked. Only states in the far right tail get routed to a forced-flat action instead of the network's own argmax.

Once an action clears the guard, ApplyAction() translates it into orders. The mapping is deliberately unambitious — close an opposing position first if one exists, then open in the requested direction at a fixed lot size — so that any trading mistake downstream is a policy problem to debug, not an execution-layer one:

//+------------------------------------------------------------------+
//| ApplyAction                                                      |
//+------------------------------------------------------------------+
void ApplyAction(const ENUM_CQL_ACTION action)
  {
//--- has_position must also check POSITION_MAGIC: PositionSelect() alone
//--- matches any position on this symbol, including one opened by a
//--- different EA or script sharing the account
   bool has_position = PositionSelect(_Symbol) && PositionGetInteger(POSITION_MAGIC) == InpMagic;
   bool is_long       = has_position && PositionGetInteger(POSITION_TYPE) == POSITION_TYPE_BUY;
   bool is_short      = has_position && PositionGetInteger(POSITION_TYPE) == POSITION_TYPE_SELL;

   MqlTradeRequest request;
   MqlTradeResult  result;
   ZeroMemory(request);
   ZeroMemory(result);

//--- the policy only ever expresses a target exposure (short/flat/long);
//--- translating that into orders is deliberately simple so any trading
//--- mistake is a policy problem, not an execution-layer one
   if(action == CQL_ACTION_FLAT && has_position)
     {
      request.action   = TRADE_ACTION_DEAL;
      request.symbol   = _Symbol;
      request.magic    = InpMagic;
      request.volume   = PositionGetDouble(POSITION_VOLUME);
      request.type     = is_long ? ORDER_TYPE_SELL : ORDER_TYPE_BUY;
      request.price    = is_long ? SymbolInfoDouble(_Symbol, SYMBOL_BID)
                                  : SymbolInfoDouble(_Symbol, SYMBOL_ASK);
      request.position = PositionGetInteger(POSITION_TICKET);
      if(!OrderSend(request, result) || result.retcode != TRADE_RETCODE_DONE)
         Print("ApplyAction: flatten OrderSend failed, retcode=", result.retcode);
      return;
     }

   if(action == CQL_ACTION_LONG && !is_long)
     {
      if(is_short)
         CloseCurrentPosition();
      request.action = TRADE_ACTION_DEAL;
      request.symbol = _Symbol;
      request.magic  = InpMagic;
      request.volume = InpLotSize;
      request.type   = ORDER_TYPE_BUY;
      request.price  = SymbolInfoDouble(_Symbol, SYMBOL_ASK);
      if(!OrderSend(request, result))
         Print("ApplyAction: open-long OrderSend failed, retcode=", result.retcode);
     }
   else if(action == CQL_ACTION_SHORT && !is_short)
     {
      if(is_long)
         CloseCurrentPosition();
      request.action = TRADE_ACTION_DEAL;
      request.symbol = _Symbol;
      request.magic  = InpMagic;
      request.volume = InpLotSize;
      request.type   = ORDER_TYPE_SELL;
      request.price  = SymbolInfoDouble(_Symbol, SYMBOL_BID);
      if(!OrderSend(request, result))
         Print("ApplyAction: open-short OrderSend failed, retcode=", result.retcode);
     }
  }

Edge cases and pitfalls

Feature drift between training and deployment. This is the single most common way an offline RL project silently fails after looking fine in training: if BuildStateVector() and build_offline_dataset.py ever compute a feature differently, the network evaluates live states against weights tuned for a distribution that never existed. The CQL_STATE_FEATURES check only catches a width mismatch, not a definition mismatch, so a change on one side needs the matching change on the other, by hand.

The OOD guard is a heuristic, not a proof. A per-feature average z-score treats every feature as independently informative and misses correlated shifts — a state where every feature individually looks ordinary but the combination never occurred could still slip through. Averaging across all nine also means one wildly out-of-range feature can be masked if the other eight sit close to their means. Treat CQL_OOD_MAX_ZSCORE as a coarse tripwire worth calibrating against your own dataset's spread, not a guarantee.

The OOD guard fails open, not closed. IsOutOfDistribution() returns false — in-distribution, safe to trade — whenever m_stats_ready is false, which happens if a weights file is loaded without its feature-statistics block. That choice favors keeping the EA trading over blocking it, the opposite of what "fail safe" usually means for a risk control: a missing or corrupted stats block disables the guard silently rather than disabling trading.

Alpha is not free to set once and forget. Too low, and the conservative penalty stops doing meaningful work, letting extrapolation error back in. Too high, and every action's Q-value gets pulled toward the same conservative floor, making SelectAction()'s argmax close to arbitrary among ties.

Reward design leaks into everything downstream. This implementation uses raw realized trade P/L as the reward, deliberately unshaped. A shaped reward changes what the Bellman target represents and what the CQL penalty is conservative about — changing it after a dataset is built means rebuilding the dataset, not just retraining.

Inputs are not normalized before they reach the network. The per-feature mean/stdev captured by ComputeFeatureStats() feed the OOD guard, but Forward() takes the nine features at raw scale rather than standardizing them, so features with very different native ranges can contribute unevenly to the hidden layer's activations purely by scale, not predictive value. Standardizing with those same statistics is a natural follow-up this version doesn't do.

Closing and opening are two separate round-trips, not one. When ApplyAction() reverses a position, it calls CloseCurrentPosition() then immediately sends a new order for the opposite side — two independent server calls, not one atomic operation. If the close fails or partially fills, the code still opens the new position, leaving the account holding exposure neither side accounted for.

Strategy Tester sandbox isolation. FILE_COMMON paths resolve to the shared %APPDATA%\MetaQuotes\Terminal\Common\Files\ folder rather than the per-terminal MQL5\Files\ tree, even inside a Tester agent's sandbox. If cql_weights.bin isn't under the Common folder before a run starts, OnInit() fails with a file-not-found error unrelated to the model — check the path first.

Silent capacity mismatch between the dataset and the network. Nothing forces the offline dataset to cover a wide enough range of conditions for a 24-unit hidden layer to generalize sensibly. A dataset from one trending quarter trains a network that looks converged while learning nothing transferable to a ranging regime it never saw. The OOD guard catches states far from the training distribution, but not a distribution that was simply too narrow — a data-collection problem, not one architecture tuning fixes.

Magic-number collisions across EAs on the same account. ApplyAction()'s position check filters on both _Symbol and POSITION_MAGIC == InpMagic, so it only treats a position as its own if this EA opened it — a symbol-only check could close or reverse another EA's position, and only helps if every other system uses a distinct magic number. On a netting account MetaTrader merges all positions on a symbol regardless, so the check identifies which trades this EA placed but can't isolate its position the way a hedging account can.

Testing in the Strategy Tester

These are the actual results from running the frozen policy in the Strategy Tester on XAUUSD M5, using cql_weights.bin trained offline from 838 logged trades, with no further learning during the run.

Metric/Setting	Value
Symbol/Timeframe	XAUUSD/M5
Training dataset size	838 logged transitions
Training epochs	200
Alpha (CQL penalty weight)	1.0
History quality	100%
Total trades	2 (both short; 0 long)
Win rate	50% (1 win, 1 loss)
Net profit	$0.00
Profit factor	1.00
Balance drawdown (max)	$20.10 (0.20%)
Equity drawdown (max)	$55.00 (0.55%)

Strategy Tester balance and equity curve

Fig. 5. Actual balance and equity progression for the frozen policy on XAUUSD M5. The account sat flat for almost the entire test window, with one brief short round trip (open and close, both logged as short in the table above) producing the visible spike.

Two trades across a multi-month test isn't a meaningful performance sample — the honest reading is "almost never acted," not "broke even." That traces back to the OOD guard: ComputeFeatureStats() builds its reference mean/stdev from the 838 dataset states, sampled only at moments the logging strategy opened or closed a position, not from bars in general. If that strategy traded under fairly specific conditions, most ordinary bars will legitimately score as unfamiliar, routing almost everything to CQL_ACTION_FLAT.

The guard doing exactly what it was built to do — refusing states outside what the training data supports — is the likely explanation, but not the only one: the OOD reference and the bar-by-bar states it judges were never drawn from the same sampling process, so a mismatch between a narrow reference and continuous deployment would produce the same near-constant rejection even with the threshold working as designed. Two trades can't separate the two explanations without re-evaluating the guard against a reference built from the full bar history, which this run doesn't do; this implementation simply reports the conservative behavior as measured.

The Tester run used the exact same cql_weights.bin the offline script produced, with no online learning switched on — a fair, if unglamorous, test of a policy trained without touching the market, not a measurement of trading accuracy. Two trades says nothing statistically meaningful about win rate, profit factor, or risk-adjusted return, and this run has no rule-based, buy-and-hold, or original-strategy baseline. The honest summary: it traded twice and broke even, and its caution traces mostly to how narrow its training-time reference distribution was, not its judgment when it acted.

Conclusion

Offline RL trades away what online RL relies on most — trying something and seeing what happens — and CQL is what makes that trade survivable. The mechanism fit comfortably inside native MQL5: a two-layer network, a softmax from values the forward pass already produced, a backprop loop barely longer than plain Q-learning's. The harder problem wasn't the math — it was keeping the dataset's and deployment-time feature definitions from diverging, why the OOD guard's statistics travel inside the same weights file as the policy they protect.

Worth being explicit about what this doesn't solve. CQL addresses extrapolation error in the value function; it doesn't address distribution shift between the dataset's historical period and whatever regime the market is in when the policy trades — related but not the same problem, and the OOD guard targets the second, not the first. A policy can be perfectly conservative toward its training data and still face a market that no longer matches it — an argument for rolling retraining, not a one-time run.

A few other gaps are worth naming. 838 transitions is a small sample for a method that can't go back and collect more; the dataset's coverage, not the CQL mechanism, is the binding constraint on how far this generalizes. Nothing here is checked against a train/test split, a rule-based baseline, buy-and-hold, or the logging strategy's own track record — the earlier alpha=0-vs-alpha=1 comparison measures what the CQL penalty does to a Q-value, useful but not a substitute for any of those.

A natural next step is a proper doubly-robust off-policy evaluation against the baselines above, before committing capital to the frozen policy, rather than relying on Strategy Tester replay alone — a topic for its own article.

File	Type	Description
CQLDefs.mqh	Include	Shared state/action dimensions, transition struct, feature-stats struct.
CQLQNetwork.mqh	Include	Native 2-layer Q-network: forward pass, CQL TrainStep, OOD guard, weight save/load.
CQL_OfflineTrain.mq5	Script	Offline training loop: loads the dataset, trains, saves weights. Never touches the live market.
CQL_OfflineRL_EA.mq5	Expert Advisor	Frozen deployment policy: loads weights, builds state per bar, selects action, sends orders.
Export_TradeLog.mq5	Script	Reconstructs round-trip trades from account history and writes trade_log.csv.
Export_BarHistory.mq5	Script	Exports OHLC, ATR, and RSI bar history alongside the trade log as bar_history.csv.
build_offline_dataset.py	Python script	Converts MQL5 trade-log and bar-history CSV exports into the offline transition dataset.
CQL_CompareAlpha.mq5	Script (diagnostic)	One-time tool comparing alpha=0 vs. alpha=1 Q-values on the real dataset; not part of the deployed system.