What Actually Belongs in a DRL Trading Agent's Observation Vector
Neural Networks

What Actually Belongs in a DRL Trading Agent's Observation Vector

26 July 2026, 21:56
Boris Armenteros
Boris Armenteros
0
38

A PPO agent trained for two million steps on EURUSD 15-minute data with flat reward curves across all five seeds is not an optimizer problem. Switching to SAC will not fix it. Changing the learning rate will not fix it. In most of the DRL rescue work we take on, the fault is in the observation vector — and the two most commonly missing features are position state and session encoding.

Position state, or why the gradient contradicts itself

An RSI reading of 65 in an uptrend means three different things depending on what the agent is already holding. Flat: consider entering long. Long: consider holding or adding. Short: consider closing or reversing. If the observation vector does not encode the current position, all three situations arrive at the network as the same input tensor.

The result is a contradictory gradient. One batch pushes the policy toward “enter long,” another pushes it away, from what looks like the identical observation. No consistent mapping from state to action exists, so the policy never converges. This is state aliasing, and it is the single most common defect we see.

The fix is not exotic. The observation must include, at minimum, position direction (-1 short, 0 flat, 1 long), unrealized P&L as a fraction of account equity, and time in trade measured in bars.

Feature parity between training and the MetaTrader bridge

Here is the part that matters for anyone deploying to MetaTrader: whatever you compute in the Python training environment, you have to reproduce exactly in the MT5 execution bridge. Any divergence between training-time and inference-time feature computation is distribution shift — the agent meets observations in live markets it never saw in training.

The position-state block is the easiest to get subtly wrong, because in live trading it comes from the terminal, not from a dataframe. A minimal, self-contained assembly on the MT5 side:

struct PositionStateBlock
{
   double direction;        // -1 short, 0 flat, 1 long
   double unrealizedFrac;   // unrealized P&L as a fraction of equity
   double barsInTrade;      // bars the current position has been open
};

bool BuildPositionState(const string symbol, const int periodSeconds, PositionStateBlock &block)
{
   block.direction     = 0.0;
   block.unrealizedFrac = 0.0;
   block.barsInTrade   = 0.0;

   if(!PositionSelect(symbol))
   {
      return(true); // flat is a valid, fully-encoded state
   }

   const double equity = AccountInfoDouble(ACCOUNT_EQUITY);
   if(equity <= 0.0)
   {
      printf("%s: WARNING: equity <= 0, cannot normalize P&L", __FUNCTION__);
      return(false);
   }
   if(periodSeconds <= 0)
   {
      printf("%s: WARNING: invalid period", __FUNCTION__);
      return(false);
   }

   const long   type    = PositionGetInteger(POSITION_TYPE);
   const double profit  = PositionGetDouble(POSITION_PROFIT);
   const datetime opened = (datetime)PositionGetInteger(POSITION_TIME);

   block.direction      = (type == POSITION_TYPE_BUY) ? 1.0 : -1.0;
   block.unrealizedFrac = profit / equity;
   block.barsInTrade    = (double)((TimeCurrent() - opened) / periodSeconds);

   return(true);
}

The guards are not decoration. A divide-by-zero on equity or an invalid period silently poisons the observation, and the agent acts on garbage. Notice too that “flat” returns a fully encoded state (all zeros), not an error — the observation must be self-contained at every bar, including when there is no open position.

Session encoding: one symbol, several regimes

EURUSD during the Asian session and EURUSD at the London open are not the same instrument from a reward-distribution standpoint. Spread, average true range, directional behavior, and liquidity all differ materially. An agent trained on three years of data with no session context averages those regimes into one blurred policy.

Encode the hour of day as a sine-cosine pair to preserve its circular nature, then add binary flags for the major session overlaps. That is four features, and it is what lets the agent learn session-conditional behavior instead of a single average. On the MT5 side, derive the hour from broker server time so training and live use the same clock — mixing a UTC training clock with server-time inference is a distribution shift hiding in plain sight.

Audit before you tune

Two more points worth carrying over from the full write-up. First, run a lookahead audit before the first training run: confirm the observation at bar t uses only data through t-1, that rolling indicators are not forward-filled over their warmup period, that normalization parameters come from a causal expanding window, and that no reward term references bar t+1. A DRL agent exploits leakage with far more precision than a supervised model — it learns exactly which action sequence extracts the forward-looking signal, and the policy looks perfect until the first live bar.

Second, validate the environment before touching a single hyperparameter. A random-action policy should earn near-zero reward over 1,000 steps; substantially positive means the reward function pays for trading itself. You should be able to reconstruct the position from any observation alone. And five seeds should show at least a weak upward trend by 300K steps — if every curve is flat after 500K, the environment is the problem, not the algorithm.

Get the observation vector right first. The optimizer can only learn from what the agent can see.

#Neural networks, machine learning, metatrader 5, expert advisor, Python, deep reinforcement learning