A Reinforcement Learning System for Algorithmic Trading in MQL5

Introduction to the World of Self-Evolving Trading Systems

Imagine a trading robot that does not just blindly follow pre-programmed rules, but actually learns from its mistakes and successes, just like a human trader does. That is exactly the kind of system reinforcement learning provides in the context of algorithmic trading. This is not just another technical indicator or set of signals — it is a fundamentally different approach to building trading strategies, in which the machine discovers market patterns on its own through continuous interaction with the market.

#property copyright "BrainRL Expert v2.0"
#property version   "2.00"
#property strict

#include <Trade\Trade.mqh>
#include <Arrays\ArrayObj.mqh>

// System input parameters
input int      BrainsCount = 5;          // Number of brains in the collective
input double   LearningRate = 0.10;      // Learning rate  
input double   ExplorationRate = 0.30;   // Initial exploration level
input double   Gamma = 0.90;             // Discount factor
input double   MemoryDecay = 0.99;       // Memory decay
input bool     SaveBrains = true;        // Save trained brains
input bool     LoadBrains = false;       // Load saved brains
input int      SaveInterval = 500;       // Auto-save interval (in bars)

In traditional trading systems, a programmer or trader must explicitly define each rule: when to buy, when to sell, and where to place stop orders. This not only requires a deep understanding of the market, but also assumes that we can formalize all the nuances of market behavior as clear instructions. But what if the market is too complex for such formalization? What if patterns are constantly changing, adapting, and evolving? This is exactly where reinforcement learning demonstrates its true power.

The system discussed in this article is implemented in MQL5 for the MetaTrader 5 platform and constitutes a complete multi-agent machine-learning architecture. It is capable of trading independently in financial markets, continuously refining its strategies based on the experience it gains.

Architecture Philosophy: From Neurons to Collective Mind

The architecture of the system described here resembles the structure of the human brain more than that of a typical Expert Advisor. It is based on the idea of a hierarchical organization of knowledge and collective decision-making. At the very bottom level are memory neurons — the basic units of experience, each of which stores information about a specific market state and the action taken in that state.

struct MarketStateBinary
{
   int       binaryCode[32];
   double    volatility;
   double    momentum;
   double    trend;
   double    price;
   datetime  timestamp;
   double    outcome;
   int       activations;
};

These neurons do not exist in isolation. They combine to form higher-order structures, which I have called "brains" for simplicity's sake — individual agents capable of making trading decisions. Each such brain is an independent trading personality with its own memories, preferences, and trading style.

class CMachineBrain : public CObject
{
private:
   CArrayObj     m_neurons;
   Episode       m_episodes[];
   int           m_currentGeneration;
   double        m_brainComplexity;
   double        m_overallIntelligence;
   double        m_learningRate;
   double        m_explorationRate;
   double        m_gamma;
   double        m_memoryDecay;
   int           m_lastActivatedNeuron;
   MarketStateBinary m_currentMarketState;
   double        m_totalReward;
   int           m_totalTrades;
   int           m_successfulTrades;
   string        m_name;
   
   // Class balancing statistics
   int           m_buySignals;
   int           m_sellSignals;
   int           m_buyTrades;
   int           m_sellTrades;
   double        m_buyReward;
   double        m_sellReward;

But even that isn't the top of the hierarchy. Several individual brain agents come together to form collective mind — a kind of governing council, where each member has a vote, but the weight of that vote depends on their past achievements.

class CCollectiveMind
{
private:
   CMachineBrain* m_brains[];
   int            m_brainsCount;
   string         m_groupName;
   
public:
   CCollectiveMind(string name, int brainsCount, double lr, double eps, double gamma)
   {
      m_groupName = name;
      m_brainsCount = brainsCount;
      ArrayResize(m_brains, brainsCount);
      
      for(int i = 0; i < brainsCount; i++)
      {
         string brainName = name + "_Brain" + IntegerToString(i);
         double brainLR = lr * (0.8 + (MathRand() % 1000) / 10000.0 * 0.4);
         double brainEps = eps * (0.8 + (MathRand() % 1000) / 10000.0 * 0.4);
         m_brains[i] = new CMachineBrain(brainName, brainLR, brainEps, gamma);
      }
      
      Print("Collective mind '", name, "' created | Brains: ", brainsCount);
   }

This three-tier architecture provides the system with several important properties at once. First, it allows the system to store and make effective use of a vast amount of experience. Each memory neuron is a microscopic fragment of market knowledge, but together they form a rich picture of market dynamics. Second, the use of multiple agents provides diversification in decision-making and reduces the risk of catastrophic errors. Even if one brain draws the wrong conclusion, the others can correct the final decision. Third, the system becomes more robust to changes in market conditions, since different agents can specialize in different types of market situations.

Anatomy of a Memory Neuron: Quantization of Market Experience

Let's take a closer look at what the system's basic building block — the memory neuron — actually is. This is not just a database entry; it is a fully-fledged structure that encapsulates a comprehensive snapshot of market experience.

class CMemoryNeuron
{
public:
   MarketStateBinary marketState;
   double    action;
   double    reward;
   double    qValue;
   double    confidence;
   datetime  birthTime;
   int       activations;
   double    successRate;
   double    importance;
   int       generation;
   
   CMemoryNeuron()
   {
      action = 0.5;
      reward = 0;
      qValue = 0;
      confidence = 0.5;
      birthTime = TimeCurrent();
      activations = 0;
      successRate = 0.5;
      importance = 1.0;
      generation = 1;
      ArrayInitialize(marketState.binaryCode, 0);
      marketState.volatility = 0;
      marketState.momentum = 0;
      marketState.trend = 0;
      marketState.price = 0;
      marketState.timestamp = 0;
      marketState.outcome = 0;
      marketState.activations = 0;
   }
   
   void UpdateMetrics(double newReward)
   {
      activations++;
      reward = newReward;
      
      // Updating the Q-value
      qValue = qValue + LearningRate * (newReward - qValue);
      
      // Updating the success rate
      if(newReward > 0)
         successRate = successRate * 0.9 + 0.1;
      else if(newReward < 0)
         successRate = successRate * 0.9;
      
      // Updating importance
      importance = successRate * MathLog(activations + 1);
      confidence = (successRate + MathAbs(qValue)) / 2.0;
   }
};

Each neuron stores a binary representation of the market state, encoded as a 32-bit array. Why a binary representation, specifically? Because it allows for compact storage and quick comparison of states. State encoding is performed through quantization of continuous market characteristics.

In addition to the market state itself, the neuron stores the action that was taken in that state. The action is represented by a number from zero to one: values closer to zero correspond to selling, values closer to one to buying, and intermediate values can be interpreted as varying degrees of directional confidence, or even as a decision to refrain from a trade. This continuous representation of actions gives the system much greater flexibility than a rigid choice between buying and selling.

A neuron also tracks its activation statistics — how many times that particular state has occurred and been used to make decisions. The share of successful trades associated with this neuron defines its success rate. The importance of a neuron is calculated as a combination of its success rate and frequency of use — in this way, the system identifies those fragments of experience that not only work well but also occur frequently enough to be practically significant.

Stationary Features: A New Perspective on Market Representation

Unlike earlier versions of the system, which used a limited set of basic characteristics, version 2.0 uses an expanded feature space based on stationary indicators. Stationarity is a key concept in time series statistics, meaning that the statistical properties of a process do not change over time. This is critically important for trading systems, as the non-stationarity of price series is one of the main causes of a decline in algorithm performance.

The system extracts eleven types of stationary features from market data. The Relative Strength Index (RSI) measures whether an asset is overbought or oversold by normalizing information about the speed and magnitude of price changes into a range from zero to 100. The Commodity Channel Index (CCI) measures the deviation of the current price from its statistical average, which makes it possible to identify cyclical patterns regardless of the absolute price level.

The stochastic oscillator determines the position of the current closing price relative to the price range over a specific period, providing a dimensionless measure of momentum. The MACD (Moving Average Convergence Divergence) indicator focuses on the interaction between trend and momentum, identifying potential reversal points by analyzing the difference between exponential moving averages.

The Average True Range (ATR) quantifies market volatility in a way that is invariant to the direction of price movement. Bollinger Bands normalize the price's position relative to its statistical volatility, creating an adaptive channel that expands and contracts depending on market conditions.

Relationships between moving averages of different periods encode information about the strength and stability of a trend. Bill Williams fractals identify local extremes that may signal potential reversals. The Average Directional Index (ADX) measures the strength of a trend, regardless of its direction.

Williams' Percent Range, like the stochastic oscillator, assesses the price's position within a recent range, but uses an inverted scale and is sensitive to short periods. Finally, normalized price changes across various time windows capture short- and medium-term dynamics without being tied to specific price levels.

void CalculateStationaryFeatures(const MqlRates &rates[], MarketStateBinary &state)
{
   int size = ArraySize(rates);
   if(size < 50) return;
   
   double prices[];
   ArrayResize(prices, size);
   for(int i = 0; i < size; i++)
      prices[i] = rates[i].close;
   
   // RSI (14 periods)
   double rsi = CalculateRSI(prices, 14);
   state.binaryCode[0] = rsi > 70 ? 1 : 0;
   state.binaryCode[1] = rsi < 30 ? 1 : 0;
   state.binaryCode[2] = rsi > 50 ? 1 : 0;
   
   // CCI (20 periods)  
   double cci = CalculateCCI(rates, 20);
   state.binaryCode[3] = cci > 100 ? 1 : 0;
   state.binaryCode[4] = cci < -100 ? 1 : 0;
   state.binaryCode[5] = cci > 0 ? 1 : 0;
   
   // Stochastic (14, 3, 3)
   double stoch = CalculateStochastic(rates, 14, 3, 3);
   state.binaryCode[6] = stoch > 80 ? 1 : 0;
   state.binaryCode[7] = stoch < 20 ? 1 : 0;
   state.binaryCode[8] = stoch > 50 ? 1 : 0;
   
   // MACD (12, 26, 9)
   double macd, signal;
   CalculateMACD(prices, 12, 26, 9, macd, signal);
   state.binaryCode[9] = macd > signal ? 1 : 0;
   state.binaryCode[10] = MathAbs(macd - signal) > 0.0001 ? 1 : 0;
   state.binaryCode[11] = macd > 0 ? 1 : 0;
   
   // ATR (14 periods)
   double atr = CalculateATR(rates, 14);
   double atrNorm = atr / rates[size-1].close;
   state.binaryCode[12] = atrNorm > 0.01 ? 1 : 0;
   state.binaryCode[13] = atrNorm > 0.02 ? 1 : 0;
   state.binaryCode[14] = atrNorm > 0.005 ? 1 : 0;
   
   // Bollinger Bands (20, 2)
   double upper, middle, lower;
   CalculateBollingerBands(prices, 20, 2, upper, middle, lower);
   double bbPos = (prices[size-1] - lower) / (upper - lower);
   state.binaryCode[15] = bbPos > 0.8 ? 1 : 0;
   state.binaryCode[16] = bbPos < 0.2 ? 1 : 0;
   state.binaryCode[17] = bbPos > 0.5 ? 1 : 0;
   
   // MA Ratios (20 vs. 50)
   double ma20 = CalculateMA(prices, 20);
   double ma50 = CalculateMA(prices, 50);
   state.binaryCode[18] = ma20 > ma50 ? 1 : 0;
   state.binaryCode[19] = (ma20 - ma50) / ma50 > 0.01 ? 1 : 0;
   state.binaryCode[20] = prices[size-1] > ma20 ? 1 : 0;
   
   // Fractals (5 periods)
   int fractal = DetectFractal(rates, 5);
   state.binaryCode[21] = fractal == 1 ? 1 : 0;
   state.binaryCode[22] = fractal == -1 ? 1 : 0;
   state.binaryCode[23] = fractal == 0 ? 1 : 0;
   
   // ADX (14 periods)
   double adx = CalculateADX(rates, 14);
   state.binaryCode[24] = adx > 25 ? 1 : 0;
   state.binaryCode[25] = adx > 40 ? 1 : 0;
   state.binaryCode[26] = adx > 15 ? 1 : 0;
   
   // Williams %R (14 periods)
   double willR = CalculateWilliamsR(rates, 14);
   state.binaryCode[27] = willR > -20 ? 1 : 0;
   state.binaryCode[28] = willR < -80 ? 1 : 0;
   state.binaryCode[29] = willR > -50 ? 1 : 0;
   
   // Price changes (10 bars)
   double priceChange = (prices[size-1] - prices[size-11]) / prices[size-11];
   state.binaryCode[30] = priceChange > 0.01 ? 1 : 0;
   state.binaryCode[31] = priceChange < -0.01 ? 1 : 0;
   
   // Additional metrics
   state.volatility = atrNorm;
   state.momentum = priceChange;
   state.trend = (ma20 - ma50) / ma50;
   state.price = prices[size-1];
   state.timestamp = rates[size-1].time;
}

Each of these indicators offers a unique perspective on market dynamics, and together they provide a multidimensional view of the market state that is far more comprehensive than a simple analysis of price movements. It is critically important that all these features are stationary or nearly stationary — their statistical properties are relatively stable over time, which makes training more effective and the results more reliable.

The Mechanics of Learning: From Exploration to Mastery

At the heart of any reinforcement learning system is the algorithm for updating action-value estimates based on experience gained. This system uses a modified Q-learning algorithm that has been adapted to the specific characteristics of financial markets. The Q-learning formula is elegant in its simplicity: the new Q-value estimate is calculated as the old estimate plus the product of the learning rate and the difference between the observed reward and the current estimate.

This formula embodies the idea of gradually approaching the true estimate of action quality. If the reward received turns out to be higher than the current estimate, the Q-value increases, signaling to the system that this action is better than expected in this state. If the result is disappointing, the estimate decreases. The learning rate controls how aggressively the system updates its estimates — a high value results in rapid adaptation but may cause instability, whereas a low value ensures smooth, conservative learning.

void Learn(double reward)
{
   if(m_lastActivatedNeuron >= 0 && m_lastActivatedNeuron < m_neurons.Total())
   {
      CMemoryNeuron *neuron = m_neurons.At(m_lastActivatedNeuron);
      neuron.UpdateMetrics(reward);
      
      // Class-based tracking
      if(neuron.action > 0.6) // BUY
      {
         m_buyTrades++;
         m_buyReward += reward;
      }
      else if(neuron.action < 0.4) // SELL
      {
         m_sellTrades++;
         m_sellReward += reward;
      }
      
      if(reward > 0)
         m_successfulTrades++;
      
      m_totalReward += reward;
      m_totalTrades++;
   }
   
   // Periodic cleanup and evolution
   if(m_totalTrades % 100 == 0 && m_totalTrades > 0)
   {
      Prune();
      Evolve();
   }
}

However, simply updating the Q-values is not enough. The system must resolve the fundamental dilemma of reinforcement learning: striking a balance between exploration of unknown strategies and exploitation of already known good solutions. If an agent always chooses the action with the highest current Q-value, it will never discover potentially better alternatives. If, on the other hand, it is constantly experimenting, it will not be able to use its accumulated knowledge effectively.

double Think(double &features[])
{
   EncodeToBinary(features, m_currentMarketState);
   int similarNeuron = FindSimilarState(m_currentMarketState);
   double action = 0.5;
   double randomValue = (double)(MathRand() % 10001) / 10000.0;
   
   if(m_totalTrades < 150)
   {
      // Initial exploration phase
      int cycle = m_totalTrades % 6;
      if(cycle == 0 || cycle == 1)
         action = 0.75 + randomValue * 0.2;  // BUY
      else if(cycle == 2 || cycle == 3)
         action = 0.1 + randomValue * 0.2;   // SELL
      else
         action = 0.4 + randomValue * 0.2;   // HOLD
      
      m_lastActivatedNeuron = -1;
      
      if(m_totalTrades % 10 == 0 && ShowDetailedLog)
         Print("Exploration: Trade ", m_totalTrades, " | Action: ", DoubleToString(action, 3));
   }
   else if(similarNeuron >= 0)
   {
      CMemoryNeuron *neuron = m_neurons.At(similarNeuron);
      
      if(randomValue > m_explorationRate)
      {
         // Exploitation
         action = neuron.action;
         
         if(neuron.successRate > 0.6)
            action = MathMin(1.0, action * 1.1);
         else if(neuron.successRate < 0.4)
            action = MathMax(0.0, action * 0.9);
         
         if(m_totalTrades % 20 == 0 && ShowDetailedLog)
            Print("Exploit: Action=", DoubleToString(action, 3), 
                  " | SR=", DoubleToString(neuron.successRate, 2));
      }
      else
      {
         // Exploration
         action = randomValue;
         
         if(m_totalTrades % 20 == 0 && ShowDetailedLog)
            Print("Explore: Random=", DoubleToString(action, 3));
      }
      
      m_lastActivatedNeuron = similarNeuron;
      neuron.activations++;
   }
   else
   {
      // New state
      action = randomValue;
      m_lastActivatedNeuron = -1;
      
      if(m_totalTrades % 20 == 0 && ShowDetailedLog)
         Print("New state: Random=", DoubleToString(action, 3));
   }
   
   m_overallIntelligence = GetSuccessRate() * m_brainComplexity * 0.1;
   
   return action;
}

This system addresses that dilemma through a multi-stage strategy. During the very earliest stage of training — over the first 150 trades — the system operates in active exploration mode. In this phase, it does not rely on past experience, but instead cycles through different types of actions: several buy trades, several sell trades, and several periods of inactivity. It is like a novice trader who has not yet formed an opinion about the market and is simply trying out different approaches to see what works.

Once the initial exploration phase is complete, the system transitions to a more complex mode. Now, for each decision, a random number is generated, and if it exceeds the epsilon parameter (the exploration level), the system selects the best-known action — it exploits its experience. If, on the other hand, the random number is less than epsilon, an exploratory action is taken — the system tries something new, even if it seems suboptimal based on current knowledge.

Class Balancing: Addressing the Problem of Imbalance

One of the key new features of version 2.0 is the integrated class balancing system for trading decisions. The problem of class imbalance is well known in machine learning: when one class occurs significantly more frequently in the training data, the model may learn to systematically predict the more frequent class while ignoring the less frequent one. In the context of trading systems, this manifests as a tendency to open only buy trades or only sell trades, which leads to suboptimal use of market opportunities and increased risk.

The system tracks the number of buy and sell signals, as well as their performance. When a significant imbalance is detected — that is, when more than seventy percent of the signals belong to a single class — the correction mechanism is activated.

double ApplyClassBalance(double action)
{
   int totalSignals = m_buySignals + m_sellSignals;
   if(totalSignals < 50) return action;
   
   double buyRatio = (double)m_buySignals / totalSignals;
   double sellRatio = (double)m_sellSignals / totalSignals;
   
   // Imbalance correction
   if(buyRatio > 0.70 && action > 0.6)
   {
      action = action * 0.7 - 0.15;
      action = MathMax(0.0, action);
      
      if(ShowDetailedLog && m_totalTrades % 50 == 0)
         Print("Balance: BUY ratio=", DoubleToString(buyRatio, 2), 
               " -> Shift toward SELL");
   }
   else if(sellRatio > 0.70 && action < 0.4)
   {
      action = action * 0.7 + 0.35;
      action = MathMin(1.0, action);
      
      if(ShowDetailedLog && m_totalTrades % 50 == 0)
         Print("Balance: SELL ratio=", DoubleToString(sellRatio, 2), 
               " -> Shift toward BUY");
   }
   
   // Reinforcing the successful class
   if(m_buyTrades > 10 && m_sellTrades > 10)
   {
      double buyWinRate = m_buyTrades > 0 ? m_buyReward / m_buyTrades : 0;
      double sellWinRate = m_sellTrades > 0 ? m_sellReward / m_sellTrades : 0;
      
      if(buyWinRate > sellWinRate + 0.3 && action > 0.4 && action < 0.6)
      {
         action = action * 1.2;
         action = MathMin(1.0, action);
      }
      else if(sellWinRate > buyWinRate + 0.3 && action > 0.4 && action < 0.6)
      {
         action = action * 0.8;
         action = MathMax(0.0, action);
      }
   }
   
   // Signal tracking
   if(action > 0.6)
      m_buySignals++;
   else if(action < 0.4)
      m_sellSignals++;
   
   return action;
}

The correction works by modifying the suggested action. If the system is biased toward an excessive number of buy signals and generates yet another buy signal, that signal is attenuated by a mathematical transformation, shifting it closer to the neutral zone or even toward a sell signal. The same logic applies when there is an excess of sell signals.

In addition, the system analyzes the performance of each action class. If buy trades consistently generate significantly more profit than sell trades, the system gives priority to buy signals in the neutral zone of uncertainty, where the decision is ambiguous. This makes it possible to effectively leverage statistical advantages without completely disregarding the less successful class.

This approach ensures diversification of trading decisions and prevents the development of systematic bias, which can be disastrous when market conditions change. A system that has learned to trade in only one direction will be helpless when the market reverses.

Managing Complexity: Pruning and Evolution

As training progresses, the number of neurons in the system steadily increases. Each new market state has the potential to generate a new neuron, and if this process is not controlled, the system will quickly become unmanageably large. But the problem isn't just a matter of size — an excess of neurons leads to overfitting, where the system memorizes random market fluctuations instead of identifying true patterns.

void Prune()
{
   if(m_neurons.Total() < 1000) return;
   
   int removed = 0;
   
   for(int i = m_neurons.Total() - 1; i >= 0 && removed < 100; i--)
   {
      CMemoryNeuron *neuron = m_neurons.At(i);
      
      if(neuron.activations > 10 && neuron.successRate < 0.3)
      {
         m_neurons.Delete(i);
         removed++;
      }
   }
   
   if(removed > 0 && ShowDetailedLog)
      Print("Pruning: removed ", removed, " neurons");
}

The pruning process — the memory cleanup — is triggered automatically when the number of neurons exceeds one thousand. The system begins to analyze its memory in search of useless or harmful neurons. The selection criteria are simple but effective: a neuron must have accumulated enough experience to allow a statistically significant estimate of its quality, while at the same time demonstrating a low success rate.

A neuron that has been activated more than ten times but has shown a success rate of less than thirty percent clearly represents a poor strategy. This is not a random failure — it is a persistent pattern of losses. Such neurons are ruthlessly removed from the system. A single pruning operation can remove up to one hundred neurons, which prevents uncontrolled network growth and frees up resources for more promising regions of the state space.

After every hundredth trade, in addition to pruning, an evolutionary transition to a new generation occurs:

void Evolve()
{
   m_currentGeneration++;
   m_explorationRate *= m_memoryDecay;
   m_explorationRate = MathMax(0.01, m_explorationRate);
   
   if(ShowDetailedLog)
      Print("Generation ", m_currentGeneration, 
            " | Exploration: ", DoubleToString(m_explorationRate, 3));
}

This is not just a counter increment — it is a symbolic milestone in the system's maturation. With each generation, the exploration level decreases, indicating that the system has accumulated enough experience and can rely more on proven strategies. The decay coefficient applied to the exploration parameter is set to 0.99. This means that with each generation, the exploration level is kept at ninety-nine percent of its previous value.

The Wisdom of the Crowd: Collective Decision-Making

An individual agent, no matter how sophisticated it may be, is always exposed to the risk of systematic errors. Collective mind solves this problem through diversification — instead of a single agent, an ensemble of independent brains is created, each learning with slightly different parameters and therefore developing its own unique perspective on the market.

double ConsensusThink(double &state[])
{
   double totalAction = 0;
   double totalWeight = 0;
   
   for(int i = 0; i < m_brainsCount; i++)
   {
      if(CheckPointer(m_brains[i]) != POINTER_DYNAMIC) continue;
      
      double action = m_brains[i].Think(state);
      double weight = m_brains[i].GetSuccessRate() + 0.1;
      
      totalAction += action * weight;
      totalWeight += weight;
   }
   
   return totalWeight > 0 ? totalAction / totalWeight : 0.5;
}

When it is time to make a trading decision, each brain in the collective independently generates its own recommendation. These recommendations are not simply averaged — they are weighted according to each agent's success rate. A brain that consistently performs well is given greater weight in the final consensus. A brain that is struggling with current market conditions makes a smaller contribution.

The consensus formula is elegant in its simplicity. Each action proposed by an agent is multiplied by a weight equal to that agent's success rate plus a small constant (usually one-tenth). This constant ensures that even an agent with a zero success rate has at least some voice — perhaps that agent’s unconventional opinion will prove decisive in an unusual situation.

void CollectiveRemember(double action)
{
   for(int i = 0; i < m_brainsCount; i++)
      if(CheckPointer(m_brains[i]) == POINTER_DYNAMIC)
         m_brains[i].Remember(action);
}

void CollectiveLearn(double reward)
{
   for(int i = 0; i < m_brainsCount; i++)
      if(CheckPointer(m_brains[i]) == POINTER_DYNAMIC)
         m_brains[i].Learn(reward);
}

The system also tracks collective metrics — the total number of neurons in all brains and the collective's average intelligence. These metrics provide a high-level view of the system's maturity and capabilities.

Encoding Market Reality: From Prices to Patterns

For the system to learn effectively, it must perceive the market not as a chaotic stream of prices, but as a structured sequence of states. The process of encoding market information into a compact binary representation is the art of striking a balance between informativeness and generalization.

int FindSimilarState(MarketStateBinary &state)
{
   int bestMatch = -1;
   double bestSimilarity = 0;
   
   for(int i = 0; i < m_neurons.Total(); i++)
   {
      CMemoryNeuron *neuron = m_neurons.At(i);
      double similarity = CalculateSimilarity(state, neuron.marketState);
      
      if(similarity > bestSimilarity)
      {
         bestSimilarity = similarity;
         bestMatch = i;
      }
   }
   
   return bestSimilarity > 0.75 ? bestMatch : -1;
}

double CalculateSimilarity(MarketStateBinary &state1, MarketStateBinary &state2)
{
   int matches = 0;
   
   for(int i = 0; i < 32; i++)
      if(state1.binaryCode[i] == state2.binaryCode[i]) 
         matches++;
   
   return (double)matches / 32.0;
}

When the system encounters a new market state, it computes its binary code and searches for similar states in its memory. Similarity is determined by simply counting the number of matching bits. If twenty-four out of thirty-two bits match, that is a seventy-five percent similarity. The 75 percent threshold was chosen as a compromise: high enough for the states to be truly similar, but low enough for the system to generalize experience across closely related situations.

The Reward Function: The Language of Feedback

Any learning system requires a feedback loop — a way to assess how good or bad the decision it made was. For a trading system, the natural reward is the profit or loss from a trade.

double CalculateReward(double action, double entryPrice, double exitPrice, double spread)
{
   double reward;
   
   if(action > 0.6) // BUY
   {
      reward = (exitPrice - entryPrice - spread) / Point;
   }
   else if(action < 0.4) // SELL
   {
      reward = (entryPrice - exitPrice - spread) / Point;
   }
   else // HOLD
   {
      reward = 0;
   }
   
   // Normalization to the range [-1, +1]
   reward = MathMax(-1.0, MathMin(1.0, reward / 100.0));
   
   return reward;
}

Mathematically, the reward is calculated as the price difference, normalized to a range from minus one to plus one. For a sell trade, the logic is reversed — the reward is positive if the price has fallen. An important point to note is that the spread — which represents transaction costs — is deducted from the price difference.

Knowledge Persistence: Saving and Restoring

Reinforcement learning requires significant computational resources and time. It would be wasteful to lose the accumulated experience every time the system is restarted. Therefore, the ability to save the state of trained models and restore them when necessary is critically important.

bool SaveToFile()
{
   string filename = BrainFolder + "/" + m_name + ".brain";
   int handle = FileOpen(filename, FILE_WRITE|FILE_BIN|FILE_COMMON);
   if(handle == INVALID_HANDLE) return false;
   
   // Metadata
   FileWriteInteger(handle, m_neurons.Total());
   FileWriteInteger(handle, m_currentGeneration);
   FileWriteDouble(handle, m_totalReward);
   FileWriteInteger(handle, m_totalTrades);
   FileWriteInteger(handle, m_successfulTrades);
   FileWriteDouble(handle, m_learningRate);
   FileWriteDouble(handle, m_explorationRate);
   FileWriteDouble(handle, m_gamma);
   
   // Balancing Statistics
   FileWriteInteger(handle, m_buySignals);
   FileWriteInteger(handle, m_sellSignals);
   FileWriteInteger(handle, m_buyTrades);
   FileWriteInteger(handle, m_sellTrades);
   FileWriteDouble(handle, m_buyReward);
   FileWriteDouble(handle, m_sellReward);
   
   // Neurons
   int toSave = MathMin(m_neurons.Total(), 5000);
   for(int i = 0; i < toSave; i++)
   {
      CMemoryNeuron *n = m_neurons.At(i);
      
      for(int j = 0; j < 32; j++)
         FileWriteInteger(handle, n.marketState.binaryCode[j]);
      
      FileWriteDouble(handle, n.action);
      FileWriteDouble(handle, n.qValue);
      FileWriteInteger(handle, n.activations);
      FileWriteDouble(handle, n.successRate);
      FileWriteDouble(handle, n.importance);
      FileWriteInteger(handle, n.generation);
   }
   
   FileClose(handle);
   Print("Brain '", m_name, "' saved: ", toSave, " neurons");
   return true;
}

Every brain in the system can save itself to a binary file. This file contains all the information needed to fully restore the brain's state: the number of neurons, the current generation, the accumulated reward, the total number of trades, the number of successful trades, learning parameters, and class balancing statistics. Then, the complete state of each neuron is saved.

Restoration from a file occurs during brain initialization. The system attempts to locate the corresponding file, and if one exists, it reads all the data and reconstructs the array of neurons. After loading, the system is in exactly the same state as it was at the time of the last save and can resume training or trading from where it left off.

Practical Results and Prospects for Further Development

Let's examine a system test without any prior training (fully online self-learning on entirely new data) for 2025 on the EURUSD pair using H1 opening prices:

Here are the system's statistics, and they are quite good, even with very basic trading logic with basic trading logic and fixed stop-loss and take-profit levels:

As we can see, the neural network adapted to the market, identified patterns, and successfully generated a return of +41% over less than a full trading year.

Here is how this system performs when using a complex quantum scheme in the Q-learning process and a more effective order and risk management system on EURUSD H1 opening prices, also with full self-learning:

Testing on historical data demonstrates the system's capacity for adaptation and self-learning. Even without prior training, it demonstrates consistent positive performance, independently identifying market patterns and generating profit.

The key improvement in version 2.0 is solving the class imbalance problem. Previously, the system tended to favor buying or selling; now, the built-in class balancing ensures an even distribution of trading decisions, improving robustness and diversification.

An expanded feature space based on 11 types of stationary indicators gives the system a deeper understanding of market dynamics, allowing it to detect subtle patterns and adapt to different market regimes.

The knowledge persistence module allows you to save and load trained models, eliminating the need for retraining and ensuring continuous online learning.

The main limitations are high resource requirements and the risk of overfitting. As the number of neurons increases, the decision-making process slows down, and pruning only partially solves the problem. In addition, effectiveness may decline when market conditions change, requiring constant monitoring and adaptation of the model.

Conclusion

The system demonstrates that a full-fledged trading strategy based on reinforcement learning is feasible in MQL5 and can be implemented compactly and efficiently. The collective mind architecture ensures robustness and adaptability to changing market conditions, and each module — from class balancing to pruning — enhances the system’s intellectual capabilities.

Reinforcement learning helps identify hidden market patterns and enhances human analytical capabilities rather than replacing them. Further developments — including the integration of deep architectures, attention mechanisms, and meta-learning — will pave the way for more flexible and adaptive systems.

Ultimately, the value of such solutions is determined by their ability to consistently generate profits while keeping risk under control. The system presented here serves as a solid foundation for future generations of self-evolving trading AI systems.