Neural Networks in Trading: Temporal Motion Aggregation (Key Components)
256
0
Introduction
The financial market, like a living organism, breathes, changes, and moves. It does not take snapshots; it creates waves. And to understand it, it is not enough to record individual points — you need to see the trajectory. It is important not only to know where the price has been, but also to get a sense of where it is headed and exactly how the price movement has unfolded. Modern analytical methods all too often reduce time to discrete points, turning the dynamic flow of the market into a series of static snapshots. However, the price does not jump from one point to another — it flows, accelerates, comes to a standstill, pauses, and then picks up speed again. Therefore, to predict its movement, we need to work not with a set of static observations, but with the dynamics themselves. With those underlying micro-movements that make up the larger trend.
In the previous article, we were introduced to the Temporal Motion Aggregation (TMA) framework, which was originally developed to analyze event data from optical flow. Strange as it may seem, it turned out to be surprisingly well aligned in spirit with financial forecasting tasks. The idea is that a stream of information can be viewed as a sequence of events linked by temporal dependencies. Every event carries not just a value, but an impulse of motion. For the authors of the original study, these were brightness-change events in the sensor. For us, these include minor price fluctuations, spikes in trading volume, and changes in liquidity. In both cases, the goal is the same — to capture the very fabric of motion and preserve in the model the temporal continuity that is usually lost in traditional data processing.
Traditional methods of analyzing financial time series view the market as a sequence of bars or candlesticks, measuring price and volume at equal time intervals. But market activity is uneven: during periods of news or panic, entire episodes can unfold in a matter of minutes, whereas during lulls, the market can seem to have almost come to a standstill. Therefore, a model that operates with fixed time intervals inevitably misses important phases of movement — it is like watching a movie with half the frames cut out. TMA, on the other hand, offers a different perspective: viewing the market as a stream of events where what matters is not the mere fact of a price change, but the nature of that change — its form, pace, and internal rhythm.
TMA is based on the sequential organization of events over time. The stream is divided into short, partially overlapping intervals, which allow the model to analyze dynamics at different scales. This is not just a matter of dividing the stream into equal intervals; it is a way to capture the market's natural rhythm — where activity intensifies and where it subsides. Features reflecting the internal structure of movement are extracted from each segment: the rate of price change, momentum strength, trading volume, and microtrend direction. But since each segment operates at its own pace, the next step is to align them and bring them to a common scale so that the model can compare and integrate the information. This stage, known as linear alignment, acts as a temporal translator. It helps synchronize segments of different scales and densities, creating a cohesive picture of movement.
Once the fragments have been aligned with one another, a key step follows: pattern aggregation. At this stage, TMA identifies stable patterns of movement that appear across multiple temporal layers. Essentially, the model looks for consistent features — recurring microstructures that together form a true trend. Noisy fluctuations are filtered out, random spikes are smoothed out, and only the essence remains — movement that follows its own internal logic. While in visual tasks this process allows us to identify an object’s trajectory amid the flicker of pixels, in a financial context it reveals the structure of capital — that invisible line along which the interests of market participants move.
This approach offers clear advantages. It naturally adapts to the irregularity of market time without imposing artificial boundaries on the market. It allows you to look beyond examining the market bar by bar, analyzing micro-movements and their consistency. It filters out noise from the signal, amplifying what really matters, and helps the model not just guess the direction, but understand the very logic behind the trend's development. As a result, TMA transforms the chaotic stream of market data into a dynamic, rhythmic structure in which each micro-wave has its own value, and the whole forms a pattern that can be analyzed and used for forecasting.
The author's visualization of the TMA framework is shown below.
In the previous article, we laid the theoretical groundwork and began implementing this approach using MQL5. The code now includes initial structures for processing the event stream and calculating correlations between temporal layers, as well as memory mechanisms that allow the model to retain information about the recent past. This marked an important step toward building a system capable of learning to understand the market the way a person senses the movement of a wave — not through numbers, but through form and rhythm.
Today we're taking the next step. The goal of this work is to move from theory to a full implementation of the TMA architecture and to demonstrate how its key principles can be embodied in algorithms. This is not an attempt to force a visual model onto financial data, but rather a thoughtful adaptation of ideas that have been proven effective in another field to a new context.
Motion Feature Encoder
Today, we will start by building the motion feature encoder (Motion Feature Encoder — MFE) module. In the original architecture, this component consists of two parallel convolutional pipelines. One is responsible for processing the main data stream, and the other for processing the correlation coefficients. After that, the two streams are combined and pass through an additional convolutional layer, where the final representation of motion features is formed.
However, if you take a closer look at the implementation, it becomes clear that the formal parallelism here is only superficial. The calculations are performed sequentially. The data streams are merely processed in alternation, rather than simultaneously. In the context of machine learning, this is perfectly acceptable, but this approach reduces efficiency and increases latency, especially when working with large amounts of data.
Our goal is not simply to replicate the original idea, but to make its implementation as high-performance as possible. Our goal is to achieve true parallelism, reduce operation execution times, and harness the full power of the OpenCL context. So today we will focus on finding ways to organize the computations so that both data streams are actually processed simultaneously. This will enable the module to become not only a logical tool but also a computationally optimized one, capable of efficiently processing high-frequency financial market data.
The multi-window convolutional layer object is ideally suited to this task. Its architecture is designed from the ground up to process multiple input data streams in parallel, making it an ideal tool for implementing the MFE concept. Unlike a conventional convolutional block, where inputs are processed sequentially, a multi-window layer allows computations to be distributed so that convolution operations are performed simultaneously.
This approach reduces latency and provides higher computational density. Each window channel of the layer can be configured to analyze a specific type of feature. As a result, we obtain a rich, multilayered representation of motion, in which each element contributes to our understanding of the overall picture.
To implement the proposed architectural solution, we create a new object, CNeuronTMAMFE, which inherits the basic functionality from the convolutional layer class, CNeuronConvOCL. This is a deliberate design choice.
Recall that the MFE module produces a generalized representation of the data using a final convolutional layer, whose purpose is to combine the preprocessing results and extract key dynamic features. We intend to implement this functionality using the parent object, since the base class already includes all the necessary tools for performing such operations in the OpenCL environment. All that remains for us to do is to add our own block to the module's structure — a system for the preliminary, independent processing of two information channels. This system will ensure that the source data and correlation coefficients are processed separately, laying the groundwork for their subsequent fusion. This approach will preserve architectural clarity while making the most effective use of computational parallelism.
class CNeuronTMAMFE : public CNeuronConvOCL { protected: CLayer cEncoder; //--- virtual bool feedForward(CNeuronBaseOCL *NeuronOCL) override; virtual bool updateInputWeights(CNeuronBaseOCL *NeuronOCL) override; virtual bool calcInputGradients(CNeuronBaseOCL *NeuronOCL) override; public: CNeuronTMAMFE(void) {}; ~CNeuronTMAMFE(void) {}; //--- //--- virtual bool Init(uint numOutputs, uint myIndex, COpenCLMy *open_cl, uint units, uint &windows[], uint window_out, ENUM_OPTIMIZATION optimization_type, uint batch); //--- virtual int Type(void) override const { return defNeuronTMAMFE; } //--- Methods for Working with Files virtual bool Save(int const file_handle) override; virtual bool Load(int const file_handle) override; //--- virtual bool WeightsUpdate(CNeuronBaseOCL *source, float tau) override; //--- virtual void SetOpenCL(COpenCLMy *obj) override; virtual void TrainMode(bool flag) override; };
The architecture of the new neuron has been designed with engineering precision. The protected section of the class contains the cEncoder object, which is a dynamic array used to store pointers to the module's components. Thanks to this approach, we can flexibly configure the encoder's internal structure by adding or removing individual elements without having to rebuild the entire architecture. The array is populated in the initialization method. This method contains the logic for building, step by step, all the components that will work together to extract, normalize, and combine motion features.
bool CNeuronTMAMFE::Init(uint numOutputs, uint myIndex, COpenCLMy *open_cl, uint units, uint &windows[], uint window_out, ENUM_OPTIMIZATION optimization_type, uint batch) { if(windows.Size() != 2) return false; //--- if(!CNeuronConvOCL::Init(numOutputs, myIndex, open_cl, 2 * window_out, 2 * window_out, window_out, units, 1, optimization_type, batch)) return false;
It all starts with initializing and validating the received parameters. Before proceeding with the assembly of the internal architecture, the method verifies that valid data has been passed. The first check verifies the size of the windows array. There should be exactly two: one channel is intended for processing the main data, and the other for analyzing correlations. If the number of windows does not match the expected number, the method stops immediately. It is like a captain who will not set sail until he is sure the crew is complete.
The next step is to call the constructor of the parent class. This is where the basic infrastructure is set up: memory is allocated, parameters are bound, and the OpenCL context is initialized. The dimensionality of the input data is doubled — after all, we'll be working with two data streams at the same time. If this step fails, there is no point in moving forward: the foundation must be solid before any structure can be built on top of it.
After successful initialization, we prepare cEncoder — a dynamic structure that will store all the layers within the module. It is cleared of any previous data and given access to the current OpenCL context. From this point on, every layer created will be more than just an object; it will be an active participant in the overall computation chain.
uint index = 0; cEncoder.Clear(); cEncoder.SetOpenCL(OpenCL); CNeuronMultiWindowsConvOCL* mw_conv = NULL; CNeuronBatchNormOCL* normal = NULL;
The first component added to the system is the multi-window convolutional block CNeuronMultiWindowsConvOCL. It acts as a primary signal analyzer — processing input data from both channels and identifying characteristic local patterns. This block can be compared to a camera lens that focuses attention on the key details of a scene, making it possible to discern the structure of motion. After successful initialization, the SoftPlus activation function is assigned to it. Smooth and free of abrupt thresholds, it helps the model remain sensitive to even the slightest changes in the analyzed data without clipping the useful range of values.
mw_conv = new CNeuronMultiWindowsConvOCL(); if(!mw_conv || !mw_conv.Init(0, index, OpenCL, windows, 2 * window_out, units, 1, optimization, iBatch) || !cEncoder.Add(mw_conv)) { DeleteObj(mw_conv) return false; } mw_conv.SetActivationFunction(SoftPlus); index++;
But feature extraction alone is not enough — it is important to stabilize the features. Therefore, the CNeuronBatchNormOCL batch normalization layer is connected immediately after the convolution. It normalizes the output data statistics, eliminating variation in values and improving the model's robustness to noise. It is like an audio filter that removes background noise and leaves only the clean signal.
normal = new CNeuronBatchNormOCL(); if(!normal || !normal.Init(0, index, OpenCL, mw_conv.Neurons(), iBatch, optimization) || !cEncoder.Add(normal)) { DeleteObj(normal) return false; } index++;
Now it is the second convolutional block's turn. At this stage, both data streams have already undergone initial processing, but they remain independent for now. We do not combine them, but continue to process them in parallel, keeping the channels separate. The temporary array temp, which contains the enlarged window sizes, plays a supporting role here — it merely sets the parameters corresponding to the output dimensions of the previous convolutional layer. Thus, the new block acts as the next level of local filtering, refining and enhancing the features already identified within each channel.
uint temp[] = {2 * window_out, 2 * window_out}; mw_conv = new CNeuronMultiWindowsConvOCL(); if(!mw_conv || !mw_conv.Init(0, index, OpenCL, temp, window_out, units, 1, optimization, iBatch) || !cEncoder.Add(mw_conv)) { DeleteObj(mw_conv) return false; } mw_conv.SetActivationFunction(SoftPlus); index++; normal = new CNeuronBatchNormOCL(); if(!normal || !normal.Init(0, index, OpenCL, mw_conv.Neurons(), iBatch, optimization) || !cEncoder.Add(normal)) { DeleteObj(normal) return false; } //--- return true; }
This is an important point. At this stage, the encoder's architecture operates like two synchronized processors working with different types of information under common control. Each convolutional stream undergoes its own cycle of activation and normalization, forming its own spatial representation of motion. This approach preserves data purity, avoids premature mixing of features, and thereby lays the groundwork for combining them later at the parent class level.
If no errors occurred during the execution of the method, true is returned, which means the module has been successfully built and is ready for use.
Essentially, the Init method establishes how the components interact, transforming a set of disparate operations into a coordinated mechanism. As a result, the encoder gains the ability to extract features and adapt to changing data dynamics.
The forward pass algorithm is simple yet elegant. There is nothing superfluous here. Just a precise sequence of computations and error-free data transfer between layers. It is this method that brings the encoder to life, transforming a set of initialized components into a functioning mechanism capable of sequentially processing and transforming the features being analyzed.
bool CNeuronTMAMFE::feedForward(CNeuronBaseOCL *NeuronOCL) { CNeuronBaseOCL* prev = NeuronOCL; CNeuronBaseOCL* curr = NULL;
The method begins by preparing local pointers. The variable prev refers to the previous layer — the one that supplies input data to the current block; curr will refer to the layer currently being processed within the encoder.
Then the main loop begins. It iterates through all the elements of the cEncoder array, which, as we recall, stores the module's layers — convolutional and normalization blocks.
for(int i = 0; i < cEncoder.Total(); i++) { curr = cEncoder[i]; if(!curr || !curr.FeedForward(prev)) return false; prev = curr; }
At each step, the method retrieves the next object from the array. If, for any reason, the reference to the layer is missing, or if the forward pass on an internal object returns an error, execution is immediately terminated. This approach makes the algorithm reliable and fault-tolerant — no single failure within the encoder will cause the entire model to crash uncontrollably.
After the operation is successfully completed, the current layer becomes the previous layer for the next element. This creates a continuous data transmission chain, in which each block receives the output of its predecessor as input.
When the entire internal encoder has finished processing, control is passed to the parent convolutional layer.
if(!CNeuronConvOCL::feedForward(prev)) return false; //--- return true; }
It is here that the data from both channels — which has already been transformed and normalized in intermediate stages — is passed up the hierarchy. The parent object performs the final convolution — the very one that generates generalized motion features and prepares them for transmission to subsequent levels of the model.
The finishing touch is the signal indicating the successful completion of the forward pass.
Visually, this process can be imagined as the smooth movement of a wave: the incoming signal passes through a series of filters, gradually being refined, structured, and imbued with meaning. Each layer is like a separate instrument in an orchestra, adding its own timbre to the overall melody of the data.
Thus, the feedForward method transforms the architectural framework of the CNeuronTMAMFE module into a living computational system. It ties all the encoder layers together, ensuring accurate, sequential information transfer between them and, through the base class, between the encoder and the rest of the model. This is the mechanism that makes it possible to implement the core concept of the TMA framework — parallel yet coordinated motion processing, in which each data fragment follows its own path toward understanding the overall dynamics of the process.
And so, step by step, an intelligent tool is born from lines of code — a module capable of capturing the rhythm of movement and transforming the chaos of market fluctuations into meaningful patterns. CNeuronTMAMFE becomes not just another element of the framework, but a bridge between theory and practice. It embodies the concept of simultaneously processing parallel data streams and lays the foundation for further expansion of TMA toward deeper analysis of motion — whether that involves forecasting market momentum, identifying trend phases, or synchronizing temporal features in financial time series. The complete code for the class and all of its methods is provided in the attachment.
Motion Pattern Aggregation (MPA) Module
The next step in building the model is the Motion Pattern Aggregation (MPA) module. It is here, as if at the focal point of a lens, that all the preprocessed features converge to form a comprehensive picture of the process dynamics. At this stage, the framework's developers are taking an unconventional — even bold — step. They introduce a cross-attention mechanism. However, it is not used in the same way as in classical architectures.
Typically, the attention-based approach involves comparing the current state with the historical context. We, so to speak, look back to understand which past data segments proved to be the most significant in shaping the current picture. Thus, context serves as a source of memory — it imbues the model with temporal coherence and helps extract patterns from a sequence of observations.
However, the creators of the TMA framework decided to take the opposite approach. Instead of framing the current state in terms of the past, they suggested considering the entire set of features in the context of the most recent state. In other words, it is no longer the past that explains the present; rather, the present becomes a kind of lens through which the significance of each preceding state is assessed.
Such a reversal of logic changes the very philosophy of analysis. While the classical attention model answers the question “Which elements of the past are important for understanding the present?”, the TMA module seeks to answer a different one: “Which states have had a real impact on where we are now?”
As a result, the mechanism ceases to be merely a tool for weighting historical data; it becomes an analytical filter capable of identifying cause-and-effect relationships between dynamic patterns. It is not just about paying attention to the past — it is about searching for the traces it has left in the present.
This approach makes the model particularly sensitive to changes in market dynamics: instead of averaging out historical data, we gain a precise understanding of which patterns actually led the system to its current state. This is precisely what makes the Motion Pattern Aggregation (MPA) module the heart of the TMA framework — it transforms a set of features into a meaningful temporal structure, where each state is evaluated in terms of its impact, rather than merely its position in the temporal sequence.
From a technical standpoint, to implement such a process, it is sufficient to simply swap the pointers to the source data objects. However, in our case, the issue is not the attention mechanism itself (which remains the same), but rather the way the data is handled. In the classical approach, the cross-attention block receives two data sources as input — the main feature tensor and a separate context. In the architecture proposed by the authors of the TMA framework, the context is represented by the system's most recent state. To do this, a final time slice is extracted from the overall data set and then passed as a separate object.
We, however, decided to take a different approach — one that is more economical and, if you will, elegant from an engineering perspective. In our data stack, the latest state is already stored in the first elements of the tensor. Additional context extraction becomes not only redundant but also resource-intensive. There is simply no point in creating a new cross-attention object just to swap the tensors. After all, we are essentially dealing with the same Self-Attention mechanism, but in a truncated form: the context is limited to a single state rather than the entire sequence.
Therefore, instead of duplicating data, we are building an extension on top of the existing cross-attention module. The idea behind it is simple: the same tensor is used simultaneously as both a source of features and context, and the length of the context window is set equal to the dimension of a single state of the system. This approach not only preserves the meaning of the original authors' idea but also significantly improves computational efficiency.
This allows us to eliminate unnecessary copies in memory, reduce the load on the OpenCL context, and speed up computations. In fact, we are not changing the structure — we are simply rethinking the logic of interaction within it.
To seamlessly integrate this logic into the architecture, a new specialized object — CNeuronTMAMPA — is created. It does not disrupt the familiar structure, but adds a link to it that enables the contextual comparison of features.
class CNeuronTMAMPA : public CNeuronBaseOCL { protected: CNeuronRelativeCrossAttention cCrossAttention; CLayer cProjection; //--- virtual bool feedForward(CNeuronBaseOCL *NeuronOCL) override; virtual bool calcInputGradients(CNeuronBaseOCL *NeuronOCL) override; virtual bool updateInputWeights(CNeuronBaseOCL *NeuronOCL) override; public: CNeuronTMAMPA(void) {}; ~CNeuronTMAMPA(void) {}; //--- virtual bool Init(uint numOutputs, uint myIndex, COpenCLMy *open_cl, uint window, uint window_key, uint units, uint segments, uint heads, ENUM_OPTIMIZATION optimization_type, uint batch); //--- virtual int Type(void) override const { return defNeuronTMAMPA; } //--- virtual bool Save(int const file_handle) override; virtual bool Load(int const file_handle) override; //--- virtual bool WeightsUpdate(CNeuronBaseOCL *source, float tau) override; virtual void SetOpenCL(COpenCLMy *obj) override; virtual void SetActivationFunction(ENUM_ACTIVATION value) override { }; };
The object is based on two key components. The first is cCrossAttention, an instance of the CNeuronRelativeCrossAttention class. It is this component that is responsible for implementing the cross-attention mechanism: it computes the matrices of keys, queries, and values; normalizes the influence coefficients; and generates the final weighted feature vector. This block essentially acts as an intelligent filter that determines which elements of the time series are most significant in the context of the current state.
The second component is a dynamic cProjection array. It serves as a container for objects used to process the analyzed features after cross-attention, converting them into a compact representation that is convenient for subsequent computations. This can be compared to the final polishing stage: after attention has identified the semantic highlights, the projection layer organizes the results so they can be passed further down the model chain.
The object initialization method serves as a bridge between theory and practice — it is here that the ideas behind the architectural redesign of the cross-attention block take shape in the code. The entire logic is structured sequentially, without any unnecessary steps, and reflects the very concept of optimization we discussed earlier: no unnecessary copies of data, maximum computational efficiency.
bool CNeuronTMAMPA::Init(uint numOutputs, uint myIndex, COpenCLMy *open_cl, uint window, uint window_key, uint units, uint segments, uint heads, ENUM_OPTIMIZATION optimization_type, uint batch) { if(!CNeuronBaseOCL::Init(numOutputs, myIndex, open_cl, window * units * segments, optimization_type, batch)) return false; activation = None;
The first step is to call the base initializer of the parent class. This call sets the neuron's basic parameters. In effect, this lays the foundation on which all the other computational blocks will be built. If the basic initialization fails, the method terminates, preventing incorrect memory usage or access to uninitialized buffers.
Next, a key component is activated — the relative cross-attention block, cCrossAttention. This block implements the intelligent mechanism for comparing states in the context of the most recent observation. During the initialization phase, it is provided with the sizes of the data and context windows, the number of elements in the sequence, and the number of segments in each of them. The key point here is that both windows use the same tensor, and the context length is artificially limited to a single state. Thus, the block has access to the entire sequence, but analyzes it exclusively in light of the latest state.
uint index = 0; if(!cCrossAttention.Init(0, index, OpenCL, window, window_key, units * segments, heads, window, segments, optimization, iBatch)) return false; index++;
Next, the projection module cProjection is constructed to perform the final structuring of the data after attention processing. It is cleared and bound to the current OpenCL context, after which several layers are added to it in sequence.
cProjection.Clear(); cProjection.SetOpenCL(OpenCL); //--- CNeuronBaseOCL* neuron = NULL; CNeuronConvOCL* conv = NULL; CNeuronBatchNormOCL* norm = NULL;
First, the base object CNeuronBaseOCL is added, which acts as a concatenation buffer. Its purpose is to combine the results of attention with the input data.
//--- Concatenated neuron = new CNeuronBaseOCL(); if(!neuron || !neuron.Init(0, index, OpenCL, 2 * Neurons(), optimization, iBatch) || !cProjection.Add(neuron)) { DeleteObj(neuron) return false; } neuron.SetActivationFunction(None); index++;
Here, too, you can see a departure from the classic implementation of transformers. Instead of summing with the residual connection stream, the authors of TMA propose concatenating the two data streams, followed by aggregation via a convolutional layer. It reduces the dimensionality of the input tensor, identifies key spatial dependencies, and prepares the features for normalization.
//--- Projection conv = new CNeuronConvOCL(); if(!conv || !conv.Init(0, index, OpenCL, 2 * window, 2 * window, window, units * segments, 1, optimization, iBatch) || !cProjection.Add(conv)) { DeleteObj(conv) return false; } conv.SetActivationFunction(SoftPlus); index++;
The SoftPlus activation function is a deliberate choice here: it provides smooth nonlinearity while maintaining stable gradients during training.
The convolutional block is followed by a CNeuronBatchNormOCL layer, which performs mini-batch normalization. This step evens out the distribution of values, speeds up convergence, and stabilizes training. After this layer, activation is disabled (None) to keep the signal unchanged before the next convolution stage.
norm = new CNeuronBatchNormOCL(); if(!norm || !norm.Init(0, index, OpenCL, conv.Neurons(), iBatch, optimization) || !cProjection.Add(norm)) { DeleteObj(norm) return false; } norm.SetActivationFunction(None); index++;
The final stage is a two-stage convolutional projection, where two more CNeuronConvOCL objects are created. The first of these processes the features directly, retaining SoftPlus activation, while the second performs an additional linear convolution without activation, serving as the final transformation stage that passes the result further through the network. This pair enables fine-tuning of data filtering: the first layer adds depth, while the second provides precision and alignment.
//--- Feed Forward conv = new CNeuronConvOCL(); if(!conv || !conv.Init(0, index, OpenCL, window, window, 2 * window, units * segments, 1, optimization, iBatch) || !cProjection.Add(conv)) { DeleteObj(conv) return false; } conv.SetActivationFunction(SoftPlus); index++; conv = new CNeuronConvOCL(); if(!conv || !conv.Init(0, index, OpenCL, 2 * window, 2 * window, window, units * segments, 1, optimization, iBatch) || !cProjection.Add(conv)) { DeleteObj(conv) return false; } conv.SetActivationFunction(None); //--- return true; }
Thus, the initialization method forms a complete computational pipeline. Everything is organized in a logical and compact manner, without redundant links or duplicate data. The architecture remains rigorous yet flexible — it is easy to scale and adapt to sequences of various lengths.
The forward pass algorithm in the feedForward method is, in essence, the heart of the Motion Pattern Aggregation (MPA) module. Here, the abstract idea of comparing states through the lens of the most recent observation is implemented in a precise and carefully calibrated sequence of operations. Each step is designed to maintain data consistency, minimize computational overhead, and fully unlock the potential of relative cross-attention.
bool CNeuronTMAMPA::feedForward(CNeuronBaseOCL *NeuronOCL) { if(!NeuronOCL) return false; if(!cCrossAttention.FeedForward(NeuronOCL, NeuronOCL.getOutput())) return false;
The method begins by verifying that the pointer to the source data object is valid. This is a formality, but a vital one: any access to an uninitialized neuron can cause the entire computation chain to fail. If the reference is null, the method terminates immediately, preventing further errors.
The next step activates a key component — the cross-attention block. This is where the core operation of the entire mechanism takes place: analyzing current features in the context of the most recent state. In fact, we provide the same tensor simultaneously as both the data to be analyzed and the context, as mentioned earlier. As a result, the system avoids copying data while still correctly calculating the relationships between states within the sequence. If the operation is successful, an output attention tensor is generated that contains features reweighted according to their importance.
Next comes the aggregation phase. The first element, cProjection[0], is retrieved from the projection block; it will be used to combine the two data streams — the original and the attention-processed streams. The Concat function is used to concatenate these streams: we literally stitch the original signal together with the attention output, forming an expanded feature representation.
CNeuronBaseOCL* neuron = cProjection[0]; uint window = cCrossAttention.GetWindow(); uint units = cCrossAttention.GetUnits(); uint segments = cCrossAttention.GetUnitsKV(); if(!neuron || !Concat(NeuronOCL.getOutput(), cCrossAttention.getOutput(), neuron.getOutput(), window, window, units)) return false;
After merging, the data is passed on through a series of layers that make up cProjection. This is where full-scale processing begins, with each layer element performing its assigned task. The loop iterates through all the components one by one, calling the forward pass method for each one. Thus, the projection module acts as a filter that gradually transforms and refines the structure of the features.
for(int i = 1; i < cProjection.Total(); i++) { neuron = cProjection[i]; if(!neuron || !neuron.FeedForward(cProjection[i - 1])) return false; }
The final touch is the SumAndNormilize operation. This step performs summation and normalization of the output data: the original signal and the projection results are combined and then undergo a normalization procedure that aligns the scale of the features.
if(!SumAndNormilize(NeuronOCL.getOutput(), neuron.getOutput(), Output, window * segments, true, 0, 0, 0, 1)) return false; //--- return true; }
The result is a clean, balanced output, ready to be passed on to the next stage of the model.
Thus, the feedForward method implements a complete cycle of semantic data processing: from identifying interdependencies between states to generating a carefully normalized result. Not a single line of code here is arbitrary — everything is guided by the principles of precision and efficiency.
Unlike the forward pass, backpropagation in the calcInputGradients method requires much finer architectural tuning. Eliminating data duplication — which is so useful for saving memory and speeding up computations — becomes a challenge here. Now, each gradient must be distributed with the utmost precision so as not to disrupt the consistency of the computation chain. This method is, in essence, the mathematical equivalent of an art restorer recovering the original meaning of each line in a complex picture of neuronal interactions.
bool CNeuronTMAMPA::calcInputGradients(CNeuronBaseOCL *NeuronOCL) { if(!NeuronOCL) return false; //--- CNeuronBaseOCL* neuron = cProjection[-1]; if(!neuron || !DeActivation(neuron.getOutput(), neuron.getGradient(), Gradient, neuron.Activation())) return false;
It all starts with a basic check to verify that the pointer to the source data object is valid. If there is no reference to the neuron, the function terminates immediately — an error at this stage could distort the entire training process.
Next, the last element of the projection block, cProjection[-1], is selected; this is where the final activation of the forward pass takes place. Calling the DeActivation method performs activation-gradient backpropagation — the gradients received from subsequent layers are adjusted according to the activation function applied during the forward pass. This is the first step toward reconstructing the chain of cause and effect.
A loop then traverses all the projection layers in reverse order. Here, each neuron computes its own contribution to the error using the CalcHiddenGradients method. This process resembles unraveling a complex fabric: layer by layer, from end to beginning, the flow of dependencies between internal feature representations is reconstructed.
for(int i = cProjection.Total() - 2; i >= 0; i--) { neuron = cProjection[0]; if(!neuron || !neuron.CalcHiddenGradients(cProjection[i + 1])) return false; }
Once processing of the projection block is complete, the next step is deconcatenation — the inverse of combining data streams during the forward pass. The DeConcat method splits the gradients into two parts: one associated with the output of the attention block cCrossAttention, and the other with the input features. This ensures that errors are correctly routed to each data source.
uint window = cCrossAttention.GetWindow(); uint units = cCrossAttention.GetUnits(); if(!DeConcat(PrevOutput, cCrossAttention.getGradient(), neuron.getGradient(), window, window, units)) return false; Deactivation(cCrossAttention)
Next, activation-gradient backpropagation occurs within the attention block — the symbolic command Deactivation(cCrossAttention) in the code indicates the need to compute the attention block's own gradients to align the direction of the backward information flow.
Next, SumAndNormilize is called to sum the gradients from the two data streams at the input data level. Values are added along the residual connection path.
if(!SumAndNormilize(PrevOutput, Gradient, PrevOutput, window, false, 0, 0, 0, 1)) return false; if(NeuronOCL.Activation() != None) if(!DeActivation(NeuronOCL.getOutput(), PrevOutput, PrevOutput, NeuronOCL.Activation())) return false;
The next step is additional activation-gradient backpropagation at the input neuron level, if an activation function has been specified for it. It ensures continuous feedback between the external input and the model's internal states.
The CalcHiddenGradients method is then called again, but this time to pass the error gradients from the attention block back to the input data level.
if(!NeuronOCL.CalcHiddenGradients(cCrossAttention.AsObject(), NeuronOCL.getOutput(), cProjection[-1].getPrevOutput(), (ENUM_ACTIVATION)NeuronOCL.Activation())) return false; if(!SumAndNormilize(NeuronOCL.getGradient(), cProjection[-1].getPrevOutput(), NeuronOCL.getGradient(), window, false, 0, 0, 0, 1) || !SumAndNormilize(NeuronOCL.getGradient(), PrevOutput, NeuronOCL.getGradient(), window, false, 0, 0, 0, 1)) return false; //--- return true; }
It is worth noting that the input data object was used as both the main stream and the context for cross-attention, so the final stage sums the data from all data streams.
If even one of these steps is not completed, the method immediately returns false, preventing the accumulation of incorrect values.
Thus, calcInputGradients is a carefully designed algorithm. Here, every line of code is designed to reconstruct the cause-and-effect relationships disrupted during the forward pass and restore the system's ability to learn. The algorithm does not simply reverse the forward pass; rather, it reconstructs the logical structure of the computations in reverse order, propagating the error across all levels of the model without losing any information.
To describe this process metaphorically, the forward pass can be thought of as an inhalation, and the backward pass as an exhalation. The first fills the system with knowledge; the second allows that knowledge to settle, be processed, and strengthen internal connections. This interdependence is what breathes life into the entire training architecture.
As a result, CNeuronTMAMPA serves as the very link that unites the semantic power of attention with the computational rigor of the framework. It does not change the architecture — it makes it smarter, allowing the model to see not just temporal dependencies, but cause-and-effect relationships as viewed through the lens of the latest state.
We have put in a lot of hard work today and have definitely earned a short break. Let's take a little time to reflect on what we have accomplished and recharge. In the next article, we will return to the project and bring what we have started to a logical conclusion.
Conclusion
In this article, we implemented the preprocessing module and the Motion Pattern Aggregation (MPA) module — the core of the TMA framework's architecture. Here, the authors' ideas took on a practical form. We eliminated redundant copying, optimized the cross-attention mechanism, and established a clear scheme for distributing gradients. The result is an elegant, high-performance system.
Next up is the final stage — training and testing the model. That is where we will test how well the implemented solutions hold up under real-world market conditions and live up to the framework's conceptual goals.
Links
Files used in this article
| # | Name | Type | Description |
|---|---|---|---|
| 1 | Study.mq5 | Expert Advisor | Expert Advisor for offline model training |
| 2 | StudyOnline.mq5 | Expert Advisor | Expert Advisor for online model training |
| 3 | Test.mq5 | Expert Advisor | Expert Advisor for model testing |
| 4 | Trajectory.mqh | Class library | Structure describing the system state and model architectures |
| 5 | NeuroNet.mqh | Class library | Class library for building a neural network |
| 6 | NeuroNet.cl | Library | OpenCL program code library |
Translated from Russian by MetaQuotes Ltd.
Original article: https://www.mql5.com/ru/articles/20208
Warning: All rights to these materials are reserved by MetaQuotes Ltd. Copying or reprinting of these materials in whole or in part is prohibited.
This article was written by a user of the site and reflects their personal views. MetaQuotes Ltd is not responsible for the accuracy of the information presented, nor for any consequences resulting from the use of the solutions, strategies or recommendations described.
Neural Additive Models for Fully Interpretable Signal Attribution in MQL5
A Grid Expert Advisor Based on a Cellular Automaton with Online Learning in MQL5 (Part I): Continuous Adaptation of the Trading Model on Each Bar
Constant Proportion Portfolio Insurance (CPPI) for MQL5: A Floor-Protected Capital Allocation Strategy
From Basic to Intermediate: Technical Indicators (II)
- Free trading apps
- Over 8,000 signals for copying
- Economic news for exploring financial markets
You agree to website policy and terms of use