Neural Networks in Trading: A Spiking-Semantic Approach to Spatiotemporal Identification (S3CE-Net)
217
0
Introduction
In today's financial world, the market has long since ceased to be a place where only assets are traded. Here, people trade information, time, and attention. Reaction speed is becoming the new currency, and the ability to identify patterns in a continuous stream of data is the key advantage. Every second, millions of quotes, orders, and news reports form a seething ocean of events in which chance and pattern are so closely intertwined that the boundaries between them sometimes blur. This stream has no clear boundaries — it is neither discrete nor synchronous — and therefore requires models capable of perceiving not only the sequence but also the structure of changes, connections, and tensions within the data.
Traditional methods for analyzing financial time series — whether classical statistical models, recurrent neural networks, or even heavyweight Transformers — face a number of limitations. They are often either too slow, overfit, or lose context while trying to capture only superficial dependencies. It is particularly difficult for them to work with sparse and asynchronous data: important events occur infrequently, but they are precisely what drive market dynamics. In such cases, every tick, every surge in volume, and every abnormal supply-demand imbalance become the equivalent of events — rare but extremely significant impulses on which the structure of subsequent price movement depends.
You could compare the market to a gigantic neuromorphic stream. It lives through bursts of activity, brief responses to external stimuli, and internal fluctuations. Like the nervous system, it responds to stimuli by transmitting signals through a complex network of connections. And if we view it precisely as an event system, it becomes clear that classical approaches based on uniformly spaced time samples do not reflect the true nature of what is happening. That is precisely why, in recent years, researchers have turned to architectures inspired by spiking neural networks and a neuromorphic approach to data analysis.
Against this backdrop, the S3CE-Net (Spike-guided Spatiotemporal Semantic Coupling and Expansion Network) framework emerges — an architecture proposed in the article “S3CE-Net: Spike-guided Spatiotemporal Semantic Coupling and Expansion Network for Long Sequence Event Re-Identification.” This model was proposed to address the problem of long-sequence event re-identification (long-sequence event re-identification) in data streams obtained from neuromorphic sensors. These sensors detect only changes — flashes of brightness, asynchronous signals that arise when something is actually happening. The S3CE-Net model operates at the intersection of spatial and temporal dependencies, amplifying informative signals and suppressing noise while maintaining high computational efficiency.
The key idea is not simply to analyze events over time, but also to identify the relationships between spatial channels — that is, figuratively speaking, between different parts of the field of view. In the context of finance, this can be interpreted as the relationships between various assets, levels of liquidity, or components of market structure. When one market sector begins to move while another lags behind or moves ahead, this creates a spatiotemporal correlation — a kind of resonance. It is precisely these resonances that the attention mechanism within S3CE-Net — the Spike-guided Spatiotemporal Attention Mechanism (SSAM) — attempts to capture. This mechanism assesses the significance of the current event and weights it based on the historical context, without looking ahead, which is fundamentally important for any predictive model.
Another innovation in the framework is the Spatiotemporal Feature Sampling Strategy (STFS). It allows the model to work with subsets of spatiotemporal features during training, thereby avoiding overfitting to fixed segments. Simply put, it teaches the model to recognize general patterns rather than memorize specific combinations of signals. In a financial context, this strategy is particularly relevant: the market does not tolerate templates; it is constantly changing, and any memorization leads to a decline in performance. STFS creates flexibility — it forces the model to adapt, training its ability to perceive dynamics rather than static conditions.
If we try to mentally transpose the S3CE-Net architecture into the world of trading, an analogy might be a system that listens to the market much like the auditory nerve perceives sound — not by registering every note, but by identifying transitions, contrasts, intensity, and rhythm. Such a model could detect micro-events: spikes in volume, the appearance of large orders, and brief spikes in volatility. In doing so, it would not merely record the fact of the event itself, but would also understand its significance in the context of other assets and previous spikes — much like a conductor who hears not a single violin, but the entire score.
Financial markets have long needed approaches like these. The transition from static models to systems capable of meaningfully analyzing sequences of events is not merely an evolution, but a necessity. The faster the market moves, the more important reaction speed becomes, and the more important the model’s ability to make sense of sparse data. S3CE-Net is notable for combining the lightweight nature of spiking neural networks with the expressiveness of attention mechanisms, without sacrificing speed or energy efficiency. It is precisely these qualities that make it conceptually well-suited to the needs of financial models, especially those operating in real time or on low timeframes.
Furthermore, the very philosophy of S3CE-Net — to emphasize what matters most and eliminate the unnecessary — aligns with the goals of trading. The market is full of false signals, noise, and random coincidences. And the true art of analysis lies in seeing what is most important.
The S3CE-Net Algorithm
The S3CE-Net framework is built around a central idea: to link the dynamics of spikes with the spatiotemporal patterns of the data stream, creating a model capable of capturing events and meaningfully establishing connections between them. If a classical neural network is like a camera that takes uniform frames at a constant rate, then S3CE-Net is closer to an auditory system: it hears only what actually happened and reacts to changes, transitions, and contrasts. This is the fundamental power of spiking models: they operate at the level of spikes — individual events that come together to form a complete picture.
A sequence of event tensors formed from an asynchronous stream is fed into S3CE-Net. Each individual spike detected by the sensor is described by its coordinates (x, y), a timestamp t, and a polarity p, which indicates the direction of the change — an increase or decrease in brightness. As a result, the event stream can be viewed as a set of four-dimensional data: time, two spatial dimensions, and polarity. To give this stream a regular structure, asynchronous events are grouped into a sequence of fixed time windows — a kind of mini-slice of the world, where each slice contains two channels: positive and negative changes. The set of such tensors forms a structure Xₑ ∈ RT×2×W×H, where T is the number of time steps, and W×H is the spatial resolution of the scene.
This approach solves two problems at once. First, it converts an irregular spiking stream into a format suitable for convolutional processing — that is, it makes it possible to apply time-tested feature extraction methods. Second, it preserves the very nature of event data — its sparsity, asynchrony, and directionality. While ordinary video records everything indiscriminately, here the network receives only pulse beats — energetic points of change. In a financial context, the analogue would be not every price value, but only its sharp moves, volume spikes, and liquidity surges — precisely those moments when the market wakes up.
The model's internal skeleton is SEW-ResNet, a lightweight spiking residual framework resistant to vanishing gradients. Its core is the Spike-Element-Wise Residual Block. Essentially, it is a building block of the network that combines two ideas: energy-efficient event processing and stable signal propagation through the depth of the architecture. Each block receives a stream of spikes, accumulates their potential at the neuronal level, and passes the transformed impulse forward, adding it to the original signal according to the principle of residual learning.
The classic ResNet addressed the problem of vanishing gradients in deep convolutional networks. In the spiking world, the task is the same, but under conditions where signals are event-driven and threshold-based. SEW-ResNet elegantly solves this by implementing the residual connection at the level of element-wise operations, allowing information to be transmitted without distortion while preserving the spike shape and potential stability.
This architecture serves as a fundamental extractor of spatiotemporal features. At the initial levels, it captures local structures — shapes, contours, and brightness transitions (or, by analogy with the market, micro-patterns in the flow of trades and quotes). At deeper layers, it aggregates them into temporal and spatial compositions — recognizing stable configurations of activity and regularities in the distribution of spikes. In other words, SEW-ResNet transforms a raw, chaotic sequence of events into a hierarchy of features, where each successive level becomes less dependent on specific coordinates and increasingly reflects the semantic structure of what is happening.
This multi-level representation plays a crucial role. It creates a foundation upon which higher-order mechanisms — such as semantic interaction and contextual attention — can be built. After all, for a model to make sense of events, it must first learn to see the shapes and rhythms of those events. SEW-ResNet provides exactly that — a clean yet expressive foundation in which spikes are transformed into patterns of activity.
However, despite the energy efficiency and the natural ability of SNNs to process streaming data, the SEW-ResNet structure alone does not solve the key problem — the fragmentation and discreteness of event semantics. Event streams — whether signals from a neuromorphic camera or price ticks in the market — are highly fragmented. Each event carries local information but has no connection to the context. As a result, the model can detect individual bursts of activity but cannot understand their meaning across time and space.
It is precisely at this stage that a mechanism is needed to link individual spikes into a meaningful dynamic sequence. This is where the Spike-guided Spatiotemporal Attention Mechanism (SSAM) comes into play — a kind of bridge between isolated fragments of perception. This module adds an attention component to the model, forming associative connections between events in space and time. Essentially, SSAM transforms SEW-ResNet from merely a fast event interpreter into a system capable of understanding— that is, of identifying meaningful patterns and contextual relationships in a data stream.
At the heart of this idea lies the understanding that the real world is never static. Events are inextricably linked, forming a continuous fabric of time. However, most models — even fairly complex ones — treat data as isolated snapshots, devoid of context. SSAM, on the other hand, aims to bridge this gap by providing semantic coupling between individual moments, transforming discrete perception into continuous understanding.
The main challenge in processing event data is its fragmented nature. Every event — whether it is a photon detected by a sensor or a price tick reflecting a change in the quote — exists instantaneously and carries no past with it. Without context, such a signal is almost useless. That is precisely why classical systems use buffering, averaging, or convolution-based aggregation mechanisms. But all of them, in essence, blur reality, depriving the model of perceptual subtlety. SSAM solves this problem differently: it does not smooth out the stream, but links events semantically, forming associative links between temporal states.
These links are established using Spike-Guided Spatiotemporal Attention, guided by the spikes themselves. Unlike classical attention models, in which each state can interact with all the others, SSAM follows a strict principle of causality: a current event can rely only on events that have already occurred, without looking ahead. This principle brings the architecture closer to the reality of living systems, where perception is always a split second behind and decisions are made based on accumulated experience rather than knowledge of the future.
In the context of financial markets, SSAM acts like an analyst who views price movement as a sequence of interconnected impulses. For such an analyst, every new tick is not just a number, but the continuation of a story whose context it already remembers and interprets. It does not know future prices, but it can sense the likely direction because associative links between past and present movement have already been built within it. The same holds for SSAM: each current state is formed as the result of integrating all preceding states, rather than simply reacting to the current input data.
Technically, the mechanism implements this concept through the trio of Q, K, and V — queries, keys, and values generated by independent SNN layers. These parameters serve as a kind of attention coordinate system, which the model uses to determine which parts of the past are most relevant to the present moment. For each time step, its own attention matrix At',t is formed, where the connections between the current time t and all preceding states t'≤t are computed according to the spiking principle, through a smoothed product operation with normalization.
The most important detail is the learnable relative positional bias B, which reflects the spatial relationships between elements within each frame. In classical networks, this information is often lost during convolution, but in SSAM it is preserved and enhanced, allowing the model to account not only for temporal relationships but also for the spatial topology of the data. For financial time series, this can be understood as the model’s ability to detect correlations not only between points in time, but also between different areas of the market — liquidity zones, volume clusters, and order-book structure.
Mathematically, attention in SSAM is implemented using an upper-triangular weight matrix:
This simple yet fundamentally important structure transforms a stream of spikes into a causally organized graph of temporal dependencies. Each new event updates the model's internal state, adding the semantic context of the past to it. In this way, the network, as it were, reflects on its own experience, forming a sort of cognitive trace.
The resulting representation Ot, after normalization, is passed to the next processing stage, where it is combined into the final output Ys = [O1; O2; …; OT]. In this set, each element carries not only local information but also the imprint of the entire history that came before — just like a trader who, looking at the latest candlestick, senses the entire past trend.
Thus, SSAM is to S3CE-Net what experience is to an analyst: the ability to see connections that are not immediately apparent. It ensures the model's smooth semantic breathing, creating stable perceptual coherence across its layers. As a result, the model understands and perceives the meaning of temporal structures.
If SSAM can be compared to an attentive gaze capable of discerning meaning over time, then the Spatiotemporal Feature Sampling Strategy (STFS) is more like the ability to view a situation from different angles without losing sight of the bigger picture. This mechanism teaches the model to perceive not only details but also context — that which lies beyond local perception. Essentially, STFS is a meta-level of perception that counteracts one of the key challenges of deep neural networks — their tendency to get stuck on local semantics, losing sight of the broader, global structure.
This kind of tunnel vision is not unique to artificial systems. This phenomenon can also be observed in financial markets, where a participant, focusing on a short-term price movement, loses sight of the long-term trend. The STFS mechanism aims to eliminate precisely this effect. It trains the model to perceive data simultaneously at multiple scales (local, sublocal, and global), thereby forming a stable, multilayered perception of dynamics.
From a technical standpoint, STFS is based on the idea of dividing the spatiotemporal feature field into subsets: temporal subsequences and spatial subregions. This partitioning is performed not to simplify the data, but to prompt the model to generalize. As the network encounters different combinations of fragments at each iteration, it is forced to seek meaning not in the specific structure of the feature map, but in the invariants that remain under any partitioning scheme. This is precisely the artificially developed skill of generalized perception.
Formally, the strategy operates on a deep feature map Y2, on which three interrelated procedures are performed sequentially:
- selecting subsequences in time,
- selecting subregions in space,
- formation of a global semantic descriptor.
At the first level, STFS randomly selects a subset of time steps Trandom ⊂ {1, 2, …, T}, thereby forming a kind of slice from the overall event stream. For each selected time instant, an averaged spatial feature map is computed and converted into a compact vector Vti = Avg2d(Y2[ti,:,:,:]). The set of these vectors, Ytemporal = {Vti | ti ∈ Trandom}, captures the essence of temporal subsequences: it stores not the entire stream, but its semantic reference points.
This procedure gives the model a unique form of experience. The network learns not to rely on data continuity, but to understand the data’s essence even from sampled fragments. This is similar to an experienced trader’s ability to gauge the market’s direction by looking at just a few characteristic candlesticks — without knowing the entire path, but sensing its structure.
At the second level, STFS divides the feature map Y2 into four spatial regions: upper left, upper right, lower left, and lower right. For each of them, an averaging operation (Avg2d) is performed to identify the most stable characteristics of the corresponding region. This approach can be interpreted as the formation of local perceptual representations, where each part of the scene (or market structure) is analyzed separately and then integrated into a coherent coordinate system.
Random selection of subregion boundaries (Wrandom, Hrandom) adds an element of stochasticity, which is particularly valuable in learning tasks. The model no longer depends on fixed object positions, as it learns to recognize meaning in any configuration. In a financial context, this is similar to how an analyst tracks the behavior of individual market segments while maintaining an understanding of their interrelationships.
At the third and most important level, STFS reconstructs the spatiotemporal representation Y2 and extracts from it the global descriptor Yglobal = Avg2d(Y2[:,:,:,:]). This is a condensed form of all the accumulated information — a kind of intellectual core of the model’s experience. It carries meaning rather than details — the quintessence of perception, distilled from a multitude of individual observations.
It is precisely at this stage that the model’s final ability to make semantic judgments is formed. In financial analysis, this is equivalent to the moment when a multitude of local signals, noise, and movements come together to form a unified understanding of the market state. The model no longer sees individual price spikes — it perceives the overall market tone, the dynamic melody of the data.
To improve the stability of training, STFS uses a combined loss function: triplet loss and cross-entropy loss with label smoothing, combined into a single formula. The first part teaches the model to distinguish between similar and distant semantic states (analogous to human perceptual contrast), while the second stabilizes classification confidence, avoiding excessive certainty.
During testing, only the global vector Yglobal is used, which underscores the idea that a single, deeply generalized perception is sufficient for accurate classification or state comparison. In terms of financial models, this is the equivalent of a final indicator — a concise yet meaningful number that encapsulates the entire multidimensional analysis of preceding events.
STFS completes the S3CE-Net architecture, transforming it into a system with multi-level awareness. While SEW-ResNet establishes a stable flow of information and SSAM endows the network with attention and memory, STFS teaches it to see the whole in the particular and to synthesize meaning from fragments. This is the highest form of machine perception — understanding patterns in time and space.
When applied to financial markets, this paves the way for neural network models capable of forecasting price changes and recognizing the hidden semantics of market behavior: trend phase shifts, the emergence of synergies between assets, and the formation of new liquidity patterns. In this sense, S3CE-Net with the STFS module becomes a tool for understanding — a system capable of seeing structure where others see only a stream of data.
The author’s visualization of the S3CE-Net framework is shown below.
Implementation in MQL5
Now that we have thoroughly examined the theoretical aspects of the S3CE-Net framework, it is time to move on to the practical part of our work and show how these ideas can be put into practice using MQL5. It is worth recalling that in previous articles, we have already explored the SEW-ResNet architecture, which serves as a solid foundation here. We implemented the proposed approaches by integrating them with ResNeXt, which enabled us to build a flexible platform capable of robustly processing complex temporal streams.
Now we can use these developments as a foundation and add the SSAM and STFS modules to them, so that the model does not merely register events, but learns to sense their interrelationships in time and space. In the context of financial markets, this is similar to the experience of a professional trader: it is not enough to see individual candlesticks or price ticks; it is important to sense their rhythm, understand how the past shapes the present, and how current events influence market dynamics. This is precisely what the SSAM and STFS modules add, transforming S3CE-Net from a simple analysis tool into a system capable of contextual perception and a deeper understanding of data structure.
We will begin our work by building the SSAM spatiotemporal attention module. The similarity between the proposed algorithm and causal Self-Attention is immediately apparent here. However, it incorporates elements of spiking architecture, which gives it unique dynamics and the ability to account for discrete events in time. This feature requires us to make the necessary changes to the algorithms we developed earlier so that the SSAM integration proceeds correctly and the model can effectively capture both spatial and temporal dependencies in the data.
To form queries (Q), keys (K), and values (V), we will use the existing convolutional layer algorithms in combination with spiking activations. At the same time, the attention algorithm itself will be implemented in OpenCL, which will make it possible to effectively parallelize computations and significantly speed up the processing of spatiotemporal relationships. This approach combines the advantages of spiking neural networks and high-performance parallel computing, ensuring both accuracy and speed in the operation of the SSAM module.
The implementation of the SSAM module is based on the OpenCL program kernel SpikeMHAttention, which allows computations to be fully parallelized across all spatiotemporal tokens and attention heads. The input data consists of a combined QKV tensor (queries, keys, and values), as well as a diagonal bias diag_bias, which ensures that the tokens themselves are processed correctly when calculating the attention weights.
__kernel void SpikeMHAttention(__global const float *qkv, __global const float *diag_bias, __global float *scores, __global float *out, const int dimension, const int mask_future ) { //--- init const int q_id = get_global_id(0); const int k_id = get_local_id(1); const int h_id = get_global_id(2); const int total_q = get_global_size(0); const int total_heads = get_global_size(2); //--- __local float temp[LOCAL_ARRAY_SIZE];
Each computation begins by determining the global and local indices for the current head and token, after which offsets are computed to access the corresponding Q, K, and V components in the tensor’s flat representation.
//--- Shifts const int shift_q = RCtoFlat(h_id, 0, total_heads, dimension, 3 * q_id); const int shift_k = RCtoFlat(h_id, 0, total_heads, dimension, 3 * k_id + 1); const int shift_v = RCtoFlat(h_id, 0, total_heads, dimension, 3 * k_id + 2); const int shift_s = RCtoFlat(h_id, k_id, total_heads, total_q, q_id); const int shift_out = RCtoFlat(h_id, 0, total_heads, dimension, q_id);
The main part of the algorithm involves calculating the dot product between Q and K to form the attention matrix. This adheres to the causal principle: if the mask_future flag is enabled, the current query will not take future keys into account, which is fully consistent with the SSAM concept and prevents looking ahead.
//--- Score float score = 0; if(mask_future == 0 || q_id <= k_id) { for(int d = 0; d < dimension; d++) { float q = IsNaNOrInf(qkv[shift_q + d], 0); if(q == 0) continue; float k = IsNaNOrInf(qkv[shift_k + d], 0); if(k == 0) continue; score += q * k; } } else score = MIN_VALUE;
It is important to note here that we are working with a spiking representation of keys and queries, where there is a high probability of obtaining zero values. To reduce the number of unnecessary operations and improve efficiency, we check the values retrieved from global buffers before performing mathematical operations. We then perform the computations only for those components that are nonzero. This approach makes it possible to significantly optimize the kernel's performance while maintaining the correctness of the calculations and the accuracy of the attention distribution.
Furthermore, if the current query matches the key (q_id == k_id, diagonal element), the value of score is increased by the learnable diagonal bias from diag_bias.
if(q_id == k_id) score += IsNaNOrInf(diag_bias[q_id], 0);
After that, the attention coefficients are normalized by calling the LocalSoftMax function.
//--- norm score score = LocalSoftMax(score, 1, temp); scores[shift_s] = score;
It is worth noting here that data normalization is performed within a work-group that corresponds to one specific query Q. In other words, SoftMax is not applied to the entire attention matrix at once, but locally across all keys K associated with the given query and attention head. This ensures a correct per-query attention distribution while preserving a high degree of parallelism at the OpenCL level.
In the final stage, the output tensor out is formed. Each value vector V is multiplied by the corresponding attention weight and accumulated using LocalSum.
//--- out for(int d = 0; d < dimension; d++) { float val = 0; if(score > 0) { float v = IsNaNOrInf(qkv[shift_v + d], 0); if(v != 0) val = v * score; } val = LocalSum(val, 1, temp); if(k == 0) out[shift_out + d] = val; } }
This allows us to obtain an accurate, weighted representation for each token by combining information from all relevant temporal and spatial positions.
The presented kernel implements the entire Spatiotemporal Attention mechanism at a low level with high performance, leveraging the capabilities of OpenCL for parallelization across tokens, attention heads, and spatiotemporal coordinates. In financial terms, this is similar to the work of an experienced analyst who simultaneously evaluates the relationships among many temporal and local indicators, assigning an importance weight to each one based on context and past experience.
To train the model correctly, it is important not only to perform forward attention computations but also to distribute backward-pass errors among all participants in the process, taking into account their impact on the final result. This functionality is implemented using the SpikeMHAttentionGrad kernel, which allows the gradients for Q, K, V, and the diagonal bias diag_bias to be computed simultaneously and in a coordinated manner.
__kernel void SpikeMHAttentionGrad(__global const float *qkv, __global float *qkv_gr, __global const float *diag_bias, __global float *diag_bias_gr, __global const float *scores, __global const float *gradients, const int dimension, const int mask_future ) { //--- init const int global_id = get_global_id(0); const int local_id = get_local_id(1); const int h_id = get_global_id(2); const int total_global = get_global_size(0); const int total_local = get_local_size(1); const int total_heads = get_global_size(2); //--- __local float temp[LOCAL_ARRAY_SIZE];
The kernel begins by defining global and local indices. To store intermediate sums and correctly compute the gradient of the SoftMax function, a local array is created to ensure efficient communication between threads within a single work-group.
First, the gradients for the V values are computed, since they directly affect the module's output representation. In the forward pass, each value of V affects all elements of the resulting tensor through the corresponding attention coefficient. This means that when calculating the error gradient for a specific element of V, we multiply the gradient at the level of the output tensor by the same attention coefficient, accurately reflecting the extent to which that particular value of V influenced the final result.
//--- Value Gradient global_id -> v_id, local_id -> q_id { //--- Shifts const int shift_v = RCtoFlat(h_id, 0, total_heads, dimension, 3 * global_id + 2); const int shift_s = RCtoFlat(h_id, glodal_id, total_heads, total_q, local_id); const int shift_out = RCtoFlat(h_id, 0, total_heads, dimension, local_id); for(int d = 0; d < dimension; d++) { float grad = 0; if(mask_future == 0 || local_id <= global_id) { float score = IsNaNOrInf(scores[shift_s], 0); if(score > 0) grad = IsNaNOrInf(score * gradients[shift_out + d], 0); } grad = LocalSum(grad, 1, temp); if(local_id == 0) qkv_gr[shift_v + d] = grad; } }
In doing so, we take the causality of the algorithm into account. This makes it possible to analytically determine which elements of the resulting sequence were not influenced and to immediately skip unnecessary mathematical operations and costly accesses to global memory.
After that, all partial gradients are accumulated using local summation, and the resulting value is written to the corresponding element of the global buffer qkv_gr. Thus, each component of V receives an accurate error signal proportional to its contribution to the result, ensuring precise and consistent model training.
Next come the queries Q. For these, the gradient of the attention coefficients is first computed as the products of the corresponding value vectors V and the output gradients; then the local gradient function of SoftMax is applied, adjusting the distribution of signals according to the normalization.
//--- Query Gradient global_id -> q_id, local_id -> k_id/v_id { //--- Shifts const int shift_q = RCtoFlat(h_id, 0, total_heads, dimension, 3 * global_id); const int shift_k = RCtoFlat(h_id, 0, total_heads, dimension, 3 * local_id + 1); const int shift_v = RCtoFlat(h_id, 0, total_heads, dimension, 3 * local_id + 2); const int shift_s = RCtoFlat(h_id, local_id, total_heads, total_q, global_id); const int shift_out = RCtoFlat(h_id, 0, total_heads, dimension, global_id); //--- 1. Score grad float grad_s = 0; if(mask_future == 0 || global_id <= local_id) for(int d = 0; d < dimension; d++) { float val = IsNaNOrInf(qkv[shift_v + d], 0); if(val == 0) continue; grad_s += IsNaNOrInf(qkv[shift_v + d] * out_gr[shift_out + d], 0); } //--- 2. SoftMax grad grad_s = LocalSoftMaxGrad(scores[shift_s], grad_s, 1, temp); if(global_id == local_id) diag_bias_gr[gloval_id] = grad_s;
Finally, the gradient of Q is formed as a combination of the score gradient and the keys K, allowing each query component to receive a weighted error signal. The diagonal elements diag_bias receive a separate gradient, taking into account their role in amplifying a token’s influence on itself.
//--- 3. Query grad for(int d = 0; d < dimension; d++) { float grad = 0; if(mask_future == 0 || global_id <= local_id) { float key = IsNaNOrInf(qkv[shift_k + d], 0); if(key != 0) grad = key * grad_s; } grad = LocalSum(grad, 1, temp); if(local_id == 0) qkv_gr[shift_q + d] = grad; } }
At the same time, we strictly account for causality, a key principle of SSAM: the model does not look ahead, because the corresponding mask is already applied during the forward pass. This requires a careful approach to backpropagation: the SoftMax derivative returns a nonzero gradient even for zero values, which could potentially generate an error signal for elements in future time steps. In real-time processing, there should be no such signals; otherwise, the principle of causality is violated. To preserve the correctness of training and avoid unnecessary computations, we check the values of the keys and queries before performing mathematical operations and exclude all future components from the calculation.
In addition, we take into account the spiking nature of the data being analyzed, a significant portion of which may be represented by zero values. Therefore, we first check the received data, and then perform mathematical operations only on nonzero elements. This not only reduces the computational load but also ensures that gradients are distributed accurately only across spikes that actually exist.
In the next step, the gradients of the keys K are computed. For each key, the contribution of all queries Q is accumulated, taking into account their interaction with the corresponding values V through the attention coefficients.
//--- Key Gradient global_id -> k_id, local_id -> score_id/v_id/dimension { //--- Shifts const int shift_k = RCtoFlat(h_id, 0, total_heads, dimension, 3 * global_id + 1); const int shift_v = RCtoFlat(h_id, 0, total_heads, dimension, 3 * local_id + 2); const int shift_out = RCtoFlat(h_id, 0, total_heads, dimension, local_id); float grad = 0; for(int q_id = 0; q_id < total_q; q_id++) { //--- 1. Score grad local_id -> score_id/v_id float grad_s = 0; const int shift_s = RCtoFlat(h_id, local_id, total_heads, total_q, q_id); int shift_q = RCtoFlat(h_id, 0, total_heads, dimension, 3 * q_id); if(mask_future == 0 || q_id <= local_id) for(int d = 0; d < dimension; d++) { float val = IsNaNOrInf(qkv[shift_v + d], 0); if(val == 0) continue; grad_s += IsNaNOrInf(val * out_gr[shift_q + d], 0); } //--- 2. SoftMax grad grad_s = LocalSoftMaxGrad(scores[shift_s], grad_s, 1, temp); BarrierLoc; if(global_id == local_id) temp[0] = grad_s; BarrierLoc; grad_s = temp[0]; //--- 3. Key grad local_id -> dimension if(local_id < dimension) { float query = IsNaNOrInf(qkv[shift_q + local_id], 0); if(query != 0) grad += IsNaNOrInf(query * grad_s, 0); } } if(local_id < dimension) qkv_gr[shift_k + local_id] = IsNaNOrInf(grad, 0); } }
The causal mask ensures that future tokens do not affect the current computation, and that gradients are accumulated and stored in qkv_gr. This approach allows errors to be distributed correctly while simultaneously updating all SSAM parameters, thereby maintaining consistency and training accuracy.
As a result, the SpikeMHAttentionGrad kernel fully implements the backpropagation mechanism for SSAM, ensuring consistent updates to Q, K, V and the diagonal biases. In the context of financial markets, this can be viewed as a process in which each forecast error is carefully distributed among all influencing factors, allowing the system to gradually learn to assess the relationships between events correctly and forecast market dynamics with a high degree of accuracy.
We have gotten a lot done today, and it is time to take a short break. Let's give our minds a rest so that the information we have received has time to sink in and be processed. In the next article, we will continue implementing the approaches of the S3CE-Net framework using MQL5. Step by step, we are moving toward the full integration of all modules and putting them to the test in practice.
Conclusion
In this article, we explored the S3CE-Net framework and its key mechanisms — the spatiotemporal SSAM and the STFS sampling strategy. The model works with spiking events and strictly adheres to causality, which allows it to accurately identify important patterns and correctly account for time-series dynamics without looking ahead.
S3CE-Net combines a lightweight architecture, high parallelism, and the ability to identify complex temporal and spatial relationships. For financial markets, this means responding quickly to signals and making precise decisions amid a massive flood of noisy data.
Implementation using MQL5 tools will pave the way for testing the model in real-world trading scenarios. S3CE-Net demonstrates how modern methods for handling sparse events can serve as a powerful tool for market analysis and forecasting. We have only just begun to explore the framework's potential, and in the next article, we will continue with its practical implementation.
References
- S3CE-Net: Spike-Guided Spatiotemporal Semantic Coupling and Expansion Network for Long-Sequence Event Re-Identification
- Other articles in this series
Programs used in the article
| # | Name | Type | Description |
|---|---|---|---|
| 1 | Study.mq5 | Expert Advisor | Expert Advisor for offline model training |
| 2 | StudyOnline.mq5 | Expert Advisor | Online model training Expert Advisor |
| 3 | Test.mq5 | Expert Advisor | Model testing Expert Advisor |
| 4 | Trajectory.mqh | Class Library | Structure for describing the system state and model architectures |
| 5 | NeuroNet.mqh | Class Library | Class library for building a neural network |
| 6 | NeuroNet.cl | Library | Code library for the OpenCL program |
Translated from Russian by MetaQuotes Ltd.
Original article: https://www.mql5.com/ru/articles/19928
Warning: All rights to these materials are reserved by MetaQuotes Ltd. Copying or reprinting of these materials in whole or in part is prohibited.
This article was written by a user of the site and reflects their personal views. MetaQuotes Ltd is not responsible for the accuracy of the information presented, nor for any consequences resulting from the use of the solutions, strategies or recommendations described.
Beyond Clone and Commit: The Advanced Side of MQL5 Algo Forge
Price Action Analysis Toolkit Development (Part 87): Searching Chart History with Recorded Double-Bottom Samples
Features of Experts Advisors
Beyond Tick Volume: Measuring Market Activity from the Waits Between Quotes
- Free trading apps
- Over 8,000 signals for copying
- Economic news for exploring financial markets
You agree to website policy and terms of use