Neural Networks in Trading: An Intelligent Forecast Pipeline (Sparse Mixture of experts)

Introduction

Financial markets form a complex, chaotic, and high-frequency system. Coarse approximations and average values do not work here. Every bar and every price movement is the result of a multitude of factors, ranging from fundamental news to momentum trading. That is precisely why working with market time series requires a special approach: one that is sensitive to minute details, robust to noise, and capable of discerning structure amid chaos.

The Time-MoE framework offers exactly this kind of architecture. This is not just a transformer adapted for time series. It is a cohesive system in which each time step of the history being analyzed is treated as a unique token. These tokens undergo a series of transformations while retaining their individuality and temporal context. This approach allows the model to process high-frequency data and identify patterns that are not detectable using traditional aggregation methods.

The first stage of processing is the embedding layer, in which the data pass through nonlinear transformations. This helps capture complex relationships between features, whether the relationship between price and volume, the direction of indicators, or the strength of the most recent price movement. The resulting hidden representation serves as the basis for further analysis.

The tokens are then sent to a series of Transformer blocks. Here, the model looks back, forming a representation of the current moment based on accumulated experience. This is achieved through an attention mechanism in which each token is compared with previous tokens, and the importance of specific elements is determined not manually but by the model itself during training. This makes it possible to take into account both short-term momentum and long-term trends.

Time-MoE places particular emphasis on robustness to noise. To this end, the architecture incorporates normalization mechanisms that help smooth out random outliers and amplify meaningful signals. As a result, attention is not focused on individual anomalies but is distributed more evenly and meaningfully, which is particularly important in conditions of volatility and market noise.

The key difference between Time-MoE and classical transformers is the use of a sparse mixture of experts (Mixture-of-Experts — MoE). In each block, the router selects only a subset of the available experts that actually participate in processing the current token. This solution significantly reduces the computational load and allows the model to be scaled without an exponential increase in resource requirements. In addition, Time-MoE includes one shared expert that is always active and provides the model with robustness against incorrect routing decisions.

The final stage is forecasting. Here, the model generates several forecasts simultaneously for different time horizons. This approach makes it possible to account for both short-term signals and long-term trends simultaneously, providing the trader or analytical system with a wide range of scenarios. During training, the model learns across all horizons simultaneously, which makes it more flexible and better adapted to changing market conditions.

Thus, Time-MoE is:

• a detail-sensitive model that works with per-time-step tokens;
• robust to noise and outliers through attention normalization;
• a scalable architecture with sparse use of experts;
• a universal mechanism for multi-horizon forecasting.

The author's visualization of the Time-MoE framework is shown below.

Today, we will continue the work we started earlier and focus on a key element of the Time-MoE framework — the sparse mixture of experts (Sparse Mixture of Experts). In the previous part, we built the model's foundation step by step, forming tokens and hidden representations with SwiGLU embeddings; now it is time to move on to the architectural highlight on which the efficiency and scalability of the entire system largely depend.

In this article, we will examine in detail how the group of experts operates and how the computations are distributed. We will not merely describe a theoretical framework; instead, we will move on to the actual implementation of a sparse MoE using MQL5, with a focus on practical aspects.

Developing the Architecture

Before we get down to the actual implementation of the sparse mixture of experts, let’s pause for a moment and reflect. As before, we remain committed to the idea of having all experts work in parallel — an approach that fits perfectly within the paradigm of massively parallel computing. Therefore, without hesitation, we will move the bulk of the workload into the OpenCL context. However, this raises an important question: how exactly should we organize sparse activation of experts while maintaining the model's efficiency and trainability?

First, let's review a key point. In their original paper, the authors of Time-MoE proposed an expert architecture that differs significantly from the classic FeedForward block, which we are familiar with from the Transformer architecture. Specifically, the first layer in the experts has been replaced with a SwiGLU transformation. This change is entirely justified: as we have already seen, SwiGLU gives the model greater flexibility in processing features and better captures nonlinear relationships. Fortunately, we already have an implementation of this layer — in the previous article, we developed the CNeuronSwiGLUOCL component. Now we can confidently use it as the first stage of our mixture of experts, scaling the number of output filters according to the number of parallel submodels.

Each filter has its own trainable parameters. In essence, this is a separate expert model. If we group the outputs by the number of experts, we will reproduce the structure we need: one input — a set of independent experts working in parallel. In the second stage of processing, we can apply multi-window convolution, which we are already familiar with from our previous work. This layer will help consolidate the information within each expert and prepare it for aggregation.

So far, everything seems quite logical. But there is one important caveat: the described algorithm lacks the main element — sparsity. Yes, we can multiply the outputs of all the experts by the activation mask and get the correct result. However, all experts still continue to compute, even if their contribution is merely suppressed by the mask. This approach is acceptable in small models, but becomes inefficient as the system scales: an increase in the number of experts, higher dimensions, and greater network depth — all of these factors dramatically increase the computational load.

This brings us to the key question: at what point in the algorithm, and how, should sparsity be introduced?

At first glance, it might seem that the best solution would be to disable unused experts at all levels — after all, this drastically reduces the computational load. However, there is a much more subtle and important issue at play here. The point is that the very idea behind MoE is for different experts to learn different subtasks. At the same time, each develops its own specialization. But who determines which experts are active in a given situation? The answer is a router that uses the input data to select a subset of models for activation.

The problem arises when the router focuses too early on only a small set of experts and begins to use them indiscriminately — regardless of the context. Such a model may produce good results in the early stages, but it will lose its ability to adapt because it never tries alternative routes. In such cases, unused experts simply don't get a chance to prove their effectiveness in other scenarios. This creates a local comfort effect: the router knows only its favorites and, without trying others, locks itself into the same pattern. This leads to a decrease in model diversity and a degradation of its generalization ability.

Thus, our goal is to train the router not simply to select experts, but to adapt its selection strategy based on the characteristics of the input data. We must create conditions under which the model will explore the behavior of other experts, even if they are initially less effective. Only in this way will it be able to learn to allocate tasks more flexibly, thereby improving overall accuracy and resilience to market changes.

In seeking a balance between computational efficiency and training coverage, it was decided to implement sparse use of experts only in the second layer of the MoE block, combining this layer with the result aggregation process. This approach simplifies the architecture, reduces the computational load, and at the same time preserves the necessary flexibility for training the router.

The key aspect of this design is the mechanism for propagating the error gradient. We deliberately reject the idea of directly training unused experts — after all, the whole point of MoE is precisely to specialize different experts for different tasks. However, to prevent the model from getting stuck on the same set of experts, we route the gradient to the router, giving it a signal that the current selection may be ineffective. This allows the routing algorithm itself to be trained: next time, it can activate a different set of experts if the current one produced a poor forecast.

Thus, even if only two experts were activated in a particular case, the router learns that others could have been selected as well. This is not merely a technical trick, but a powerful adaptation tool that allows the model to gradually learn the relationship between the characteristics of the input data and the optimal configuration of active experts.

Masked-Window Convolution

The main approaches have been defined, and now, rolling up our sleeves, we move from theory to practice. In the first phase of implementation, we need to develop a key component — the second-layer MoE object, which is responsible for sparse activation of experts and aggregation of their results. This component will serve as the central link in the model: data from all experts will pass through it, but in each individual pass, only a few of them will be activated.

We have moved the selection of active experts into a separate module — the router. Its task is to analyze the input data and generate an activation mask that determines which experts should be involved for each token. This mask is fed into this object along with the main tensor of the features being analyzed, determining the configuration of active computations at the current step.

An important assumption: all experts have the same architecture and return results of a fixed dimension. At any given moment, however, only a limited number of models are active — usually far fewer than the dimensionality of the output space. The remaining experts stay in dormant mode; they do not participate in the computations and do not consume resources.

This approach offers two key advantages:

1. A significant reduction in computational load — critically important when scaling models and increasing the number of experts.
2. Increased specialization: each expert can focus on studying its own narrow task subspace. As a result, the model forms genuine expert knowledge, distributed among the participating experts.

We have moved the bulk of the computations into the OpenCL context, which allows us to make the most efficient use of parallel computing on GPUs and other compatible devices. Our focus is on the forward-pass kernel of the second MoE layer, which implements sparse activation of selected experts and aggregates their results.

In the kernel parameters, we receive the weights of all experts, the input data, the activation mask, and several parameters that define the window structure and dimensions.

__kernel void FeedForwardMaskMultWinConv(__global const float *matrix_w,
                                         __global const float *matrix_i,
                                         __global const float *masks,
                                         __global float *matrix_o,
                                         const int inputs,
                                         const int window_in,
                                         const int windows_total,
                                         const int activation
                                        )
  {
   const size_t u = get_global_id(0);
   const size_t w = get_global_id(1);
   const size_t v = get_global_id(2);
   const size_t units = get_global_size(0);
   const size_t window_out = get_global_size(1);
   const size_t variables = get_global_size(2);

Each computational thread corresponds to a specific element of the results tensor — by sequence position, token element, and variable. This pointwise addressing provides a high level of parallelism.

It is important to emphasize that we made an assumption in advance: the number of active experts in each forward pass is significantly smaller than the token dimensionality in the results tensor. This is a key point that determines the task distribution logic within the kernel. We split the task space by token elements and iterate over the active experts in a loop inside the kernel itself. This approach makes it possible to use computational resources efficiently while preserving the sparsity of expert activation.

In the kernel body, we identify the computational thread in the task space and determine the offset in the data buffers to the elements being analyzed.

   const int shift_in = u * window_in * windows_total;
   const int shift_in_var =  v * units * window_in * windows_total;
   const int shift_out = (u + v * units) * window_out + w;
   const int shift_mask = (u + v * units) * windows_total;
   const int shift_weight = (v * window_out * windows_total + w) * (window_in + 1);
   const int step_weight = window_out * (window_in + 1);

Next, we set up a loop structure. The outer loop iterates over all experts and checks their status through the activation mask. If an expert is not active in this window, its contribution is skipped, eliminating unnecessary computations and saving resources. For active experts, a weighted sum is computed over the input data with a bias added. The final values are accumulated and then passed through an activation function.

float sum = 0;
for(int w_in = 0; w_in < windows_total; w_in++)
  {
   float m = IsNaNOrInf(masks[shift_mask + w_in], 0);
   if(m < FLT_EPSILON)
      continue;
   const int shift_in_loc = shift_in + w_in * window_in;
   const int shift_weight_loc = shift_weight + w_in * step_weight;
   for(int i = 0; i < window_in; i++)
      if((shift_in_loc + i) < (inputs / variables))
         sum += IsNaNOrInf(matrix_i[shift_in_var + shift_in_loc + i], 0) * 
                matrix_w[shift_weight_loc + i] * m;
   sum += matrix_w[shift_weight_loc + window_in] * m;
 }

Please note that during the aggregation process, we do not simply add up the outputs of the active experts; rather, we multiply each output by the corresponding mask value. In the case of binary masking, where the mask contains only zeros and ones, this adds no additional value. However, this approach leaves us with an important option: to implement weighted aggregation. If the mask contains not just flags but actual weighting coefficients, we will be able to control each expert’s contribution to the result— up to soft selection and probabilistic routing.

The resulting values are passed through an activation function and stored in the results buffer.

 matrix_o[shift_out] = Activation(sum, activation);
}

After performing the forward pass — where active experts process their feature subspaces and the results are weighted by a mask — we move on to the second important phase: backpropagation of the error gradient. At this stage, we need to carefully propagate the error not only to the input data of each active model, but also to the activation mask itself, so that it can be adjusted during training.

To solve this task, we developed an OpenCL kernel, CalcHiddenGradientMaskMultWinConv, which handles all of this within the parallel computing paradigm.

__kernel void CalcHiddenGradientMaskMultWinConv(__global const float *matrix_w,
                                                __global const float *matrix_i,
                                                __global float *matrix_ig,
                                                __global const float *matrix_og,
                                                __global const float *masks,
                                                __global float *masks_g,
                                                const int outputs,
                                                const int window_in,
                                                const int window_out,
                                                const int activation
                                               )
  {
   const size_t u = get_global_id(0);
   const size_t w_in = get_global_id(1);
   const size_t v = get_global_id(2);
   const size_t units = get_global_size(0);
   const size_t windows_total = get_global_size(1);
   const size_t variables = get_global_size(2);

The kernel takes weights, input data, output-level error gradients, a mask, and buffers for writing the gradients of the input data and the mask. Each work-item in the OpenCL context is responsible for a separate combination of a token, an expert, and a variable. In the kernel body, we then immediately identify the work-item along all dimensions of the task space. Next, we determine the offsets in the data buffers.

const int shift_in = (u + v * units) * window_in * windows_total + w_in * window_in;
const int shift_out = u * window_out;
const int shift_out_var = v * units * window_out;
const int shift_mask = (u + v * units) * windows_total + w_in;
const int shift_weight = (v * window_out * windows_total + w_in * window_out) * (window_in + 1);

In the first step, we propagate the error gradient down to the input data level. Here, the process begins by checking the mask: if the corresponding expert was inactive for the given token, zero values are simply written to the input data gradient buffer, and computational resources are not wasted. If, however, the expert participated in the computation, gradient propagation begins.

const float m = IsNaNOrInf(masks[shift_mask], 0);
for(int i = 0; i < window_in; i++)
  {
   float sum = 0;
   if(m >= FLT_EPSILON)
     {
      for(int out = 0; out < window_out; out++)
        {
         if((shift_out + out) >= (outputs / variables))
            continue;
         sum += IsNaNOrInf(matrix_og[shift_out_var + shift_out + out] *
                           matrix_w[shift_weight + out * (window_in + 1) + i] *
                           m, 0);
         sum += IsNaNOrInf(matrix_w[shift_weight + out * (window_in + 1) + window_in] *
                           m, 0);
        }
     }
   matrix_ig[shift_in + i] = Deactivation(sum, matrix_i[shift_in + i], activation);
  }

First, the kernel iterates over all output channels, calculating the contribution of each element of the input data to the total error. The resulting values are adjusted by the derivative of the activation function of the input data layer and stored in the corresponding element of the global data buffer.

Next, the second step is performed — propagation of the error gradient to the mask. Here, we aggregate the contributions of all channels at the level of this expert’s results, taking into account the current expert’s impact on the outcome, and generate a feedback signal indicating how useful the choice of this expert turned out to be.

 float sum = 0;
 for(int out = 0; out < window_out; out++)
   {
    int shift_weight_loc = out * (window_in + 1) + shift_weight;
    float temp = matrix_w[shift_weight_loc + window_in];
    for(int i = 0; i < window_in; i++)
       temp += IsNaNOrInf(matrix_i[shift_in + i], 0) * matrix_w[shift_weight_loc + i];
    sum += IsNaNOrInf(temp * matrix_og[shift_out_var + shift_out + out], 0);
   }
 masks_g[shift_mask] = IsNaNOrInf(sum, 0);
}

Even if the mask is binary, this value helps the router make more informed choices in the future and, if necessary, can be interpreted as a weight score for weighted activation.

Thanks to this mechanism, we obtain a truly trainable architecture in which every routing decision can be adjusted through gradients, and dormant experts can awaken if they become needed. This enables the sparse mixture of experts to operate efficiently, flexibly, and truly intelligently.

The next critical step is to update the model parameters. This is where the model learns by adjusting its weights based on the error signal it receives. To implement this process, we use a separate OpenCL kernel, UpdateWeightsMaskMultWinConvAdam, which is tailored to the specifics of a sparse mixture of experts and supports optimization using the Adam method.

As before, we offload the bulk of the computations to the OpenCL context, since each weight involved in training can be processed independently.

__kernel void UpdateWeightsMaskMultWinConvAdam(__global float *matrix_w,
                                               __global const float *matrix_og,
                                               __global const float *matrix_i,
                                               __global const float *masks,
                                               __global float *matrix_m,
                                               __global float *matrix_v,
                                               const int windows_total,
                                               const int inputs,
                                               const int outputs,
                                               const float l,
                                               const float b1,
                                               const float b2
                                              )
  {
   const size_t id_in = get_global_id(0);
   const size_t id_out = get_global_id(1);
   const size_t id_v = get_global_id(2);
   const size_t window_in = get_global_size(0) / windows_total - 1;
   const size_t window_out = get_global_size(1);
   const size_t variables = get_global_size(2);

The kernel's operation is organized along three coordinates: id_in corresponds to the input dimension and position within the window, id_out corresponds to the output filter index, and id_v corresponds to the current variable in the batch. This three-dimensional organization fully covers all parameters of the experts and allows them to be processed in parallel.

It is important to note here that we deliberately use a structural assumption: all experts operate on identical analysis windows, and the result tensors have the same shape. This means that the entire system can be reduced to a two-dimensional matrix: one axis represents the number of filters, and the other represents the total number of elements in the analysis windows, combined across all experts. This simplification makes it possible to address weights in memory efficiently and compactly, which is especially critical during bulk parameter updates.

Within the kernel, we first identify each computational thread across all dimensions of the task space. Next, we determine the offset in all data buffers for the elements under analysis.

const int w_id = id_in / (window_in + 1);
const int shift_in = id_in - w_id;
const int step_in = window_in * windows_total;
const int units = outputs / window_out;
const int shift_in_var = id_v * inputs;
const int shift_out_var = id_v * outputs;
const int shift_mask_var = id_v * units * windows_total;
const int shift_weight = ((id_v * windows_total + w_id) * window_out + id_out) *
                         (window_in + 1) + id_in % (window_in + 1);
const bool bias = (id_in % (window_in + 1) == window_in);

Next, we iterate sequentially over all the tokens (units) that the experts worked with. For each such fragment, we use a mask to check whether the corresponding expert was active.

float grad = 0;
for(int u = 0; u < units; u++)
  {
   const int shift_in_loc = shift_in + u * step_in;
   if(shift_in < inputs)
      continue;
   float m = IsNaNOrInf(masks[shift_mask_var + u * windows_total + w_id], 0);
   if(m < FLT_EPSILON)
      continue;
   float inp = (bias ? 1 : IsNaNOrInf(matrix_i[shift_in_var + shift_in_loc], 0));
   grad += IsNaNOrInf(inp * m * matrix_og[shift_out_var + u * window_out + id_out], 0);
  }

If an expert was in dormant mode at a given step (the mask is close to zero), we do not spend resources computing the error gradient and move on to the next token. This is one of the key properties of sparse learning.

For active experts, we compute the error gradient: the input data are multiplied by the error value at the level of the output tensor (matrix_og) and by the mask.

float mt = IsNaNOrInf(clamp(b1 * matrix_m[shift_weight] + (1 - b1) * grad, -1.0e5f, 1.0e5f), 0);
float vt = IsNaNOrInf(clamp(b2 * matrix_v[shift_weight] + (1 - b2) * pow(grad, 2), 1.0e-6f, 1.0e6f), 1.0e-6f);
float weight = clamp(matrix_w[shift_weight] + IsNaNOrInf(l * mt / sqrt(vt), 0), -MAX_WEIGHT, MAX_WEIGHT);

Next, we apply an Adam optimization step: we update the first-order (mt) and second-order (vt) moments, normalize the gradient, and adjust the weight value. All updates pass through the clamp function, which limits the weight values to the permissible range.

We store the resulting values in global buffers.

 matrix_w[shift_weight] = weight;
 matrix_m[shift_weight] = mt;
 matrix_v[shift_weight] = vt;
}

Thus, each weight is updated strictly in accordance with its contribution to the final result — and only if it participated in the actual computations. This type of organization ensures high computational efficiency and also makes it possible to train highly specialized experts who not only remain dormant at unsuitable moments but also actively learn when their knowledge is truly needed.

Now that the computational logic on the OpenCL side has been fully implemented, we move on to the next step — organizing the entire process within the main program. This is where the object responsible for calling the appropriate kernels and working with masks is created and configured. This role is handled by the specialized class CNeuronMaskMultiWinConv, which inherits from CNeuronConvOCL. The structure of the new object is shown below.

class CNeuronMaskMultiWinConv    :  public CNeuronConvOCL
  {
protected:
   uint              iWindowsTotal;
   //---
   virtual bool      feedForward(CNeuronBaseOCL *NeuronOCL) override { return false; }
   virtual bool      feedForward(CNeuronBaseOCL *NeuronOCL, CBufferFloat *SecondInput) override;
   virtual bool      updateInputWeights(CNeuronBaseOCL *NeuronOCL) override { return false; }
   virtual bool      updateInputWeights(CNeuronBaseOCL *NeuronOCL, CBufferFloat *second)override;
   virtual bool      calcInputGradients(CNeuronBaseOCL *NeuronOCL) override { return false; }
   virtual bool      calcInputGradients(CNeuronBaseOCL *NeuronOCL, CBufferFloat *SecondInput,
                       CBufferFloat *SecondGradient, ENUM_ACTIVATION SecondActivation = None) override;

public:
                     CNeuronMaskMultiWinConv(void) {};
                    ~CNeuronMaskMultiWinConv(void) {};
   //---
   virtual bool      Init(uint numOutputs, uint myIndex, COpenCLMy *open_cl, uint window,
                          uint windows_total, uint window_out, uint units_count, uint variables,
                          ENUM_OPTIMIZATION optimization_type, uint batch);
   //---
   virtual int       Type(void) override const  {  return defNeuronMaskMultiWinConv;   }
   //--- methods for working with files
   virtual bool      Save(int const file_handle) override;
   virtual bool      Load(int const file_handle) override;
  };

It becomes the link between the model and the low-level computational pipeline. Its task is to properly prepare the data, pass it to the OpenCL context, launch the required kernel, and retrieve the results.

In the structure of the CNeuronMaskMultiWinConv object, we deliberately avoid declaring new internal components. Everything needed is already provided by the parent class CNeuronConvOCL, and this is quite sufficient for organizing computations on the OpenCL side. This approach keeps the architecture clean and free of redundancy, while ensuring that all resources are managed centrally.

The layer is initialized in the overridden Init method, where we adjust only the key parameters without affecting the overall logic.

bool CNeuronMaskMultiWinConv::Init(uint numOutputs, uint myIndex, COpenCLMy *open_cl, uint window,
                                   uint windows_total, uint window_out, uint units_count, uint variables,
                                   ENUM_OPTIMIZATION optimization_type, uint batch)
  {
   uint win = window * windows_total + MathMax(windows_total, 1) - 1;
   if(!CNeuronConvOCL::Init(numOutputs, myIndex, open_cl, win, win, window_out, units_count, variables, optimization_type, batch))
      return false;
   iWindowsTotal = windows_total;
   iWindow = window;
//---
   return true;
  }

Specifically, we set the value of iWindowsTotal, which determines the number of parallel experts. This value will be used when generating offsets and calculating addresses within the kernels.

The main focus of the initialization method is on correctly calculating the width of the window being processed. Since each expert operates on its own fixed-length window, we combine them into a single input space. As a result, the size of the combined analysis window over the source data is formed. This calculation ensures that even in boundary cases, the window size will not drop to zero.

Next, we delegate the execution to the method of the same name in the parent class, passing the updated parameters, and then return from the method.

In the forward and backward pass methods of this layer, no independent computational logic is implemented; instead, the layer merely services the OpenCL kernels described above. Each of these methods is responsible for preparing arguments, passing parameters, and queuing the corresponding kernel for execution.

The procedure in this case is standard: we pass pointers to the data buffers, including masks, and other parameters, and then call the OpenCL function with the required work dimensions. I think you are already quite familiar with this procedure. Therefore, an in-depth analysis of each method is not necessary here, and we suggest leaving these methods for independent study. The complete source code for the CNeuronMaskMultiWinConv class and all of its methods is provided in the attachment.

Sparse Mixture of Experts

The next important stage in our work is to build a sparse mixture of experts module. In our implementation, its architecture is defined as the CNeuronTimeMoESparseExperts class, which inherits from the base class CNeuronBaseOCL. The object includes all the key components needed to launch and coordinate the operation of the specialized and shared experts.

class CNeuronTimeMoESparseExperts   :  public CNeuronBaseOCL
  {
protected:
   CNeuronSwiGLUOCL        cExpertsIn;
   CNeuronSwiGLUOCL        cSharedIn;
   CNeuronMaskMultiWinConv cExpertsOut;
   CNeuronConvOCL          cSharedOut;
   CNeuronTopKGates        cMasks;
   CNeuronConvOCL          cSharedGates;
   //---
   virtual bool      feedForward(CNeuronBaseOCL *NeuronOCL)          override;
   virtual bool      updateInputWeights(CNeuronBaseOCL *NeuronOCL)   override;
   virtual bool      calcInputGradients(CNeuronBaseOCL *NeuronOCL)   override;

public:
                     CNeuronTimeMoESparseExperts(void) {};
                    ~CNeuronTimeMoESparseExperts(void) {};
   //---
   virtual bool      Init(uint numOutputs, uint myIndex, COpenCLMy *open_cl, uint window,
                          uint window_out, uint units_count, uint variables, uint experts,
                          uint topK, ENUM_OPTIMIZATION optimization_type, uint batch);
   //---
   virtual int       Type(void) override const {   return defNeuronTimeMoESparseExperts; }
   //--- methods for working with files
   virtual bool      Save(int const file_handle)   override;
   virtual bool      Load(int const file_handle)   override;
   //---
   virtual void      SetOpenCL(COpenCLMy *obj)     override;
   //---
   virtual bool      WeightsUpdate(CNeuronBaseOCL *source, float tau) override;
   virtual void      TrainMode(bool flag) override;
  };

In the structure of the CNeuronTimeMoESparseExperts class, we can identify several internal objects, each of which performs a strictly defined function within the sparse mixture of experts mechanism. We will gradually become familiar with their internal logic as we build the algorithms for how these methods work. At this stage, it is important to note the following architectural decision: all internal objects are declared statically, without dynamic memory allocation. This approach not only simplifies resource management, but also allows the class constructor and destructor to remain empty. All initialization processes are implemented in the Init method.

bool CNeuronTimeMoESparseExperts::Init(uint numOutputs, uint myIndex, COpenCLMy *open_cl, uint window,
                                       uint window_out, uint units_count, uint variables, uint experts,
                                       uint topK, ENUM_OPTIMIZATION optimization_type, uint batch)
  {
   if(!CNeuronBaseOCL::Init(numOutputs, myIndex, open_cl, window * units_count * variables,
                                                                 optimization_type, batch))
      return false;
   SetActivationFunction(None);

The method parameters provide a set of key constants that lets us unambiguously define the architecture of the object being created. Everything is specified here: the sizes of the input and output tokens, the total number of experts and the number selected by the Top-K mask, the type of optimization used, and the number of variables and computational blocks.

Some of these parameters are passed on to the method of the same name in the parent class. That is where basic validation of the architectural settings and initialization of global interfaces, including the OpenCL context, have already been implemented. In this way, we build a clear hierarchical structure in which each level is responsible for its own scope of responsibility, and all elements of the architecture come together to form a single, cohesive whole.

After successfully completing initialization at the parent class level, we move on to configuring the internal components — this is where the structure of the mixture of experts itself begins to take shape.

The first object in this chain is cExpertsIn, which implements the SwiGLU functionality. It serves as the first stage of input data processing — a kind of preliminary filter that amplifies useful signals and generates an embedding that is sent to each expert. The number of filters in this layer scales proportionally to the number of experts, allowing each of them to obtain a sufficiently expressive representation of the task.

int index = 0;
if(!cExpertsIn.Init(0, index, OpenCL, window, window, window_out * experts, units_count,
                                                       variables, optimization, iBatch))
   return false;
index++;
if(!cSharedIn.Init(0, index, OpenCL, window, window, window_out, units_count, variables,
                                                                  optimization, iBatch))
   return false;

In the same way, we initialize the cSharedIn component, which serves as the first processing layer for the shared expert. Unlike specialized experts, there is no need to scale the dimensionality of the filters here — on the contrary, it is fixed at a level equivalent to that of a single expert. This is because the shared expert covers the entire task space, and its role is not to compete with the others but to offer a universal perspective — a kind of baseline for decision-making.

Using SwiGLU in this role provides nonlinear selectivity, amplifying the distinction between tokens that are potentially relevant to a particular expert.

Next, we move on to forming the second processing layer for both the specialized experts and the shared expert. Here, the architecture strictly follows the logic established earlier: each block performs its specific function within the overall sparse mixture-of-experts mechanism.

For the specialized mixture of experts, we use the previously created CNeuronMaskMultiWinConv module, which implements convolution with masked windows. This component filters the analyzed features based on each expert's individual mask, enabling targeted activation and a clear division of responsibilities among experts.

index++;
if(!cExpertsOut.Init(0, index, OpenCL, window_out, experts, window, units_count,
                                               variables, optimization, iBatch))
   return false;
cExpertsOut.SetActivationFunction(None);
index++;
if(!cSharedOut.Init(0, index, OpenCL, window_out, window_out, window, units_count,

                                                 variables, optimization, iBatch))
   return false;
cSharedOut.SetActivationFunction(None);

For the shared expert, in turn, we use the standard CNeuronConvOCL convolution layer.

As the object that creates the active-expert mask, we use the CNeuronTopKGates module, which we already know from the DUET framework. This component plays a key role in the sparsification mechanism: it analyzes the input tokens and selects a strictly limited number of the most relevant experts.

In other words, CNeuronTopKGates generates an activation mask according to the Top-K principle, strictly zeroing out irrelevant channels. This approach significantly reduces the computational load while maintaining the model's high expressiveness. The number of active experts (topK) is specified by a parameter and can be easily adapted to the specifics of a particular task.

index++;
if(!cMasks.Init(0, index, OpenCL, window, units_count, experts, topK, optimization, iBatch))
   return false;

It is important to understand that the CNeuronTopKGates object does not simply return a binary mask. It also generates a normalized probability distribution over the selected channels. This allows for flexible adjustment of each expert's contribution to the final result. Active experts contribute to the calculations with varying degrees of confidence, and these probabilities are explicitly reflected in the weights of the final mask.

Thanks to this approach, the architecture gains not only compactness and computational efficiency, but also robustness against overfitting. Each expert is used strictly when appropriate, rather than at random, which is particularly important when working with noisy or multimodal time series.

The object's structure is completed by the shared expert gates module — cSharedGates. Its purpose is to determine the extent to which the shared expert contributes to the model's final output. To do this, we use a standard convolutional layer configured to extract spatiotemporal patterns. Unlike the mask used for individual experts, a sigmoid activation function is used here, which produces smooth values in the range from 0 to 1.

   index++;
   if(!cSharedGates.Init(0, index, OpenCL, window, window, window, units_count, variables,
                                                                    optimization, iBatch))
      return false;
   cSharedGates.SetActivationFunction(SIGMOID);
//---
   return true;
  }

This approach does not produce a binary on/off decision, but instead enables flexible scaling of the shared expert's contribution depending on the current context. The sigmoid function acts as a flexible damper: if confidence in the general pattern is high, the value approaches one; if the signal is weak, the corresponding mask suppresses the output. All of this makes it possible to precisely control the interaction between the localized and generalizing components of the architecture.

After all iterations have been completed successfully, we return the resulting Boolean status to the calling program and exit the method.

We then move on to the next key stage — organizing the forward pass in the feedForward method. This is where the operating logic of the sparse mixture of experts module comes fully into view.

bool CNeuronTimeMoESparseExperts::feedForward(CNeuronBaseOCL *NeuronOCL)
  {
   if(!cExpertsIn.FeedForward(NeuronOCL))
      return false;

First, we run a forward pass through the cExpertsIn layer, which transforms the signal being analyzed into the expert space by scaling it according to the number of specialized processing paths. In parallel, the cSharedIn layer is started; this layer represents the first processing level of a more versatile shared expert.

if(!cSharedIn.FeedForward(NeuronOCL))
   return false;

Next, two routing branches are activated. The cSharedGates layer calculates the shared expert's participation mask: it uses a sigmoid activation function to generate a graded significance mask. In turn, the cMasks object is a specialized module that selects the most relevant experts to form a discrete Top-K mask. It not only excludes inactive channels, but also returns a normalized distribution of weights across the selected paths.

if(!cSharedGates.FeedForward(NeuronOCL))
   return false;
if(!cMasks.FeedForward(NeuronOCL))
   return false;

At this stage, result generation begins. The cExpertsOut module is used to run a convolution over the masked expert block: each active filter has its own weight, and all processing is organized based on the previously prepared mask.

if(!cExpertsOut.FeedForward(cExpertsIn.AsObject(), cMasks.getOutput()))
   return false;

For the shared expert, a standard convolution is performed using cSharedOut, after which its output is further scaled according to a significance mask previously calculated by cSharedGates. Scaling is performed through element-wise multiplication.

if(!cSharedOut.FeedForward(cSharedIn.AsObject()))
   return false;
if(!ElementMult(cSharedOut.getOutput(), cSharedGates.getOutput(), cSharedOut.getPrevOutput()))
   return false;

After obtaining the output of the sparse mixture of experts and the output of the shared expert scaled by the significance mask, the two information streams are combined. At this stage, the data are summed and written to an intermediate data buffer.

   const int window = (int)cSharedGates.GetWindow();
   if(!SumAndNormilize(cExpertsOut.getOutput(), cSharedOut.getPrevOutput(), PrevOutput,
                       window, false, 0, 0, 0, 1))
      return false;
   if(!SumAndNormilize(NeuronOCL.getOutput(), PrevOutput, Output,
                       window, true, 0, 0, 0, 1))
      return false;
//---
   return true;
  }

However, this is only the first part of the final stage of the forward pass. Next, residual connections — the stored outputs of the previous layer of the neural architecture — are added to the obtained values. This technique preserves important information from the input signal and improves the model’s convergence by stabilizing the gradients.

After all components have been combined, the results are normalized within each token (along the analysis window). This operation brings the data to a consistent scale, ensuring correct interpretation of the output tensor. The result is stored in the global results interface buffer.

As you can see, the forward-pass algorithm in our module has a highly branched structure. The input data are fed simultaneously into five different information streams, which significantly increases the model’s flexibility and adaptability. However, such a multi-channel architecture introduces certain challenges during the error backpropagation phase.

In the calcInputGradients method, error gradients are carefully distributed among all internal components according to their contribution to the final result. This resembles masterful conducting, where each instrument plays in perfect harmony with the orchestra.

bool CNeuronTimeMoESparseExperts::calcInputGradients(CNeuronBaseOCL *NeuronOCL)
  {
   if(!NeuronOCL)
      return false;

First, the procedures for distributing the error gradient across individual information streams are activated:

• for the output of the sparse mixture of experts layer (cExpertsOut), where activation-function derivatives are applied to the gradients;
• for the shared expert and gates (cSharedOut and cSharedGates), where the interaction between output values and masks is taken into account, and gradient products are applied with regard to the activation functions.

if(!DeActivation(cExpertsOut.getOutput(), cExpertsOut.getGradient(), Gradient,
                                                    cExpertsOut.Activation()))
   return false;
if(!ElementMultGrad(cSharedOut.getOutput(), cSharedOut.getGradient(),
                    cSharedGates.getOutput(), cSharedGates.getGradient(),
                    Gradient, cSharedOut.Activation(), cSharedGates.Activation()))
   return false;

This is followed by a recursive call to calculate hidden gradients in the internal objects — this makes it possible to unroll the contribution of each component step by step, starting from the first expert layers (cExpertsIn, cSharedIn) all the way down to the raw data level, ensuring a thorough and correct calculation.

   if(!cExpertsIn.calcHiddenGradients(cExpertsOut.AsObject(), cMasks.getOutput(),
                                      cMasks.getGradient(), (ENUM_ACTIVATION)cMasks.Activation()))
      return false;
   if(!cSharedIn.calcHiddenGradients(cSharedOut.AsObject()))
      return false;
//---
   if(!NeuronOCL.calcHiddenGradients(cExpertsIn.AsObject()))
      return false;

Particular attention is paid to the accumulation of gradients in intermediate buffers. Here, gradients are summed over the tokens in the window, ensuring proper alignment of signals from the different components involved in the process.

   if(!DeActivation(NeuronOCL.getOutput(), PrevOutput, Gradient, NeuronOCL.Activation()))
      return false;
   const int window = (int)cSharedGates.GetWindow();
   if(!SumAndNormilize(PrevOutput, NeuronOCL.getGradient(), PrevOutput, window, false, 0, 0, 0, 1))
      return false;
   if(!NeuronOCL.calcHiddenGradients(cSharedIn.AsObject()))
      return false;
   if(!SumAndNormilize(PrevOutput, NeuronOCL.getGradient(), PrevOutput, window, false, 0, 0, 0, 1))
      return false;
   if(!NeuronOCL.calcHiddenGradients(cMasks.AsObject()))
      return false;
   if(!SumAndNormilize(PrevOutput, NeuronOCL.getGradient(), PrevOutput, window, false, 0, 0, 0, 1))
      return false;
   if(!NeuronOCL.calcHiddenGradients(cSharedGates.AsObject()))
      return false;
   if(!SumAndNormilize(PrevOutput, NeuronOCL.getGradient(), NeuronOCL.getGradient(), window, false,
                                                                                       0, 0, 0, 1))
      return false;
//---
   return true;
  }

Ultimately, the calcInputGradients method carefully balances the complex flow of information and ensures efficient training of the entire sparse mixture of experts module, making this architecture not only powerful but also robust to errors and overfitting.

The method for updating model parameters follows the classic approach: control is delegated to internal components. Each of them implements its own weight adaptation strategy based on the obtained gradients. In the body of the updateInputWeights method, the corresponding procedures are simply called sequentially. Therefore, to avoid repeating ourselves, we suggest that you study this fragment on your own. The complete source code for the CNeuronTimeMoESparseExperts class, including implementations of all methods, is available in the attachment.

Conclusion

In this article, we have examined in detail the implementation of the masked convolution module and the architecture of the sparse mixture of experts block, adapted for time series processing tasks in the OpenCL environment. The key aspects of configuring device-side computations and organizing interaction with the main program were examined. Special attention was paid to the correct distribution of the gradient in the context of a branched information flow.

In the next part, we will continue developing the architecture and focus on building and training models that use this structure as part of more complex neural systems.

References

• Time-MoE: Billion-Scale Time Series Foundation Models with Mixture of Experts
• Other articles in this series

Programs used in the article

#	Name	Type	Description
1	Study.mq5	Expert Advisor	Expert Advisor for offline model training
2	StudyOnline.mq5	Expert Advisor	Expert Advisor for online model training
3	Test.mq5	Expert Advisor	Expert Advisor for model testing
4	Trajectory.mqh	Class Library	Structure for describing the system state and model architecture
5	NeuroNet.mqh	Class Library	Class library for building neural networks
6	NeuroNet.cl	Library	Code library for the OpenCL program