Recurrent neural networks

We have already discussed the multilayer perceptron and convolutional neural networks earlier. All of them operate with static data within the framework of Markov processes, where the subsequent state of the system depends only on its current state and is independent of the system's past states. Indeed, to compensate for this limitation, we fed the neural network not only the latest price data and indicator states but also historical data for the past few bars. However, the network itself did not memorize the processed data and the obtained results. During each new forward pass iteration into the neural network, we re-input the complete set of historical data, even if we had previously provided this data to the neural network. Essentially, with each forward pass, the neural network started with a "clean slate." The only memory such a neural network possesses is the weight matrix it learned during the training process.

Another drawback of using such neural networks for time series is that the position of a specific value/pattern in the data array has absolutely no impact on the outcome values. Let me remind you of the mathematical model of a neuron.

Note that the sum of values is at the center of the entire model. Permuting the terms does not change the sum, so from a mathematical perspective, it is absolutely irrelevant whether a value appears at the beginning or at the end of the array. In practical usage of time series, however, it is quite common for the latest values to have a greater impact on the outcome compared to older values.

Now I suggest looking at Recurrent Neural Networks. They represent a special type of neural network designed to work with time sequences. The key feature of recurrent neurons is the transmission of their own state as input to themselves in the next iteration.

With each new input from the external environment, the neuron, along with the new data, essentially evaluates the outcome of its past performance as if reviewing its own history.