Discussing the article: "Neural Networks in Practice: ReLU Function Graph"

[MetaQuotes]

Check out the new article: Neural Networks in Practice: ReLU Function Graph.

We will create an indicator written entirely in MQL5 to display the activation function and its derivative on the chart, using ReLU as an example. We will explain how derivatives affect the choice of activation function and what precautions are needed at nondifferentiable points. Readers will be able to interactively visualize the curves and gain a practical foundation for deciding when to use certain activation functions and when not to.

In the previous article, “Neural Networks in Practice: The Birth of C_Neuron” we began to take a slightly more organized approach. Let's start by creating a class designed to store everything we'll need to include in the neuron. This will allow us to connect several neurons in the future within an architecture essentially intended to build a small, simple neural network. Nothing on the level of ChatGPT or anything like that. The goal here is to show neural network enthusiasts how neural networks work internally. Without using any additional resources. By programming everything ourselves.

So, our first basic neuron is already capable of doing certain things, since it converges in many simple situations in which we can test it. It already allows us to change the number of inputs extremely easily and quickly, which makes it much more interesting in terms of its potential applications. However, all of this is still in the very early stages. Its model does not yet include an activation function or a structure that would allow us to arrange neurons in a cascade, thereby forming a sequence of neurons and, consequently, a small neural network.

In this article—and probably in further articles as well—I'll explain exactly what happens inside an activation function. The idea is for you, dear reader, to understand when, why, and how to use and choose the most appropriate activation function. Every case is unique. There is no one-size-fits-all solution or anything that you should always do or use. However, to make the best decisions, it is necessary to understand how each activation function works and what effect it has on the neural network as a whole.

Author: Daniel Jose