You are missing trading opportunities:
- Free trading apps
- Over 8,000 signals for copying
- Economic news for exploring financial markets
Registration
Log in
You agree to website policy and terms of use
If you do not have an account, please register
Check out the new article: Neural Networks in Practice: The Gradient.
I've already touched on the topic of the gradient in another article. However, I believe that certain aspects were explained very poorly there. Perhaps that is precisely why so many people are confused about this issue. I'll explain this much more simply here, because we'll soon be moving on to a much more complex topic. For now, we won't be adding activation functions to the C_Neuron class. That's because I don't want to complicate things prematurely and unnecessarily.
So, the gradient is a much simpler and more convenient way to represent the derivative. Or, more precisely, a way to actually use the tangent line when we're working with multiple variables. Thus, in many cases, we can use a more efficient computation method to achieve convergence.
First and foremost, it is important to understand that any gradient is, in essence, a vector. So what does that mean in practice? The first implication is that, as a vector, it points in a specific direction. In this case, it points in the direction of steepest increase. However, if we use it in the opposite direction, we can move toward the smallest increase or, in our case, the lowest loss. That is precisely why, to distinguish this direction from others, we call it gradient descent.
The second point—and this is where many people get confused—is that the gradient is an orthogonal vector. For this reason, it is literally perpendicular to another line passing through the vector's origin. It is precisely this line that is the tangent line. Please keep this in mind, because it will be important later. This differs from what happens in the least squares method, where a secant line is actually constructed. Many people mistakenly claim that a perceptron uses a tangent line. However, if we analyze the calculations being performed, it becomes clear that we are actually dealing with a secant line. Therefore, be careful when specifying which type of line you are using in trigonometric calculations. There are other aspects related to vectors, but for the purposes of these articles, what we've just mentioned is more than enough to explain everything else.
Therefore, since the gradient is a vector, it is much more convenient to work with it in many situations. Here, however, we will focus on the most basic aspect—at least for now. Otherwise, we'll start discussing topics unrelated to the main issue.
Author: Daniel Jose