Discussion of article "Neural networks made easy (Part 7): Adaptive optimization methods" - page 2

New comment
 

TempData.Add((double)((buy || sell) ? 0 : 1));//Undef

Can't Undefine as in the previous code write 0.5 instead of 0 to reduce the number of undefined?

 

Great and excellent work Dimitry! your effort on this one is immense.

and Thanks for sharing.

one small observation: 

I've tried the script, the backpropagation is executed before the feedforward.

My suggestion would be to feedforward first and then backpropagate correct result.

If the correct results are backpropagated after knowing what the network thinks , you might see reduction in missing fractals. up to 70% of the results could be refined.


also,

doing this : 

TempData.Add((double)((!buy && !sell) ? 0.5 : 0));

could potentially result in prematurely trained network. so, we should avoid this.

for the network learning,

we can start with the Adam optimizer and a learning rate of 0.001 and iterate it over the epochs.

(or)

to find a better Learning Rate, we can use LR Range Test (LRRT)


Lets say, If the defaults aren't working, the best method for finding a good learning rate is the  Learning Rate Range Test.

  1. Start with a very small learning rate (e.g., 1e-7).

  2. On each training batch, gradually increase the learning rate exponentially.

  3. Record the training loss at each step.

  4. Plot the loss vs. the learning rate.

  5. Look at the plot. The loss will go down, then flatten out, and then suddenly shoot upwards. (immediate next learning rate is the optimal after this upward shoot)

 we need the fastest learning rate where the loss is still consistently decreasing.

Thanks again

12
New comment