Discussing the article: "Neural networks made easy (Part 57): Stochastic Marginal Actor-Critic (SMAC)" - page 2

[MrBrooklin]

JimReaper #:

//--- layer 5

if (!(descr = new CLayerDescription())) return false;

descr.type = defNeuronConvOCL;

prev_count = descr.count = prev_count - 1;

descr.window = 3;

descr.step = 1;

descr.window_out = 8;

descr.activation = LReLU;

descr.optimisation = ADAM;

if(!actor.Add(descr))

{

delete descr;

return false;

}

To insert the code you need to apply the corresponding button

Regards, Vladimir.

[Viktor Kudriavtsev]

If you follow the scheme proposed by JimReaper or simply add more indicators or increase the depth of HistoryBars 50 or 100, then when creating a database of examples in the tester by the Research Expert Advisor, fewer examples are saved for one pass of optimisation. The MinProfit parameter is set to -10000 and has no effect. For example, I set 50 passes, but only 41 passes get into the database. Moreover, the more I try to capture the depth of HistoryBars history, the fewer passes are saved in the database. And when collecting additional passes during training iterations, with each pass less and less passes are saved in the database. I also noticed that the size of the *.bd file cannot exceed 3 GB. As if something bumps on the size. Can you tell me how to overcome this?

[Tran Ngoc Dung]

Can someone help me understand how to use the code in the article for testing and demo trading.  Appreciate everyone's help!

[Chris]

Every pass of the Test EA generates drastically different results as if the modell were different from all previous ones. It is obvious that the model evolves every single pass of Test but the behaviour of this EA is hardly an evolution, so what stands behind it?

Here are some pictures:

[Chris]

Buy and sell transactions seem to be insufficiently controlled in the Test and possibly Research scripts. Here are some messages:

2024.04.27 13:40:29.423 Core 01 2024.04.22 18:30:00   current account state: Balance: 9892.14, Credit: 0.00, Commission: 0.00, Accumulated: 0.00, Assets: 0.00, Liabilities: 0.00, Equity 9892.14, Margin: 0.00, FreeMargin: 9892.14

2024.04.27 13:40:29.423 Core 01 2024.04.22 18:30:00   calculated account state: Assets: 0.00, Liabilities: 0.00, Equity 9892.14, Margin: 11359.47, FreeMargin: -1467.33

2024.04.27 13:40:29.423 Core 01 2024.04.22 18:30:00   not enough money [market buy 0.96 EURUSD.pro sl: 1.06306 tp: 1.08465]

2024.04.27 13:40:29.423 Core 01 2024.04.22 18:30:00   failed market buy 0.96 EURUSD.pro sl: 1.06306 tp: 1.08465 [No money]

Unless margin overruns are intended, simple limits put on buy_lot after line 275 and after line 296 put on sell_lot would eliminate this behaviour of the Test script.

[Dmitriy Gizlyk]

Chris #:

Every pass of the Test EA generates drastically different results as if the modell were different from all previous ones. It is obvious that the model evolves every single pass of Test but the behaviour of this EA is hardly an evolution, so what stands behind it?

Here are some pictures:

This model use stochastic politic of Actor. So in the beginning of study we can see random deals at every pass. We collect this passes and restart study of the model. And repeat this process some times. While Actor find good politic of actions.

[Chris]

Let's put the question another way. Having collected (Research) samples and processed them (Study) we run the Test script. In several conscutive runs, without any Research or Study, the results obtained are completely different. 

Test script loads a trained model in OnInit subroutine (line 99). Here we feed the EA with a model which should not change during Test processing. It should be stable, as far as I understand. Then, final results should not change.

In the meantime, we do not conduct any model training. Only collecting more samples is performed by the Test.

Randomness is rather observed in the Research module and possibly in the Study while optimizing a policy.

Actor is invoked in line 240 in order to calculate feedforward results. If it isn't randomly initialized at the creation moment, I believe this is the case, it should not behave randomly.

Do you find any misconception in the reasoning above? 

[Dmitriy Gizlyk]

Chris #:

Let's put the question another way. Having collected (Research) samples and processed them (Study) we run the Test script. In several conscutive runs, without any Research or Study, the results obtained are completely different. 

Test script loads a trained model in OnInit subroutine (line 99). Here we feed the EA with a model which should not change during Test processing. It should be stable, as far as I understand. Then, final results should not change.

In the meantime, we do not conduct any model training. Only collecting more samples is performed by the Test.

Randomness is rather observed in the Research module and possibly in the Study while optimizing a policy.

Actor is invoked in line 240 in order to calculate feedforward results. If it isn't randomly initialized at the creation moment, I believe this is the case, it should not behave randomly.

Do you find any misconception in the reasoning above? 

The Actor use stochastic policy. We implement it by VAE.

//--- layer 10
   if(!(descr = new CLayerDescription()))
      return false;
   descr.type = defNeuronBaseOCL;
   descr.count = 2 * NActions;
   descr.activation = SIGMOID;
   descr.optimization = ADAM;
   if(!actor.Add(descr))
     {
      delete descr;
      return false;
     }
//--- layer 11
   if(!(descr = new CLayerDescription()))
      return false;
   descr.type = defNeuronVAEOCL;
   descr.count = NActions;
   descr.optimization = ADAM;
   if(!actor.Add(descr))
     {
      delete descr;
      return false;
     }

Layer CNeuronVAEOCL use data of previous layer as mean and STD of Gaussian distribution and sample same action from this distribution. At start we put in model random weights. So it generate random means and STDs. At final we have random actions at every pass of model test. At time of study model will find some means for every state and STD tends to zero.    

[Chris]

Rather, the Test script provides insight into the capabilities of the rest of the algorithm. Since there is still a degree of freedom in the form of variable, unregistered initial weight values at the stage of creating VAE, it is not possible to establish and recreate the full, optimal model. Was that supposed to be the purpose of this script?

[Wen Feng Lin]

It's a great look.