Discussing the article: "Neural networks made easy (Part 47): Continuous action space"

[MetaQuotes]

Check out the new article: Neural networks made easy (Part 47): Continuous action space.

In this article, we expand the range of tasks of our agent. The training process will include some aspects of money and risk management, which are an integral part of any trading strategy.

In our previous article, we trained the agent only to determine the trading direction. The Agent's range of actions was limited to only 4 options:

• buy, 
• sell, 
• hold/wait,
• close all positions.

Here we do not see capital and risk management functions. We used the minimum lot in all trading operations. This is enough to evaluate training approaches, but not enough to build a trading strategy. A profitable trading strategy simply must have a money management algorithm.

In addition, to create a stable trading strategy, we need to manage risks. This block is also missing from our designs. The EA assessed the market situation at each new trading candle and made a decision on a trading operation. But every upcoming bar carries risks for our account. Price movement within a bar can be detrimental to our balance. This is why it is always recommended to use stop losses. This simple approach allows us to limit risks per trade.

After about 3000 passes, I was able to get a model that was able to generate profit on the training set. During the training period of 5 months, the model made 334 transactions. More than 84% of them were profitable. The result was the profit of 33% of the initial capital. At the same time, the drawdown on balances was less than 1%, and by Equity - 7.6%. The profit factor exceeded 26 and the recovery factor amounted to 3.16. The graph below shows an upward trend in the balance. The balance line is almost always below the Equity line, which indicates that positions are being opened in the right direction. At the same time, the load on the deposit is about 20%. This is a fairly high figure, but does not exceed the accumulated profit.

Unfortunately, the results of the EA's work turned out to be more modest outside the training set.

Author: Dmitriy Gizlyk

[Viktor Kudriavtsev]

Hello. The Expert Advisor stops opening trades after 30-50 passes. Is it normal or something should be corrected? I made 5-7 attempts with new model files. When it is a little more passes continues to open deals, and when it is a little less. But still stops opening trades. I tried to train one of the models in 4000 passes. The result is the same - a straight line.

[Dmitriy Gizlyk]

Viktor Kudriavtsev #:
Hello. The Expert Advisor stops opening trades after 30-50 passes. Is it normal or something should be corrected? I made 5-7 attempts with new model files. When it is a little more passes continues to open deals, and when it is a little less. But still stops opening trades. I tried to train one of the models in 4000 passes. The result is the same - a straight line.

Good day, Victor.

Training a model is a rather long process. The training coefficient in the library is set at 3.0e-4f. I.e. if you train the model only on 1 example, it will learn it in about 4000 iterations. Such a small learning rate is used so that the model can average the weights to maximise the fit to the training sample.

Regarding the lack of transactions, this is not a reason to stop the learning process. The process of training the model resembles a "trial and error" method. The model gradually tries all possible options and looks for a way to maximise the reward. When building the learning process, we added a penalty for no trades, which should stimulate the model to open positions. If at some pass the model does not make any trades, then after one or more repetitions the penalty should do its job and get the model out of this state. To speed up this process, you can increase the penalty for no trades. But you should be careful here, it should not exceed the possible loss from a trade. Otherwise, the model will open unprofitable positions to avoid the penalty for their absence.

[Too Chee Ng]

Dmitriy Gizlyk Many thanks for the detailed article. I retrained the model between 1st Jan 2023 and 31st May 2023 to produced DDPGAct.nnw and DDPGCrt.nnw. However, when testing the EA with test.ex5, there wan't a single trade.

I have taken the following steps:

1. Download, unzip and compile Study.mq5 and test.mq5 from https://www.mql5.com/en/articles/download/12853.zip
2. In Strategy Tester, run Study.ex5 once as indicated in https://c.mql5.com/2/55/Study.png
3. In Strategy Tester, Optimize Study.ex5 as indicated in https://c.mql5.com/2/55/Study_opt.png; ref. to the attached Optimized.png
   
4. In Strategy Tester, run test.ex5 (Optimization: Disabled) for the same period between 1st Jan 2023 and 31st May 2023
5. (No error and no trade at all!)

Trying to debug with the following PrintFormat lines:

...
   double sell_sl = NormalizeDouble(Symb.Bid() + ActorResult[5], Symb.Digits());
   PrintFormat("ActorResult[0]=%f  ActorResult[1]=%f ActorResult[2]=%f buy_sl=%f",ActorResult[0], ActorResult[1],  ActorResult[2], buy_sl);
   PrintFormat("ActorResult[3]=%f  ActorResult[4]=%f ActorResult[5]=%f sell_tp=%f",ActorResult[0], ActorResult[1],  ActorResult[2], sell_tp);
//---
   if(ActorResult[0] > 0 && ActorResult[1] > 0 && ActorResult[2] > 0 && buy_sl > 0)
      Trade.Buy(buy_lot, Symb.Name(), Symb.Ask(), buy_sl, buy_tp);
   if(ActorResult[3] > 0 && ActorResult[4] > 0 && ActorResult[5] > 0 && sell_tp > 0)
      Trade.Sell(sell_lot, Symb.Name(), Symb.Bid(), sell_sl, sell_tp);
...

reviewed the following:

...
2023.12.01 23:15:18.641	Core 01	2023.05.30 19:00:00   ActorResult[0]=0.085580  ActorResult[1]=-0.000476 ActorResult[2]=-0.000742 buy_sl=1.072910
2023.12.01 23:15:18.641	Core 01	2023.05.30 19:00:00   ActorResult[3]=0.085580  ActorResult[4]=-0.000476 ActorResult[5]=-0.000742 sell_tp=1.070290
2023.12.01 23:15:18.641	Core 01	2023.05.30 20:00:00   ActorResult[0]=0.085580  ActorResult[1]=-0.000476 ActorResult[2]=-0.000742 buy_sl=1.072830
2023.12.01 23:15:18.641	Core 01	2023.05.30 20:00:00   ActorResult[3]=0.085580  ActorResult[4]=-0.000476 ActorResult[5]=-0.000742 sell_tp=1.070210
2023.12.01 23:15:18.641	Core 01	2023.05.30 21:00:00   ActorResult[0]=0.085580  ActorResult[1]=-0.000476 ActorResult[2]=-0.000742 buy_sl=1.072450
2023.12.01 23:15:18.641	Core 01	2023.05.30 21:00:00   ActorResult[3]=0.085580  ActorResult[4]=-0.000476 ActorResult[5]=-0.000742 sell_tp=1.069830
2023.12.01 23:15:18.641	Core 01	2023.05.30 22:00:00   ActorResult[0]=0.085580  ActorResult[1]=-0.000476 ActorResult[2]=-0.000742 buy_sl=1.072710
2023.12.01 23:15:18.641	Core 01	2023.05.30 22:00:00   ActorResult[3]=0.085580  ActorResult[4]=-0.000476 ActorResult[5]=-0.000742 sell_tp=1.070090
2023.12.01 23:15:18.641	Core 01	2023.05.30 23:00:00   ActorResult[0]=0.085580  ActorResult[1]=-0.000476 ActorResult[2]=-0.000742 buy_sl=1.073750
2023.12.01 23:15:18.641	Core 01	2023.05.30 23:00:00   ActorResult[3]=0.085580  ActorResult[4]=-0.000476 ActorResult[5]=-0.000742 sell_tp=1.071130
...

May I know what has gone wrong or missed please?

Many thanks.

[star-ik]

Hello. It seems to me that it is not that the model does not open trades, but that it is not set up to make profit. The straight line on the screenshot says exactly that. Something needs to be done in the reward rules.

float reward = (account[0] - PrevBalance) / PrevBalance;

if(account[0] == PrevBalance)

if((buy_value + sell_value) == 0)

reward -= 1;

I tried these variants

float reward = (account[0] - PrevBalance) / PrevBalance;

if(account[0] == PrevBalance)

if((buy_value + sell_value) == 0)

reward -= 1;

if(buy_profit<10)

reward -= 1;

if(buy_profit>10)

reward += 1;

if(sell_profit<10)

reward -= 1;

if(sell_profit>10)

reward += 1;

Doesn't help. Please tell me what to do.