I am having a very unusual issue with my backtesting. I do not know if it has to do with the code or what.

 

Hello,

I have been running into this issue for awhile now. (I will have the image attached to this message). I have an extremely fast computer running thread ripper 64 core / 128 threads and many other specs. So, the issue is most definitely not the computer. I have been working with a freelance worker for a year now building various bots. We believe it is the code causing this issue, but i wanted to get on here and ask to see if anyone else had experienced this issue and has some solutions.

I will explain the issue as best as i can with the image. Basically, when you go to run an optimization using MT5 and go to your agents. You can see all the task assigned to each agent. It seems to only be assigning a set number of task to each agent, coming nowhere close to the amount of task it actually needs. Now I am unaware if the task are supposed to increase or start out at a specific number but something is defiantly wrong.

When the test makes it too optimization 509 out of 10,... whatever, it STOPS. When i take a look at the task menu it says it has completed almost all task but 3-4 cores are not completed and it is no where near the amount of test it says it will run. For example it tells me each core will be completing 12 task and 95% of them will say "12/12 task completed" then the other 5% will say "12/11 task completed"

It should also be noted i have paid money for the use of cloud servers temporarily to double check the speeds of my computer. Yet, when it hits test number 506-509, it stops and will not go further. 

It should also be noted. I have had instances where it will sometimes make jumps for a moment, but eventually, it will completely stop. Also, The image attached shows me using the cloud servers. When you use the cloud servers it will only assign 1 task too each core.

Please, if you have any incite at all. I would appreciate it very much. 

Distributed Computing in the MQL5 Cloud Network
Distributed Computing in the MQL5 Cloud Network
  • cloud.mql5.com
Connect to the MQL5 Cloud Network (Cloud Computing) and earn extra income around the clock — there is much work for you computer!
Files:
 

I am assuming that it is a "Fast genetic" optimisation. I suspect that it is indeed the code that is causing the issue, probably by entering into endless loop, and leaving the "genetic algorithm" waiting for the result in order to be able to continue.

The "genetic algorithm" will carry out a fixed number of tests, then await the results, before deciding how to proceed with the next batch of tests. So, if some of those initial tests hang, it can't make a decision for the next batch. There is probably a timeout period, but I don't know for how long.

Here are a few questions ...

  • At this stage of the 500+ passes, are there already any results displayed in the "Optimization Results" tab?
    Remember to click out of the tab and back in for it to update with newer results.
  • And how many results are there compared to the 500+ reported passes?
    Remember to enable all the filters so that all results are shown.
EDIT: Do the above without using the MQL5 Cloud network. There is no sense in using the cloud until it can work properly on your own computer.
 
Your topic has been moved to the section: Expert Advisors and Automated Trading
Please consider which section is most appropriate — https://www.mql5.com/en/forum/172166/page6#comment_49114893
 
Fernando Carreiro #:

I am assuming that it is a "Fast genetic" optimisation. I suspect that it is indeed the code that is causing the issue, probably by entering into endless loop, and leaving the "genetic algorithm" waiting for the result in order to be able to continue.

The "genetic algorithm" will carry out a fixed number of tests, then await the results, before deciding how to proceed with the next batch of tests. So, if some of those initial tests hang, it can't make a decision for the next batch. There is probably a timeout period, but I don't know for how long.

Here are a few questions ...

  • At this stage of the 500+ passes, are there already any results displayed in the "Optimization Results" tab?
    Remember to click out of the tab and back in for it to update with newer results.
  • And how many results are there compared to the 500+ reported passes?
    Remember to enable all the filters so that all results are shown.
EDIT: Do the above without using the MQL5 Cloud network. There is no sense in using the cloud until it can work properly on your own computer.

I am using slow complete algorithm.

Yes, there are results.

When i export it to excel it shows the correct number of backtest in relations to what MQL5 says. However, if i stop it then start again. I will export the result to an excel again, and even though it stopped at the same number of backtest. The excel will show an additional 500 lines of inputs it ran, and it will continue to add 500 lines everytime i start and stop. I even checked for duplicate settings and could find none.

 
john Praytor #: I am using slow complete algorithm. Yes, there are results. When i export it to excel it shows the correct number of backtest in relations to what MQL5 says. However, if i stop it then start again. I will export the result to an excel again, and even though it stopped at the same number of backtest. The excel will show an additional 500 lines of inputs it ran, and it will continue to add 500 lines everytime i start and stop. I even checked for duplicate settings and could find none.

For a "Slow Complete" optimisation, I don't know exactly how the passes are distributed between the various available threads, but I suspect that if any one pass freezes, then that thread will block, and if the same happens on the other threads, then eventually everything blocks.

I would suggest running small batches of optimisations, of a select few parameters, to narrow down which input combinations may be causing a "freeze"/"endless loop". Then once you have identified which inputs are involved, you can  then analyse and debug the code to fix it.

On a side note, it is also good to add the check of the _StopFlag condition, to all your loops, to help it break out of the endless loop, when the terminal instructs the EA to terminate execution.

Documentation on MQL5: Predefined Variables / _StopFlag
Documentation on MQL5: Predefined Variables / _StopFlag
  • www.mql5.com
_StopFlag - Predefined Variables - MQL5 Reference - Reference on algorithmic/automated trading language for MetaTrader 5
 
Fernando Carreiro #:

For a "Slow Complete" optimisation, I don't know exactly how the passes are distributed between the various available threads, but I suspect that if any one pass freezes, then that thread will block, and if the same happens on the other threads, then eventually everything blocks.

I would suggest running small batches of optimisations, of a select few parameters, to narrow down which input combinations may be causing a "freeze"/"endless loop". Then once you have identified which inputs are involved, you can  then analyse and debug the code to fix it.

On a side note, it is also good to add the check of the _StopFlag condition, to all your loops, to help it break out of the endless loop, when the terminal instructs the EA to terminate execution.

Thank you very much. This is what i have been doing. I appreciate the fast responses though!
 
Fernando Carreiro #:

For a "Slow Complete" optimisation, I don't know exactly how the passes are distributed between the various available threads, but I suspect that if any one pass freezes, then that thread will block, and if the same happens on the other threads, then eventually everything blocks.

I would suggest running small batches of optimisations, of a select few parameters, to narrow down which input combinations may be causing a "freeze"/"endless loop". Then once you have identified which inputs are involved, you can  then analyse and debug the code to fix it.

On a side note, it is also good to add the check of the _StopFlag condition, to all your loops, to help it break out of the endless loop, when the terminal instructs the EA to terminate execution.

Hey Fernando. Can you add me back as a friend i want to ask you a question through messages. They are not related directly to this forum, so i figured this would be best
 
john Praytor #: Hey Fernando. Can you add me back as a friend i want to ask you a question through messages. They are not related directly to this forum, so i figured this would be best

I have not cancelled any "friends", and currently your profile shows me as being one of your "friends", and you are listed among mine too.

 
john Praytor:

Hello,


When the test makes it too optimization 509 out of 10,... whatever, it STOPS. When i take a look at the task menu it says it has completed almost all task but 3-4 cores are not completed and it is no where near the amount of test it says it will run. For example it tells me each core will be completing 12 task and 95% of them will say "12/12 task completed" then the other 5% will say "12/11 task completed"

I had same issue couple of times.


it is most of time is code. for me all was because of code but one. (may be I never found-out what was code problem).

As Fernando said, check if there is probable non end loops for some exceptional and very rare conditions from inputs. specially for loops over trailing and breakeven.


I had these issues before:

- some times we use negative/positive amounts from inputs and consider there is always a non zero value there. but if we set start in negative values and end in positive, then we may have zero in values.  even for one of all that long list of passes.

- check for global variables if there are any to be sure you delete them over OnDeinit. those can make non end loops too. (better Delete all Resources to increase speed of opt too).

- check for new candle function if you use any. some times in middle of code a "return" can cause a code based on open prices acts like real tick in backtest. cause each tick will be like a new candle. will take so long to complete!

- for cloud server it happens if your internet disconnects when mql tries to send data batch to server. even if reconnects there will be trouble and some items will freeze.

Reason: