MT5 forward testing bugs.
To detect error in one owns program/EA/indicator use the debugger (in your case with hist. data beginning at the forward period with the preferred setting:
Code debugging: https://www.metatrader5.com/en/metaeditor/help/development/debug
Error Handling and Logging in MQL5: https://www.mql5.com/en/articles/2041
Tracing, Debugging and Structural Analysis of Source Code, scroll down to: "Launching and Debuggin": https://www.mql5.com/en/articles/272
some add. hints:
Tips from a professional programmer (Part I): Code storing, debugging and compiling. Working with projects and logs
https://www.mql5.com/en/articles/9266
Tips from a professional programmer (Part II): Storing and exchanging parameters between an Expert Advisor, scripts and external programs
https://www.mql5.com/en/articles/9327
Tips from a professional programmer (Part III): Logging. Connecting to the Seq log collection and analysis system
https://www.mql5.com/en/articles/10475
- www.metatrader5.com
Please consider which section is most appropriate — https://www.mql5.com/en/forum/172166/page6#comment_49114893
I mentioned the first bug happened to the highest scoring backtests in the forward results tab. I now think it might be happening to the ending of the optimization results (where the results are the highest). I guess those are the same thing but I just thought Id mention it.
I doubt that anyone read your whole post and understood it. I tried (first bug) but I am giving up.
I am using the Strategy Tester including forward option and I didn't notice any issue on my side.
I doubt that anyone read your whole post and understood it. I tried (first bug) but I am giving up.
I am using the Strategy Tester including forward option and I didn't notice any issue on my side.
Ok I got a simplified version of my code that still produces the bugs. Ill attach the file if you wouldn't mind trying it. The include files that it references come with mt5 so I wont post them. I made the ea by opening a new document, generating a new ea, and selecting the signals: AC, AO, Stoch, and WPR with no trailing and minimum trading amount. The custom date selected is just the last 2 months. Forward is set to 1/4. Modeling is: Every tick based on real ticks. Deposit is $100. Its fast genetic based algorithm with balance max. The inputs: threshold value to open/close 0-1-100, Stoploss/Takeprofit 1-1-100, the weights for each indicator is 0-0.1-1. The inputs for stoch and the one for wpr are selected and left to default range. Price level and expiration are left unselected. The MT5 build is 3980. (I don't think it matters what version though). "I doubt that anyone read your whole post and understood it." I was afraid of that. If you wouldn't mind trying to read bugs 2-4 I would appreciate it. I think they are worded slightly less badly than the first one.
Ok I got a simplified version of my code that still produces the bugs. Ill attach the file if you wouldn't mind trying it. The include files that it references come with mt5 so I wont post them. I made the ea by opening a new document, generating a new ea, and selecting the signals: AC, AO, Stoch, and WPR with no trailing and minimum trading amount. The custom date selected is just the last 2 months. Forward is set to 1/4. Modeling is: Every tick based on real ticks. Deposit is $100. Its fast genetic based algorithm with balance max. The inputs: threshold value to open/close 0-1-100, Stoploss/Takeprofit 1-1-100, the weights for each indicator is 0-0.1-1. The inputs for stoch and the one for wpr are selected and left to default range. Price level and expiration are left unselected. The MT5 build is 3980. (I don't think it matters what version though). "I doubt that anyone read your whole post and understood it." I was afraid of that. If you wouldn't mind trying to read bugs 2-4 I would appreciate it. I think they are worded slightly less badly than the first one.
Thanks for providing code and technical details. You are probably the first one I see doing it about this issue.
An other time please provide an ini file for the Strategy Tester settings.
And a set file for the EA parameters, unless they are the default ones.
Seems you like to write instead of providing data files or pictures.
The optimization is running then I will check the "Run Single Test" behaviour.
This screenshot from your initial post is strange.
242.32 (-1.46 for the forward) is obviously not the balance ? So what is it ?
And to what are you comparing it in this ?
These screenshots are obviously not using the same parameters. So if you right-click on the line with Forward=-1.46 and you get these results that seems to mean the input parameters were to transmitted correctly.
This screenshot from your initial post is strange.
242.32 (-1.46 for the forward) is obviously not the balance ? So what is it ?
And to what are you comparing it in this ?
These screenshots are obviously not using the same parameters. So if you right-click on the line with Forward=-1.46 and you get these results that seems to mean the input parameters were to transmitted correctly.
Those screenshots are from my main EA with a custom max, I'll upload some of the simpler EA. The first is the forward results tab. The top pass shows 0 trades and 0 profit. The second screenshot is of the forward tab when you run a single test of the top pass. It shows the correct results from those inputs. The third shows the input tab where you can see they are the same inputs as in the forward results tab. I took a screen recording of it if you want to check it out. At 10 seconds I right click and select run single test at the top, but the windows game bar screen recorder doesn't show doesn't show the popup for some reason though. Here is the link to the video (https://drive.google.com/file/d/1Gqd_UYFmVhYcBh85BLGfe9CoCXoDu7pc/view?usp=sharing) I wouldn't know why the code bugs for me and not you though. The only possible thing I could think of is maybe my cpu is amd and yours and others are intel? I don't think that would be why but maybe. Do you know what steps I could take to further investigate this? I also will upload the settings file. It wont let me upload a .ini file but I think the .set file is the same.
I think I might have found the problem. Looking at my log file in Terminal\(name)\Tester\Logs each core goes through these steps : Core 01 connecting to , Core 01 connected, Core 01 authorized, Core 01 common synchronization completed, Core 01 EURUSD: ticks synchronized already [43 bytes], and Core 01 connection closed. For the optimization side all my all my cores do all of these successfully. When it comes to the forward testing all my cores connected and were authorized but 9 of them don't have the common sync or the ticks synced already. I don't really know how all this works but it looks like something went wrong here that could explain the problem. Maybe some buffer/cache is stuck in a core or something and when it goes to save them all it uses the wrong buffer or there is an overflow or something. I don't know what I'm talking about but maybe you have an idea? Also when I look in the logs the numbers for the passes/cached results look inconsistent to me. It shows:
Tester result cache used 9374 times, Tester genetic optimization finished on pass 20480, Tester reading of 6194 forward result records from cache, Tester forward testing finished, total passes 10927, Tester 10927 new records saved, Statistics local 32448 tasks, and at the bottom of my terminal after it is done it says 31407 out of 10927. Shouldn't these numbers match up somehow?
Anyway let me know what you think, Thanks.
- Free trading apps
- Over 8,000 signals for copying
- Economic news for exploring financial markets
You agree to website policy and terms of use
This post is about a some weirdly related (I think) bugs that happen when forward testing optimized results. Please read through it all.
The first bug is that the displayed results from the top section of backtesting passes in forward tab are wrong. (Highest scoring passes in the backtest category, in the forward results tab) You can see this if you double click a pass in the forward results tab to run it in a single test, the resulting profits/trades will be different. The first two images give an example of this. The first is the listed profit/trades/ect seen in the forward results tab, the second is the (real) results for running the same pass in a single test. I don't know how many of the top passes this affects, it seems to vary on each run of the ea. (One that I have pulled up here doesn't have real results until a couple thousand) I also can't find out exactly where the incorrect passes stop for this reason: The shown result for most of the top is equal to the real result in a single test. So I also don't know if every pass up to a certain point is wrong (and the ones that actually do match, only match because they have the data from another pass that has the same result) or if only the outliers that sporadically appear are wrong. I am not wording this well so I am going to try to draw it out:
(1,1,5,1,1,1,5,1,1,1,5,1,1,5,1,1,1,1,1,5,1,1,1,1,1,5) Shown result.
(1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,5,1,1,1,1,1,5) Real results from single testing.
|here?|here? Where do the bad passes end?
5 5 5 5 Are just the 5's bad?
1,1,5,1,1,1,5,1,1,1,5,1,1,5,1,1,1 Or is this whole section bad?
This is just a really annoying bug because you see good forward passes in the top section for the optimization passes but it's just a visual bug and they are actually terrible.
The second bug happens during the first 512 passes in the forward testing. For the first 512 passes the agents/cores are assigned and complete way more passes than 512. For example say there are 10 cores that are each be assigned 1000 passes to complete. When they are all done only 512 passes will have been added to the final amount. You can see many passes being completed by each of the cores but they do not get counted towards the total. Instead the total number will go up to 512 seemingly separate from what is done by the cores. (It also starts out fast and slows down as if cores are finishing their section and only a couple cores are still calculating (like what you see in normal passes)). If this bug were alone I would think that that is just the way it does forward testing, but the number 512 comes up again and it might be doing something weird with the excess passes that messes something else up. This can be seen in the third picture, the seemingly arbitrary total number of passes got to 512 but the cores are still chugging away at the excess passes they were assigned. (they started around 4000 because the amount carries over from the optimization (but still most of them were assigned 1600))
The third bug happens from 512-1024. It does the same thing that 0-512 does (it is assigned and completes more passes than what it should). But this time it runs it on passes that should be on the end of the results. I don't really know how to word this so look at image 4. In the graph view you can see a chunk of passes that belong to the end, are tested at the start (at 512). Is this part of the reason why the first 512 passes are wrong? (bug 1) I don't know but this is another 512 chunk of passes that are weird.
I don't know exactly when but somewhere at the start (around 512-1024) the journal tab gives an error "x forward passes not processed and returned to task queue"
The passes from 1024 to 65536 work as you would expect. Each pass a core does corresponds to a pass being counted to the final result.
The fourth bug is that after 65536 passes (again very specific and this one I am less sure why its that number) It will just start failing passes (for lack of a better term). Up to that point you can see the results of passes vary throughout the graph as it does each one. But at 65536 passes it just flatlines, every result is the same as it keeps failing over and over. I say fail because you can see lots of passes being completed by the cores but the 65536 number will not go up. In the journal it keeps repeating "x forward passes not processed and returned to task queue" the number that it says doesn't seem to be significant, its not a random power of two and it changes on each rerun of the optimizations. (its 10753 for the one I'm doing right now). If you stop the test while its stuck on 65536 the whole program will freeze and never unfreezes. I have left it for hours to unfreeze but I just end up having to close the program in task manager because it never does. What is unfortunate is that when I close it with task manager sometimes the forward results tab disappear on reopen. I have had to redo hundreds of hours of calculation to the results disappearing like this. Another user has posted about this in forum post 424890. I don't know if this is another bug or not. You can see it get stuck in image 5. Some random maybe helpful info: In a forward test I did the total number of passes it needed to calculate was 75,000 most of the cores were finished as it neared 65536 (some were faster than others) It stopped at 65536 per usual but instead of the other cores finishing what they had left (and getting to 75,000) they all got assigned more tasks to do for whatever reason.
Its been so long sense I let one get to 65536 I don't remember if it actually finishes, I usually stop them right before. I have one running right now that I did let get there, it is at 65,536/75,000 passes but the chart shows that the total passes of failed + unfailed is at 85,000 and it will keep going up. I don't know if it will ever finish, I'll give it a little longer but man I hate wasting hours just to document what happens. I'll post later if it finishes or it finishes me.
This has happened with many different EAs. All my EA's are variations of the generated expert advisor template. I don't really want to post the code (I like to think my code is good and I'm special and I didn't just copy others) but I think this is replicable by just running an EA with the fast genetic based algorithm with lots of inputs. To see the last bug you need enough inputs that it does ~100,000 passes in the optimization to get at least 65536 passes in forward testing. I don't think my code or my computer is the problem. That being said I would think these issues would have been brought up before though if it wasn't just me. The only other mention of the first bug is in forum post 382346. The other bugs I don't know how to word them in a google search but I don't think anyone has posted about those either. Let me know if my code could be the problem and I'll try to get it to bug out on a simplified version of my EA. I hope I worded everything well enough for you to understand. I can run more tests if you need but it will just take a while as each one takes a couple days. Anyway these bugs have cost me many many hours of my life so I hope these can get fixed.