MT5 Strategy Tester improvement request: Task allocation is currently not optimal
Over 2 months with no reply ... disappointing.
I'm now nearing the end of another optimisation, currently in the forward pass testing, & nearly 11 DAYS since the optimisation began. Again I'm rather disappointed by the far too basic task scheduling which will cause the testing to take nearly a day longer than it needs to!
There are just over 900 passes remaining. If ALL agents (5) were being used, those tasks would be done in around 10 hours. Unfortunately there are already 2 idle agents doing absolutely nothing, having completed their task lists!
Of those still processing one agent now has ~600 of the 900 tasks left to process by itself - which will take over 20 hours, if not closer to a day, to complete.
As a programmer myself I find the current task optimisation to be highly simplistic (this is really the only real annoyance I have with MT5), & have a VERY simple solution to keep ALL agents busy until the tasks are completed: when an agent completes all its tasks (the scheduler already knows when this happens!) then have it take half minus one of the tasks remaining from the agent with the most tasks remaining (again, the scheduler also knows this). Keep doing this until there are no more remaining tasks left to process.
This very simple change to task allocation (& more importantly: RE-allocation) would ensure the optimisation finished as quickly as possible on the hardware & agents available. Every time. This would save many people many hours, or even days, of time for a single optimisation run, especially for complex EA's that take several minutes to complete a single pass.
I would love to see this very simple task allocation algorithm change implemented in the Strategy Tester. It would be a MAJOR time saver for many people - especially in the forward pass testing phase, which currently ALWAYS has a single agent at the end with hundreds of passes left to process while all other agents sit idle!
Thanks,
Brendan.
Over 2 months with no reply ... disappointing.
I'm now nearing the end of another optimisation, currently in the forward pass testing, & nearly 11 DAYS since the optimisation began. Again I'm rather disappointed by the far too basic task scheduling which will cause the testing to take nearly a day longer than it needs to!
There are just over 900 passes remaining. If ALL agents (5) were being used, those tasks would be done in around 10 hours. Unfortunately there are already 2 idle agents doing absolutely nothing, having completed their task lists!
Of those still processing one agent now has ~600 of the 900 tasks left to process by itself - which will take over 20 hours, if not closer to a day, to complete.
As a programmer myself I find the current task optimisation to be highly simplistic (this is really the only real annoyance I have with MT5), & have a VERY simple solution to keep ALL agents busy until the tasks are completed: when an agent completes all its tasks (the scheduler already knows when this happens!) then have it take half minus one of the tasks remaining from the agent with the most tasks remaining (again, the scheduler also knows this). Keep doing this until there are no more remaining tasks left to process.
This very simple change to task allocation (& more importantly: RE-allocation) would ensure the optimisation finished as quickly as possible on the hardware & agents available. Every time. This would save many people many hours, or even days, of time for a single optimisation run, especially for complex EA's that take several minutes to complete a single pass.
I would love to see this very simple task allocation algorithm change implemented in the Strategy Tester. It would be a MAJOR time saver for many people - especially in the forward pass testing phase, which currently ALWAYS has a single agent at the end with hundreds of passes left to process while all other agents sit idle!
Thanks,
Brendan.
how many parameters are you optimizing? 11 days for an optimization is crazy even on real tick data across 28 pairs simultaneously. Also I don't use the forward testing feature I just do complete optimizations on the in sample then start another for the out of sample once its done and use excel stuff to compare. Also I'm pretty sure if the agent is idle you can reassign it to a talk by clicking the "enable" option unless you are referring to something different. I plan on building a new PC with 128GB of RAM (I want to do large time period real tick data optimizations) so I can relate about the frustration of being able to only run five agents.
I am really curious about how many parameters you are optimizing though.
how many parameters are you optimizing? 11 days for an optimization is crazy even on real tick data across 28 pairs simultaneously. Also I don't use the forward testing feature I just do complete optimizations on the in sample then start another for the out of sample once its done and use excel stuff to compare. Also I'm pretty sure if the agent is idle you can reassign it to a talk by clicking the "enable" option unless you are referring to something different. I plan on building a new PC with 128GB of RAM (I want to do large time period real tick data optimizations) so I can relate about the frustration of being able to only run five agents.
I am really curious about how many parameters you are optimizing though.
I have improved things a little ... until this latest update (MT5 v 5120) utterly ruined the Strategy Tester performance when I let it update yesterday!
I had a 3 year backtest period + 3.5 year forward test period coming in at just under a week (~165 hours, 161 generations, 25938 total tasks between back + forward periods, averaging around 2.1 minutes per pass). The new version update (v 5120) utterly ruined performance (no change even when recompiled in the new version)! I am currently just 412 passes in (so still on generation 0) and it's already over 16.5 HOURS! I am down to just 2 agents (from 6 last week) & even that is still close to maxing out CPU-RAM bandwidth! (sitting around 25-30 GB/s - last week it took 6 agents to use that bandwidth!)
As for number of parameters: 18 (for long, & again for short on separate runs). Complete Optimisation is impossible. Not even the best super computer on the planet could hold the results in RAM when the parameter sample space is 28 digits long!
I hadn't been using the Forward testing much, but manually going through results on a longer period attempting to find a good results that continue trading decently tends to take longer than the forward testing anyway. The Forward testing is useful to cut down how many passes I end up checking manually - at around 5 minutes per pass on the combined period.
As for real ticks ... nope, this is just on M1 OHLC mode! Just under 4GB RAM usage per agent. I'm only doing calcs on the start of a new M1 bar so "Every tick" & "Real Ticks" modes are not needed thankfully.
There is significant overhead for the tester alone, just syncing that many pairs properly, but it does a solid job. Even when exiting OnTick() before major calcs or trading are done (by simply returning early as a temporary test) it's still over 1.5 minutes per pass, but that's in line with 28 times a single chart pass time of a few seconds for the same period.
There's a lot of calculation & data needed for currency strength EA's - my indicator alone has 80 lines of data that need to be calculated. (sped up a little, recently, by making a new version to reduce CPU-RAM bandwidth that can use data precomputed & stored in a file by a separate simple EA made for that purpose)
CPU-RAM bandwidth is the limiting factor though. I have ~37GB/s to work with (dual channel, dual rank DDR4 3200, 2x32GB), & considering going to 4x32GB to get more available. Current run (just 2 agents) with the new MT5 v 5120 has the CPU running 10 degrees C cooler than the previous run (6 agents) on the previous version, but using MORE CPU-RAM bandwidth! (& 20 C cooler than when I do a single chart optimisation for another of my EA's with 9 agents! I am yet to test this EA on MT5 v 5120.)
Right now I'd just like the performance from the previous version! Fixing Forward pass task allocation would just be a nice bonus, given the current major tester performance loss issues! 1 week for a Genetic Optimisation plus Forward testing I can handle. Over 6 weeks for the same thing is nuts! (& Windows will force an update & reboot before that length of time despite disabling updates [only for 5 weeks max])
Sorry for the long rant. I'll leave it there for now!
im not 100% sure but i think i read something when optimizing the task is split up into 256 per batch then on each iteration of a new 256 batch it the optimization is computed to calculate the best parameters i believe. Im sorry i dont know. I have also never experienced this problem
The genetic optimisation initially does 512 passes (not 256 as reported in the documentation) for generation 0.
For generations 1 and later it does 256 passes per generation, using its internal genetic algorithm to choose a set of parameters to use for each pass (task), attempting to find "better" scoring passes based on the criteria chosen. (I have my own OnTester() function to score a pass, that has multiple methods available to choose from.)
As more generations go by it tends to use more & more passes from the cache from previous generations & actually calculate fewer & fewer new passes as time goes by - ultimately only actually calculating 10 to 20 passes on a generation as the optimisation nears its end.
I'd also like some user control over that algorithm, but there is none as far as I can tell.
None of that affects the Forward testing task allocation I originally flagged, which would benefit greatly from my simple modification suggestion - & allow near full Agent utilisation for the entire optimisation (which *should* be the goal of the tester!) with very little low utilisation periods.
I have improved things a little ... until this latest update (MT5 v 5120) utterly ruined the Strategy Tester performance when I let it update yesterday!
I had a 3 year backtest period + 3.5 year forward test period coming in at just under a week (~165 hours, 161 generations, 25938 total tasks between back + forward periods, averaging around 2.1 minutes per pass). The new version update (v 5120) utterly ruined performance (no change even when recompiled in the new version)! I am currently just 412 passes in (so still on generation 0) and it's already over 16.5 HOURS! I am down to just 2 agents (from 6 last week) & even that is still close to maxing out CPU-RAM bandwidth! (sitting around 25-30 GB/s - last week it took 6 agents to use that bandwidth!)
As for number of parameters: 18 (for long, & again for short on separate runs). Complete Optimisation is impossible. Not even the best super computer on the planet could hold the results in RAM when the parameter sample space is 28 digits long!
I hadn't been using the Forward testing much, but manually going through results on a longer period attempting to find a good results that continue trading decently tends to take longer than the forward testing anyway. The Forward testing is useful to cut down how many passes I end up checking manually - at around 5 minutes per pass on the combined period.
As for real ticks ... nope, this is just on M1 OHLC mode! Just under 4GB RAM usage per agent. I'm only doing calcs on the start of a new M1 bar so "Every tick" & "Real Ticks" modes are not needed thankfully.
There is significant overhead for the tester alone, just syncing that many pairs properly, but it does a solid job. Even when exiting OnTick() before major calcs or trading are done (by simply returning early as a temporary test) it's still over 1.5 minutes per pass, but that's in line with 28 times a single chart pass time of a few seconds for the same period.
There's a lot of calculation & data needed for currency strength EA's - my indicator alone has 80 lines of data that need to be calculated. (sped up a little, recently, by making a new version to reduce CPU-RAM bandwidth that can use data precomputed & stored in a file by a separate simple EA made for that purpose)
CPU-RAM bandwidth is the limiting factor though. I have ~37GB/s to work with (dual channel, dual rank DDR4 3200, 2x32GB), & considering going to 4x32GB to get more available. Current run (just 2 agents) with the new MT5 v 5120 has the CPU running 10 degrees C cooler than the previous run (6 agents) on the previous version, but using MORE CPU-RAM bandwidth! (& 20 C cooler than when I do a single chart optimisation for another of my EA's with 9 agents! I am yet to test this EA on MT5 v 5120.)
Right now I'd just like the performance from the previous version! Fixing Forward pass task allocation would just be a nice bonus, given the current major tester performance loss issues! 1 week for a Genetic Optimisation plus Forward testing I can handle. Over 6 weeks for the same thing is nuts! (& Windows will force an update & reboot before that length of time despite disabling updates [only for 5 weeks max])
Sorry for the long rant. I'll leave it there for now!
bro 18 is guaranteed overfitted stick to maximum 3 unless its involved with timed exits. Also 4GB of RAM per cpu means 37/4 ~= 8 or 9 cpu cores before you get memory errors. I would recommend just not doing 18 parameters because I promise you it will overfit even if you do regularization (which in my opinion makes no sense to do for periods of indicators and everything else that isn't neural network crap) Also depending on where you get your currency strength data (I'm assuming you are using COT data or something of that type) they often don't publish until much later so you may want to check that.
if you're only doing calcs on the start of a new M1 bar maybe try and clean your data so that its only the open and close ticks idk but I can promise you 18 parameters is 100% overfitted.
bro 18 is guaranteed overfitted stick to maximum 3 unless its involved with timed exits. Also 4GB of RAM per cpu means 37/4 ~= 8 or 9 cpu cores before you get memory errors. I would recommend just not doing 18 parameters because I promise you it will overfit even if you do regularization (which in my opinion makes no sense to do for periods of indicators and everything else that isn't neural network crap) Also depending on where you get your currency strength data (I'm assuming you are using COT data or something of that type) they often don't publish until much later so you may want to check that.
if you're only doing calcs on the start of a new M1 bar maybe try and clean your data so that its only the open and close ticks idk but I can promise you 18 parameters is 100% overfitted.
"Also 4GB of RAM per cpu means 37/4 ~= 8 or 9 cpu cores before you get memory errors."
This statement makes ZERO sense!
I never said anything about memory errors. My RAM is fine. 4GB RAM is per agent. I have 64GB RAM in total, so there is no issue with RAM usage, & a 6 core 12 thread CPU. Your 37/4 calculation makes no sense! The 37 GB per second I quoted is the maximum read/write CPU to RAM bandwidth available - ie., how fast can the CPU get data from the RAM & vice versa. THIS is the ultimate limit on how many agents can be used during testing, not total RAM divided by RAM per agent. Once that RAM bandwidth limit is hit, then using more agents actually SLOWS down the rate at which results are returned. I have run timed tests with everything from 1 to 8 agents to verify this.
"I'm assuming you are using COT data"
You assume too much! I have no idea what you are referring to there.
As I stated: I do MY OWN currency strength calculations, with MY OWN currency strength indicator. The only data I need is the chart data coming in on each chart in the terminal itself, so there is no issue with third party data feeds coming in late. That REQUIRES significant calculations & data from all 28 currency pairs used, & is the primary reason why I'm only looking for entries on a new M1 bar. Calculations every tick is not needed, as designed when I started writing the program, & simply wastes CPU time during testing.
As for only optimising 3 parameters at a time ... that would be a pointless exercise, given the flexibility I have built into the program that requires many parameters (more than just the 18 I'm optimising on for Long trades, with a matching set for Short trades), for various entry criteria, trailing stop settings, exit criteria. This is after fixing the currency strength settings to a single set of several parameters, & not optimising those. This is NOT a 5 parameter simple single chart basic EA! (My other trend based EA is closer to that, but also uses the same trade management class I developed, with access to the same trade sizing & trailing stop functions as the currency strength EA - & that EA does only take ~10 seconds for a 10 year pass. But that is just on a single chart.)
These days I generally do the main optimisation, & use those results as a starting point for further full optimisations with just 1 or 2 parameters at a time - attempting to pick values away from sharp edges in the results.
I'm aware of the curve fitting problem, & the highest score results obtained are rarely the ones I end up using - & forward testing is helpful for getting to parameter sets that work on data outside the optimisation period.
This is not an easy task. If it was, everyone would be doing it.
Oh am having ptsd from past experiences with RAM running out and getting memory issues, I thought thats what you were referring to but apparently not. Didn't read that you had 64gb of RAM so yeah the cpu/core calculation doesn't really make sense. Sounds like you know what your doing at least. I haven't actually tried running optimizations on the new mt5 build and I kept one of my mt5 instances on the previous build just in case and that seems to have been a good move, unless this problem is uniquely related to whatever EA your using so I don't really feel entitled to comment.
Whenever people talk about optimization I immediately think of memory issues lol. Idk I can't really comment because I don't know how your EA actually works and idk what type of parameters you are optimizing coz I've never bothered with currency strength type EAs/indicators
EDIT: are you writing/reading stuff to/from a file because in my experience that can slow stuff down a lot
2nd EDIT: also if your optimizing like 18 periods of different moving averages (which I have done similar stupidity before) I am going to lose my shit
Oh am having ptsd from past experiences with RAM running out and getting memory issues, I thought thats what you were referring to but apparently not. Didn't read that you had 64gb of RAM so yeah the cpu/core calculation doesn't really make sense. Sounds like you know what your doing at least. I haven't actually tried running optimizations on the new mt5 build and I kept one of my mt5 instances on the previous build just in case and that seems to have been a good move, unless this problem is uniquely related to whatever EA your using so I don't really feel entitled to comment.
Whenever people talk about optimization I immediately think of memory issues lol. Idk I can't really comment because I don't know how your EA actually works and idk what type of parameters you are optimizing coz I've never bothered with currency strength type EAs/indicators
EDIT: are you writing/reading stuff to/from a file because in my experience that can slow stuff down a lot
2nd EDIT: also if your optimizing like 18 periods of different moving averages (which I have done similar stupidity before) I am going to lose my shit
2nd edit: ROFL, no. I do have one SMA to somewhat smooth the final calculated values for currency strength, though, since it can get a bit rough, especially when using RSI as an input to the strength calculations. I'm optimising stuff like number of entry/exit bars to use, threshholds for currency strength based pair movement for entries/exits, stop loss & trailing stop settings, and so on. I have a "nightmare mode" version in my head where I have all those settings for each of the 28 currency pairs - 28 element arrays of parameters instead of single values! ... which would be a nightmare to test & get values for, & take crazy amounts of time without a small network of fast machines to use agents on! (Way beyond my highly limited budget at present. Also why I'm not using the MT5 cloud network.) I suspect attempting a decent neural net version would be far easier than that version - but that would also require a complete rewrite of the primary EA code & lots of training, etc!
On files: I'm reading precomputed currency strength data from a file (only in testing - it makes no sense in live trading since you can't precompute live data) because that is quicker than doing the calculations "live" during testing, & solidly reduces the CPU-RAM bandwidth needs, since the currency strength calculations have already been done before the optimisation ever begins, rather than 12000 times during the optimisation run (& another 12000 times in the forward test period, if used). Doing so allowed me to go from 2 or 3 agents to 5. The big reason is that the Windows disk cache gets used - which is where lots of RAM is nice, even when you don't need that much for programs. So the file essentially gets read from disk/ssd once (generally when I do a single pass before the main optimisation), and then sits in the disk cache for everything else, since it doesn't get changed after creation by the other EA, & so is accessed at RAM speeds. (which is nice since the file is 1GB & I'm running a SATA3 SSD, due to my previous hardware being 12 years old before I fried my old CPU 2 years ago) My disk cache is frequently around 30GB or more, so I rarely notice speed issues with the SATA SSD since the cache masks it well. If I go to 128GB for the extra bandwidth, then the cache will be even more comically large & effective. Having just a 1GB file to read on each pass is far lower overhead than needing many GB of data transferred between RAM and CPU to calculate on to get the same end data on every pass, so it saves on both CPU time & RAM bandwidth, making each pass faster, & boosts the number of agents that can be run at the same time, making the whole optimisation much faster.
I only managed to identify CPU-RAM bandwidth as a major issue back in January, after more than a year trying to work out why my new Ryzen 5 5600 (6 core/12 thread) CPU was only returning results about twice as fast as my old (12 years at frying) Core i5 2500K (4 core/4 thread), when it should have been around 15 times faster. After spend ages on attempting to optimise calculations, I finally thought about CPU-RAM bandwidth - & then found HWiNFO to be able to actually see it in real time. That's when I saw it getting maxxed out & started working on ways to reduce those needs. The latest method being only looking at entries every 5 minutes instead of 1 - which reduced the data file from 5GB down to 1GB, boosted each pass speed by around 45%, & let me go from 5 to 6 agents, which roughly doubled overall optimisation speed with about the same bandwidth usage. The CPU still isn't fully utilised, & it runs about 10 C cooler than when I do a 9 agent single chart optimisation, so there is still more work to be done - or perhaps that extra RAM might get it there & help feed the CPU much more quickly, allowing another agent or 2 or 3 to be used effectively.
Keeping the previous build is definitely a good move in this case. Build 5120 appears well below par at this point! So much so, that the Strategy Tester won't even properly open for me at this point - I just get a blank white window now! I'll reboot again within the next day or so, & hopefully it will behave itself again, & at least let me do some testing on my single chart trend base EA. I've never previously had any issues anywhere near as bad as what build 5120 is causing. A fixed build after 5120 can't come soon enough at this point!Been doing some testing on my single chart trend EA, & it looks like my original issue & reason for the thread has been resolved in build 5120.
Task allocation, at least on a single chart EA optimisation, now does get reassigned as needed to free agents until all tasks have been completed, utilising the CPU far better than before. Even on the Forward testing period. Also appears to be a little more memory bandwidth optimised, allowing more agents to be used.
This is a big improvement over the previous versions, & is the highlight of this release, for me at least.
Sadly muti-chart performance has a major regression & urgently needs to get back to the previous build's performance, but with this new task scheduler for solidly better performance.
- Free trading apps
- Over 8,000 signals for copying
- Economic news for exploring financial markets
You agree to website policy and terms of use
Hi,
I've noticed this ever since I switched to MT5 and multi-agent testing around 2 years ago.
When testing with all local agents (I have not yet used cloud testing) the task scheduler does not distribute tasks optimally - particularly when getting "near" the end of testing.
For genetic optimisation it initially allocates about 90% of the first generation's tasks (which is good), but then when the first agent finishes its tasks it gets basically the entire remaining 10% of tasks, which means that when the other agents finish they sit idle waiting for a single agent to crunch through many tasks, when they could be done many times faster by being redistributed to idle agents. Thus the first generation takes solidly longer than it should.
This gets even worse for forward testing, as my current run shows. After the genetic portion of the test, the forward testing began - & it looks like it is forward testing ALL passes done on the backtest period, not just the 25% or 40% best scored that the documentation states should be tested. What this has meant is that 1 agent has now finished its assigned tasks, & others have hundreds left to process - 1 agent has around 900 to do, while the others have fewer. This is one reason why I generally don't use forward testing.
What would be good is that the scheduler notices when an agent has finished its tasks, & reassigns tasks from other agents to the idle agent - particularly the agent with the most tasks left. That way, the testing would finish significantly faster, with ALL agents sharing the load until there are no more tasks to process.
I'm surprised this hasn't been requested before now, or perhaps not even noticed by the MT5 team, & I have found no settings where I can change this behaviour.
I've attached an image showing the idle agent, with over 2000 tasks left to process - which will take around 18 hours. This time would be much reduced with the addition of the currently single idle agent (& multiple idle agents the closer it gets to the end of testing) - though given the major difference in number left to process (between 150 & 900) it could be more than a day before the testing finishes, where it should be under 16 hours with all agents utilised.
For those wondering why I'm not using more agents given my CPU, the reason is simple: 5 agents has already nearly maxed out my CPU-RAM bandwidth (testing my currency strength EA, so accessing data from 28 currency pairs simultaneously), & running more agents actually returns results more slowly than with just 5 agents. I'm considering an upgrade to 128GB (dual channel 4x32GB DDR4-3200, up from dual channel 2x32GB DDR4-3200) to increase CPU-RAM bandwidth & hence number of agents that can be used in my testing. My CPU will happily run 9 agents for a single chart EA optimisation & rip through a 12000 pass optimisation (using M1 OHLC mode) on a 5 year period in 3 hours or less, but currency strength EA's looking at 28 pairs are far more demanding.
Being able to tweak some of the Strategy Tester settings (especially those used in genetic testing) would be nice to have as well, since it can get hyper-focused on narrow ranges as more generations go by, with almost no passes using even moderately randomised settings.
Thanks,
Brendan.