Genetic Optimization Limits
Hello all,
MetaTrader 5 - Within my EA I have many inputs - Around 200. When running the Genetic Based Algorithm for Optimization as you would expect I reach the limit quite quickly, which is around 1^310. However due to the fact that i have a very powerful computer I am able to use this large number of combinations, as well as test across a 10yr period of time (using every tick based on real ticks), and my machine will still complete the calcs in around 3hrs. Not an issue.
Therefore, the question is "Is there any way, or code that can be applied to remove the limit in the Genetic Based Optimization?". I understand that this means that my machine might take many hours or days to process such vast combinations, but with a fast machine I do not want to be limited to the current 64 bit limitations set by an internal cut-off, rather than a limitation by my pc or my time.
I have noticed this question being raised almost 10yrs ago, and it seems that these limits are still there, even though machines are far superior now.
Thanks to anyone who has a solution.
No thanks to anyone who recommends reducing my inputs - ;)
J :)
I recall achieving millions of passes by forcing the terminal to process one bar but i did all the tests internally , the equivalent of math calcs .
So each optimization run 1 bar but in that bar i loaded up my own test environments , run them and extracted a score for the generation.
But it was not for 10years of data nor it had the vast amount of tick data you need.
I had used the Custom Max optimization method , and open prices only to process one bar .
Please consider which section is most appropriate — https://www.mql5.com/en/forum/172166/page6#comment_49114893
Hello all,
MetaTrader 5 - Within my EA I have many inputs - Around 200. When running the Genetic Based Algorithm for Optimization as you would expect I reach the limit quite quickly, which is around 1^310. However due to the fact that i have a very powerful computer I am able to use this large number of combinations, as well as test across a 10yr period of time (using every tick based on real ticks), and my machine will still complete the calcs in around 3hrs. Not an issue.
Therefore, the question is "Is there any way, or code that can be applied to remove the limit in the Genetic Based Optimization?". I understand that this means that my machine might take many hours or days to process such vast combinations, but with a fast machine I do not want to be limited to the current 64 bit limitations set by an internal cut-off, rather than a limitation by my pc or my time.
I have noticed this question being raised almost 10yrs ago, and it seems that these limits are still there, even though machines are far superior now.
Thanks to anyone who has a solution.
No thanks to anyone who recommends reducing my inputs - ;)
J :)
I'm sorry this doesn't answer your question, but I always struggle with optimizations endless time and I couldn't help asking you this: what computer do you have working that fast?
Cheers
MetaTrader 5 - Within my EA I have many inputs - Around 200. When running the Genetic Based Algorithm for Optimization as you would expect I reach the limit quite quickly, which is around 1^310. However due to the fact that i have a very powerful computer I am able to use this large number of combinations, as well as test across a 10yr period of time (using every tick based on real ticks), and my machine will still complete the calcs in around 3hrs. Not an issue.
Therefore, the question is "Is there any way, or code that can be applied to remove the limit in the Genetic Based Optimization?". I understand that this means that my machine might take many hours or days to process such vast combinations, but with a fast machine I do not want to be limited to the current 64 bit limitations set by an internal cut-off, rather than a limitation by my pc or my time.
I have noticed this question being raised almost 10yrs ago, and it seems that these limits are still there, even though machines are far superior now.
What is your problem exactly? Genetic optimization from MQ is designed to run a predefined number of generations during single optimization, but you can continue launching the genetic optimization (with the same settings) again and again, which will in effect continue the same optimization task, and will be producing more and more optimal results, until you decide it's enough (or did not produce sufficiently decent results) after hours/days/weeks of the process, if you want and can afford.
You can even pause and continue the optimization after a break (with powered off PC), provided that you keep the optimization cache intact in the tester folder.
Some folks automate the continuous/resumable optimization by bat-files or from mql5.
I'm sorry this doesn't answer your question, but I always struggle with optimizations endless time and I couldn't help asking you this: what computer do you have working that fast?
Cheers
Hey,
Quite often the trick is in the coding rather than the machine. I am certainly not an expert in coding and i have a couple of programmers that do the complicated stuff for me. But i did have an EA (my maths) programmed by someone that worked fine, but took forever to run backtesting. Then i had someone strip it down and use different methods to get the same results. Really good programmers will be able to optimise the Code in different ways. I dont know the rules of this website, but i expect i am unable to tell you which Freelance service website i used, or who the programmer was. Once my EA was optimised i was able to take the run time from 6 weeks down to 6 hours. Exactly the same results, but the code used less demeanding systems. My PC is an i9-14900KF processor with 64GB of RAM. But i believe it is almost entirely down to the processor power and the number of cores.
One thing i have really wanted to do is utilise my Nvidia RTX 4090 Graphics Card to complete the optimisation using OpenCL - However after many hours and many ocnversations i have still not found a thread or person who can tell me how to add that to my EA code so that the GPU is used rather than the CPU - Similarly to how Bitcoin mining changed many years ago.
What is your problem exactly? Genetic optimization from MQ is designed to run a predefined number of generations during single optimization, but you can continue launching the genetic optimization (with the same settings) again and again, which will in effect continue the same optimization task, and will be producing more and more optimal results, until you decide it's enough (or did not produce sufficiently decent results) after hours/days/weeks of the process, if you want and can afford.
You can even pause and continue the optimization after a break (with powered off PC), provided that you keep the optimization cache intact in the tester folder.
Some folks automate the continuous/resumable optimization by bat-files or from mql5.
Hey,
The issue is the number of inputs exceeds the limit within MT5 optimisation. For example, if i had 2 inputs, and each of those inputs through optimisation had 1000 possibilities, then the total number of possibilities would be 1000x1000 - 1,000,000 (1e+6). If i had 3 inputs each with 1,000 it would be 1,000,000,000 - or 1e+9
The issue is that i have about 200 inputs. If each of these 200 inputs had 1000 possibilities. This will give me about 1e+600 number of combinations. MT5 however has a limit around 1e+308. I also note that in Excel as an example it also has a similar limit in any calculation.
This 1e+308 limit has been around for over a decade, and computers have vastly changed in that time.
Within MT5 if your number of possible combinations exceeds the 1e+308 a message appears stating the calculation cant be done "Amount of Optimization passes overflows 64 bit long value". I want to know if there is a way to bypass this limit.
Just to add - I know that this lmit is derived by the 64-bit technogy and is something to do with the limit or width of mathematical data being processed at one time by the CPU - So the real limit is with the Processor, but..... Within software such as MT5 you would be able to do the individual computations in the background using the 1e+308 limit, and then run a separate calculation using another 1e+308 limit and compare and combine the results. This is very simple maths, so i am unsure why MT5 needs to have kept this limit for over 10yrs, rather than removing the limit but optimising multiple batches in the background - at the expense of time.
I know this is pretty complicated, and i doubt many people reach this issue - but is there any way of forcing MT5 not to acknowledge this error message and to get on with the optimisation of any size?
The issue is the number of inputs exceeds the limit within MT5 optimisation. For example, if i had 2 inputs, and each of those inputs through optimisation had 1000 possibilities, then the total number of possibilities would be 1000x1000 - 1,000,000 (1e+6). If i had 3 inputs each with 1,000 it would be 1,000,000,000 - or 1e+9
The issue is that i have about 200 inputs. If each of these 200 inputs had 1000 possibilities. This will give me about 1e+600 number of combinations. MT5 however has a limit around 1e+308. I also note that in Excel as an example it also has a similar limit in any calculation.
This 1e+308 limit has been around for over a decade, and computers have vastly changed in that time.
Within MT5 if your number of possible combinations exceeds the 1e+308 a message appears stating the calculation cant be done "Amount of Optimization passes overflows 64 bit long value". I want to know if there is a way to bypass this limit.
Just to add - I know that this lmit is derived by the 64-bit technogy and is something to do with the limit or width of mathematical data being processed at one time by the CPU - So the real limit is with the Processor, but..... Within software such as MT5 you would be able to do the individual computations in the background using the 1e+308 limit, and then run a separate calculation using another 1e+308 limit and compare and combine the results. This is very simple maths, so i am unsure why MT5 needs to have kept this limit for over 10yrs, rather than removing the limit but optimising multiple batches in the background - at the expense of time.
I know this is pretty complicated, and i doubt many people reach this issue - but is there any way of forcing MT5 not to acknowledge this error message and to get on with the optimisation of any size?
Even if you'd wanted to have as "small" number of parameters as 1^100 - this would mean curve fitting. This is self-deception. It's useless for future online trading.
I strongly suggest you to re-consider your approach. No one will change software maths beyond current up to day consumer computer's standards.
- Free trading apps
- Over 8,000 signals for copying
- Economic news for exploring financial markets
You agree to website policy and terms of use
Hello all,
MetaTrader 5 - Within my EA I have many inputs - Around 200. When running the Genetic Based Algorithm for Optimization as you would expect I reach the limit quite quickly, which is around 1^310. However due to the fact that i have a very powerful computer I am able to use this large number of combinations, as well as test across a 10yr period of time (using every tick based on real ticks), and my machine will still complete the calcs in around 3hrs. Not an issue.
Therefore, the question is "Is there any way, or code that can be applied to remove the limit in the Genetic Based Optimization?". I understand that this means that my machine might take many hours or days to process such vast combinations, but with a fast machine I do not want to be limited to the current 64 bit limitations set by an internal cut-off, rather than a limitation by my pc or my time.
I have noticed this question being raised almost 10yrs ago, and it seems that these limits are still there, even though machines are far superior now.
Thanks to anyone who has a solution.
No thanks to anyone who recommends reducing my inputs - ;)
J :)