Question for developers - using all computational cores during optimisation - page 8
You are missing trading opportunities:
- Free trading apps
- Over 8,000 signals for copying
- Economic news for exploring financial markets
Registration
Log in
You agree to website policy and terms of use
If you do not have an account, please register
Interesting approach.
There could be several solutions to this problem depending on the specifics.
If all parameters, those that already exist and those that may appear, have a strictly defined place in the set of parameters (chromosome or other equivalent), then there is no problem at all.
If a certain place of each parameter cannot be defined beforehand, then it is necessary to divide parameters into types that can be combined strictly only with each other (then it does not matter which place a parameter is in) - modification of AO is required.
There are other ways to solve the problem, but in any case you will need a custom AO, a superstructure over the standard one.
confirmed memory overload .... Although strange, no one cancelled swap, once again, I think developers need to take this into account
You should let us know how the problem is solved. We are worried about you, aren't we?
How much minimum memory do you need per agent (have you reduced the number of activated agents until all are evenly loaded?)
Increased the pagefile? The physical memory will be constantly swapped, it may slow down a lot, you need to find a number of cores where the total optimization time is minimal.
You should let us know how the problem is solved. We are worried about you, aren't we?
How much minimum memory do you need per agent (have you reduced the number of activated agents until all are evenly loaded?)
Increased the pagefile? Physical memory will be constantly swapped, there could be severe slowdowns, we need to find a number of cores where the total optimization time is minimal.
It all depends on tick usage, history depth, number of tools. Just look at task manager, you will see there everything, the amount of all RAM minus 1-2 GB for the operating system divided by the amount used by one agent. It's different for each one.
A real improvement can be made by developers if one area of RAM is used for the quotes and possibly for the indicators.It all depends on tick usage, depth of history, number of tools. Just look at the task manager, you can see the total RAM size minus 1-2 GB for the operating system divided by the size used by one agent. It's different for each one.
The developers can really improve the situation, if one area of RAM is used for quotes and, perhaps, for indicators.Are you explaining it to me? I was explaining the problem to a man, and now I am interested in the result. There is no feedback.
And we have already discussed inefficient use of memory by the terminal many times, and MQ several times promised to change the situation with duplicated tick history and temporary files for each agent, and their long creation before each tick optimization. Personally I've had to disable almost half of the agents and tick optimisations over a number of years. I have 8GB and 8 agents. But for now we use what we have, and we can either increase the memory size or disable agents.
Are you explaining it to me? I explained the problem to the man, and now I'm wondering about the result. There is no feedback.
And inefficient use of memory by the terminal we have already discussed many times, and MQ several times promised to change the situation with duplication of tick history and temporary files for each agent, and their long creation before each tick optimization. Personally I've had to disable almost half of the agents and tick optimisations over a number of years. I have 8GB and 8 agents. But for now we use what we have, and we can either increase the memory size or disable agents.
>You just have to let us know how the problem is solved. We're worried about you.
>I explained the problem to the man, and now I'm interested in the result. No feedback.
I am sorry, I was working, I had no time.
I have optimised the EA. I have removed some "unimportant" parts in order to make the optimizer work (specifically, everything related to OpenCL and SQLite). I don't have memory overflow now. BUT ... it is not a solution of course.
On another computer I had to disable some of the cores just to avoid freezing ... So, for example, the system on a 3950X (16 cores/32 threads) and 32 GB, using all the threads just hangs. Moreover, it hangs on agents and hangs until you manually kill the process through the task manager .... I disabled some of the cores, the computation went on. I think the developers should do something to make the problem explicit. If the optimizer really needs more than ten gigs of memory for calculation it should be clearly written in something like an alert. I will remind you once again about swap, though. I have Xmeters installed ... So I can see the memory load visually right on the taskbar.
I think there is some other glitch related specifically to amdc CPU and it wasn't there before - although the advisor was the same. Symptoms are as follows - only 5 cores .... and the miscalculation hangs... And it's not exactly the memory, i.e. same Expert Advisor force 16 threads with no problem, and the problem is floating, now and then it's not. If I run the test not in the optimizer, it runs fine. I noticed it more than once. So I have to check it out.
Concerning brakes on network agents, I still can't get to it. The "one core - one job" is beyond the developers' comprehension. As before, 10 cores will each get 30 jobs and another 30 network agents will get idle. Takes a long time to distribute tasks thinking about something. So all in all there is a lot of lag.
And yes I forgot: Thank you all very much for your participation, help and advice.toRenat Fatkhullin
Still, I want to raise the subject once again and ask a question specificallyto Renat Fatkhullin
1. I am optimizing a local farm that consists of several servers with quite different capacities and I saw some really terrible delays during optimization caused by CPUs with different performances. Suppose there are servers with CPU Xeon E5-2620 v2 (2.1 GHz), Xeon E5-2620 0 (2.00 GHz), i7-3820 (3.6 GHz), Xeon E3-1245 (3.7 GHz), Ryzen 7 2700, Ryzen 9 3900X, Ryzen 9 3950X. The current algorithm works like this: it forms a job stack, divides that job stack equally for each available core. Due to low-speed Xeon E5-2620 0 (2.00GHz) and Xeon E5-2620 v2 (2.1GHz) CPUs, the farm was idle and counted its tasks, but these two CPUs didn't count even half of them. Thus the whole farm just stands idle. It happens and will continue to happen because CPUs have different speed and as long as jobs are distributed in packets. Experience shows that network latency doesn't matter at all and is negligible. Is it already possible to rework the job distribution algorithm: not to distribute several jobs to each available core, but"give each freed core one job from the current generation stack"?
2. Is it possible to add saving test results to xml file every 10 minutes .... and running from last save?
We have engaged in a complete rewrite of the tester and optimiser.
We will drastically overhaul and fix the accumulated problems.
We have engaged in a complete rewrite of the tester and optimiser.
We will drastically overhaul and fix the accumulated problems.
Will there be:
1. Faster reaction to agent's dropout, where, say, if there is no withdrawal from agent, packet delivery is interrupted for minutes and packets are redistributed.
2. Will it be possible to upload in one instance data for all agents on a remote machine?
3. will the problem of transferring large amounts of data, when the connection with an agent is broken, having not downloaded all the data (files) needed for optimization?
There's another thing that bothers me ....
let's say optimization is going on and at that moment the agent's metatrader is updated .... the agent has a job (or rather a batch of jobs), will it be lost or recalculated?