Use 128 threads as agents for optimization on 64 core CPU

[A G]

Hello community, 

My first post here. I have a 64 core AMD Threadripper CPU workstation with 512GB of RAM. 

I'm currently using all 64 available cores for optimizing my algos, although the CPU has 128 threads.

From what I've read, agents are assigned to each thread... Does that mean that I can run 128 agents (1 thread per agent) instead of the current 64 agents (1 core per agent) that I'm using? 

How would I go about enabling this on my workstation?

Thanks.

[Carl Schreiber]

Your local terminal can use all the 128 threads (unless there is an internal limit) but each thread loads its own data completely (as far as I know). That means a RAM requirement of aprox. 128*backtestdata and then it could be that your RAM is too small and the system starts to swap and the performance drops to practically zero.

For the Cloud Network see this: https://www.mql5.com/en/forum/382257

[Gamuchirai Zororo Ndawana]

A G:

Hello community, 

My first post here. I have a 64 core AMD Threadripper CPU workstation with 512GB of RAM. 

I'm currently using all 64 available cores for optimizing my algos, although the CPU has 128 threads.

From what I've read, agents are assigned to each thread... Does that mean that I can run 128 agents (1 thread per agent) instead of the current 64 agents (1 core per agent) that I'm using? 

How would I go about enabling this on my workstation?

Thanks.

I believe it's 1 agent per core and also by default it uses all cores. 

[A G]

Carl Schreiber #:

Your local terminal can use all the 128 threads (unless there is an internal limit) but each thread loads its own data completely (as far as I know). That means a RAM requirement of aprox. 128*backtestdata and then it could be that your RAM is too small and the system starts to swap and the performance drops to practically zero.

For the Cloud Network see this: https://www.mql5.com/en/forum/382257

Hey thanks,

I have 512GB of RAM so plenty available. 

I prefer testing over my own custom data so dont want to use MQL Cloud, which is why I want to increase the optimization process as much as possible.

[A G]

Gamuchirai Zororo Ndawana #:
I believe it's 1 agent per core and also by default it uses all cores. 

I thought that too, but read on various forums that its 1 agent per thread - so im confused... Would like a definitive answer because I can have 128 agents instead of my current 64 agents.

[Dominik Egert]

A G #:

I thought that too, but read on various forums that its 1 agent per thread - so im confused... Would like a definitive answer because I can have 128 agents instead of my current 64 agents.

Search the forum on this topic, please.

As far as I remember, there has been discussion about that topic, where the conclusion was, the maximum possible agents per one machine were 64.

The solution to get around that was to use virtual machines, and then connecting them per local farm to your host machine.

I suggest to use max 120 cores for agents, leave some for your OS and the host Terminal. If you assign all 128 cores, the load on your CPU will be so high, you won't be able to use your PC anymore. I mean, it will not even be able to update your mouse pointer in a reasonable time, if you load all cores.

Oracle Virtual Machine, VMware Player, or MS HyperVisor will do it just fine. Partition 2 VMs and assign 62 cores to each. Install 60 agents on each and connect them via local network farm to your host Terminal.You will have very little speed impact by the VM overhead. VMware has the lowest with around 3%, if you are not using Linux KVM with QEmu (around 1 to 2%).

[A G]

Dominik Egert #:

Search the forum on this topic, please.

As far as I remember, there has been discussion about that topic, where the conclusion was, the maximum possible agents per one machine were 64.

The solution to get around that was to use virtual machines, and then connecting them per local farm to your host machine.

I suggest to use max 120 cores for agents, leave some for your OS and the host Terminal. If you assign all 128 cores, the load on your CPU will be so high, you won't be able to use your PC anymore. I mean, it will not even be able to update your mouse pointer in a reasonable time, if you load all cores.

Oracle Virtual Machine, VMware Player, or MS HyperVisor will do it just fine. Partition 2 VMs and assign 62 cores to each. Install 60 agents on each and connect them via local network farm to your host Terminal.You will have very little speed impact by the VM overhead. VMware has the lowest with around 3%, if you are not using Linux KVM with QEmu (around 1 to 2%).

So just to be sure, I have 64 cores on my AMD Threadripper CPU but it has 128 threads. Are you saying that I can use 1 agent per thread (128 agents) using your method?

[Dominik Egert]

A G #:

So just to be sure, I have 64 cores on my AMD Threadripper CPU but it has 128 threads. Are you saying that I can use 1 agent per thread (128 agents) using your method?

Yes, but only for your local farm, not for selling to the cloud as Cloud Agent.

EDIT:

As suggested, dont use all cores/threads available, you wont be able to interact with your PC anymore.

[Alain Verleyen]

A G #:

So just to be sure, I have 64 cores on my AMD Threadripper CPU but it has 128 threads. Are you saying that I can use 1 agent per thread (128 agents) using your method?

The official recommendation by MetaQuotes is to use a maximum of 1 agent by physical core. So 64 in your case.

[Emerson Gomes]

If you're not using a local farm, but only on your local machine, it will be able to use 128 Threads. Under local agents, each "core" means a thread.

So just disable 2-3 cores so you can continue using your machine and you should be able to fully use your CPU.

However for local farms, there's this limitation of 1 agent per physical CPU core. Here, each core really means a core. Honestly I think this limitation is kind of pointless. Most of the times I can get close to 2x perf just by using the HT cores. So while you cannot sell computing to the cloud, if you need to rely on several local farm nodes, one workaround is to run virtualization and run 2 VMs, one pinned to physical cores, the other one to HT cores.

[A G]

Alain Verleyen #:
The official recommendation by MetaQuotes is to use a maximum of 1 agent by physical core. So 64 in your case.

Dominik Egert #:

Yes, but only for your local farm, not for selling to the cloud as Cloud Agent.

EDIT:

As suggested, dont use all cores/threads available, you wont be able to interact with your PC anymore.

Thank you guys. That clarifies everything for me.