Bug Excessive Memory Usage - Metatrader 5 Latest Build 2190

[binlebin]

Hello,

The strategy tester of the latest terminal uses a lot of memory. The amount of memory used also increases significantly with the length of the Data Period.

For example, when optimizing my simple strategy using iMA with Fast Genetic Based Algorithm:

- 1 month uses 459 MB per process (metatester64.exe)

- 3 months use 1.05 GB per process

- 6 months use 2.16 GB per process

- 1 year uses 3.04 GB per process

(See pictures)

Due to the excessive usage of RAM, it is impossible to train on a longer period (1 year, etc.) on my current machine (Threadripper 2950x, 16 cores, 32 threads, 32 GB RAM).

I have been training fine using the older build (2007 Feb 2019) which uses around 800-900 MB per process even for 1 year data period. 

Could this be due to a memory leak in the strategy tester? I would appreciate if anyone can share how to overcome this problem.

More details:

Problem description:

- Excessive memory usage in the latest version of Metatrader 5 Terminal, build 2190.

Sequence of performed actions:

- Use strategy tester to optimize an Expert over a period of time using Genetic Algorithm. Adjust the period.

Expected result:

- Moderate, bound usage of memory per process (800MB - 1GB) like older versions of the terminal.

Obtained (undesired) result:

- The larger the time period, the more memory used per metatester64.exe process.

Additional information:

- System: Threadripper 2950X, 32GB RAM. Windows 10 Pro, Version 10.0.18362 Build 18362

[Alain Verleyen]

binlebin:
...

Due to the excessive usage of RAM, it is impossible to train on a longer period (1 year, etc.) on my current machine (Threadripper 2950x, 16 cores, 32 threads, 32 GB RAM).

...

What is happening if you use more than 1 year ? "Impossible" is not very useful.

[binlebin]

Alain Verleyen:

What is happening if you use more than 1 year ? "Impossible" is not very useful.

At 1 year period, 32 GB can afford to run on about 5 cores (10 threads), which spawns 10 metatester64.exe  processes. If I use all 16 cores, which spawns 32 processes, the computer freezes and becomes unresponsive after a few seconds of running. All memory is consumed by these processes. The number of hard page faults increases. In the terminal journal, the agent keeps reconnecting, and there is no progress. For this reason, I didn't attempt to run on a longer period.

[Alain Verleyen]

binlebin:

At 1 year period, 32 GB can afford to run on about 5 cores (10 threads), which spawns 10 metatester64.exe  processes. If I use all 16 cores, which spawns 32 processes, the computer freezes and becomes unresponsive after a few seconds of running. All memory is consumed by these processes. The number of hard page faults increases. In the terminal journal, the agent keeps reconnecting, and there is no progress. For this reason, I didn't attempt to run on a longer period.

Ok thanks for the detailed explanation.

Is this happening with your EA only or with any EA ? If it's with your EA only it's certainly related to your code (I know you said it's a new problem but it's still possible it comes from your code, so please check).

There is a new release in preparation for the 28th November, current beta is 2214. You can either try the beta or wait for the release.

[binlebin]

Alain Verleyen:

Ok thanks for the detailed explanation.

Is this happening with your EA only or with any EA ? If it's with your EA only it's certainly related to your code (I know you said it's a new problem but it's still possible it comes from your code, so please check).

There is a new release in preparation for the 28th November, current beta is 2214. You can either try the beta or wait for the release.

Hello Alain,

Thanks a lot for your suggestions. I went back and tried to optimize one of the example EAs provided with the terminal: "Moving Average.ex5", and I saw similar symptoms.

- 1 month uses ~900 MB

- 3 months use ~1.5 GB

- 6 months use ~2.3 GB

- 1 year uses ~3 GB

I then upgraded the terminal to the latest beta build 2218, 18 Nov 2019. And its RAM usage was about 100 MB more across all 4 periods.

[Alain Verleyen]

binlebin:

Hello Alain,

Thanks a lot for your suggestions. I went back and tried to optimize one of the example EAs provided with the terminal: "Moving Average.ex5", and I saw similar symptoms.

- 1 month uses ~900 MB

- 3 months use ~1.5 GB

- 6 months use ~2.3 GB

- 1 year uses ~3 GB

I then upgraded the terminal to the latest beta build 2218, 18 Nov 2019. And its RAM usage was about 100 MB more across all 4 periods.

I tried the same EA, 1 year, 4 cores each consuming 375 MB (I have 8G RAM and an Intel processor).

I have read somewhere on the Russian forum there is a problem with AMD processor and the Strategy Tester. A user reported similar problem to yours.

Metaquotes CEO said there is no general bug with AMD but they can check if all informations are provided to reproduce the problem. I posted a link to this topic.

[fxsaber]

binlebin:

- 1 month uses 459 MB per process (metatester64.exe)

- 3 months use 1.05 GB per process

- 6 months use 2.16 GB per process

- 1 year uses 3.04 GB per process

The history of quotes uses so much memory.

Someday they will make a single place for use by all agents.

[binlebin]

Alain Verleyen:

I tried the same EA, 1 year, 4 cores each consuming 375 MB (I have 8G RAM and an Intel processor).

I have read somewhere on the Russian forum there is a problem with AMD processor and the Strategy Tester. A user reported similar problem to yours.

Metaquotes CEO said there is no general bug with AMD but they can check if all informations are provided to reproduce the problem. I posted a link to this topic.

Thanks a lot for the update, @Alain Verleyen.

This is interesting. I just tested the same EA on my laptop running Intel i8750H (6 cores 12 threads), 16GB RAM, beta build 2220, 18 Nov 2019.

And each metatester64.exe process uses about 1.78GB for 1 year of EURUSD, M5 data. This is less than my AMD workstation's number but still significantly more than yours.

(I disabled 8 threads, see picture below).

 I certainly hope this could be fixed soon.

@fxsaber I am looking forward to that improvement, too.

[fxsaber]

Forum on trading, automated trading systems and testing trading strategies

(MT5) There are two very restrictive errors that make me leave the platform.

Renat Fatkhullin , 2019.11.20 22:24

This we will most likely do in January. After we release a large set of detailed settings for the tester's conditions that have been waiting for.

Now with multicore under 64 memory is really not enough even on computers with 128 GB.

But I’ll give good advice - use agents by the number of physical, not logical, kernels. Hypertreading almost does not help in difficult computational tasks.

[zemo]

mt5 staff could upgrade the mt5 plataform to use more threads to be used to the indicartors on screen,
 even with a fews chars and indicators the mt5 lags the bid/ask lines with the candles...
if you use indicartors that has zigzag or bbands in 5 charts this is the hell... and renko is the real nightmare..lol

unfortunatelly the last mt5 still only using one thread to calculate all charts and indicators on the main screen...

==================================================================

Персонал mt5 может обновить платформу mt5, чтобы использовать больше нитей, которые будут использоваться для индикаторов на экране,

даже с несколькими символами и индикаторами, mt5 откладывает линии спроса / предложения со свечами ...

если вы используете индикаторы с зигзагообразными или bbands в 5 чартах это ад ... а ренко настоящий кошмар .. лол

К сожалению, последний MT5 все еще использует только один поток для расчета всех графиков и индикаторов на главном экране ...

[Alain Verleyen]

zemo:

mt5 staff could upgrade the mt5 plataform to use more threads to be used to the indicartors on screen,
 even with a fews chars and indicators the mt5 lags the bid/ask lines with the candles...
if you use indicartors that has zigzag or bbands in 5 charts this is the hell... and renko is the real nightmare..lol

You can have such problems with really heavy indicators, but certainly not with ZigZag or BBands under normal conditions.

If you want to be constructive, you need to provide information to reproduce your issue.

unfortunatelly the last mt5 still only using one thread to calculate all charts and indicators on the main screen...

That's not exact, there is 1 thread by symbol.

And your post is off-topic for this thread which is about excessive memory usage when using CopyXXX functions.