High DISK (SSD) Usage While Optimizing - page 3

[cangreburguer]

Alain Verleyen #:

You forgot to say you are using a Mac ! Not sure what is the impact, I don't have one to check.

And around 16 GB for the backtest only. It's all normal.

P.S: Not related to the RAM/Disk issue, but your trading log is full of errors. Also the trades stopped in 2019 for a backtest supposed to run up to December 2024.

Im not using Mac, I said the components of my PC on the original post.

Should we check the RAM used for the optimization?

Yeah, those issues seem to be data related, werid.

Thanks!

[Alain Verleyen]

cangreburguer #:

Im not using Mac, I said the components of my PC on the original post.

Should we check the RAM used for the optimization?

Yeah, those issues seem to be data related, werid.

Thanks!

No Mac ? Why is the file you sent showing this when I unzip it ?

The ram used for optimization is the same, just multiply that amount for each agent. Obviously you can hardly use more than 1 agent without changing your test period.

The number of ticks include USDJPY as even if you are using a custom symbol, the profit is in JPY and your account in USD, so the Tester needs USDJPY data to convert profit currently.

You could try using this option :

[[Deleted]]

cangreburguer #: The single backtest log is attached below.

These are the points I picked up from a quick scan of the log. I did not study it in depth.

Testing date range ...

KL      0       00:58:52.277    Tester  USDJPY Dukascopy FTMO,M5 (FTMO-Demo2): testing of Experts\EA Rango (Buy).ex5 from 2015.01.01 00:00 to 2024.12.01 00:00

First and last transactions ... (no trades after 2019.09.26).

RJ      0       00:59:06.385    Core 01 2015.01.14 10:00:46   market buy 3.17 USDJPY Dukascopy FTMO sl: 116.450 tp: 119.174 (116.695 / 116.698)

QN      0       00:59:30.888    Core 01 2019.09.26 11:02:08   order performed sell 8.35 at 107.612 [#837 sell 8.35 USDJPY Dukascopy FTMO at 107.612]

Over 19999 invalid stops modifications (the text editor stopped counting, so there was probably much more) ...

PL      2       00:59:12.499    Core 01 2015.03.02 10:16:33   failed modify #24 buy 8.7 USDJPY Dukascopy FTMO sl: 119.914, tp: 120.764 -> sl: 119.914, tp: 120.764 [Invalid stops]

Final memory usage metrics ...

MG      0       00:59:32.355    Core 01 8280 Mb memory used including 255 Mb of history data, 7360 Mb of tick data

This means that for a maximum of 32GB, you will only be able to run 3-4 threads.

However, this is inconclusive as the EA stopped transacting after 2019.09.26.

[[Deleted]]

cangreburguer #: Other stuff that happens now and before didn't': When visual backtesting, the first trade that is opened freezes the tester for 15 seconds, and then it continues with the ticks and moving till the next trade

Yes, that can be seen in the logs. The tester proceeds to collect historical data for the real "USDJPY" symbol (not only the custom symbol "USDJPY Dukascopy FTMO").

Something in the code is referencing the normal "USDJPY" symbol instead of your custom symbol, forcing the tester to carry out the collection of the extra historical data.

RJ      0       00:59:06.385    Core 01 2015.01.14 10:00:46   market buy 3.17 USDJPY Dukascopy FTMO sl: 116.450 tp: 119.174 (116.695 / 116.698)
EF      0       00:59:06.385    Core 01 USDJPY: symbol to be synchronized
LM      0       00:59:06.385    Core 01 USDJPY: symbol synchronized, 3720 bytes of symbol info received
JE      0       00:59:06.385    Core 01 USDJPY: load 6.43 Mb of history data to synchronize in 0:00:00.510
NF      0       00:59:06.385    Core 01 USDJPY: history synchronized from 2014.01.01 to 2024.11.29
NS      0       00:59:06.385    Core 01 USDJPY: ticks synchronization started
CH      0       00:59:06.385    Core 01 USDJPY: load 206.01 Mb of tick data to synchronize in 0:00:02.844
IH      0       00:59:06.385    Core 01 USDJPY: history ticks synchronized from 2023.11.16 to 2024.11.29
KO      0       00:59:06.385    Core 01 2015.01.14 10:00:46   deal #2 buy 3.17 USDJPY Dukascopy FTMO at 116.698 done (based on order #2)
KQ      0       00:59:06.385    Core 01 2015.01.14 10:00:46   deal performed [#2 buy 3.17 USDJPY Dukascopy FTMO at 116.698]
NM      0       00:59:06.385    Core 01 2015.01.14 10:00:46   order performed buy 3.17 at 116.698 [#2 buy 3.17 USDJPY Dukascopy FTMO at 116.698]

[Alain Verleyen]

Fernando Carreiro #:

These are the points I picked up from a quick scan at the log. I did not study it in depth.

Testing date range ...

First and last transactions ... (no trades after 2019.09.26).

Over 19999 invalid stops modifications (the text editor stopped counting, so there was probably much more) ...

Final memory usage metrics ...

This means that for a maximum of 32GB, you will only be able to run 3-4 threads.

However, this is inconclusive as the EA stopped transacting after 2019.09.26.

You are right, I misread, so I added numbers wrongly to get a total of 16GB.

Though I find strange it's only 8GB for 382 millions ticks...anyway, that's details.

[cangreburguer]

Good news!

After disabling and enabling every part of my code I found the problem.

As you guys said, there was a trade modification not working correctly, and that was causing too many logs = no memory = no optimizations.

After modifying the code and eliminating that error the optimizations are running smooth.

Thanks a lot for helping me diagnose the problem.

[cangreburguer]

Fernando Carreiro #:

These are the points I picked up from a quick scan of the log. I did not study it in depth.

Testing date range ...

First and last transactions ... (no trades after 2019.09.26).

Over 19999 invalid stops modifications (the text editor stopped counting, so there was probably much more) ...

Final memory usage metrics ...

This means that for a maximum of 32GB, you will only be able to run 3-4 threads.

However, this is inconclusive as the EA stopped transacting after 2019.09.26.

Hello Fernando, thanks a lot for the help, I did resolve the problem using some info that you gave me, however I have a new problem that is related but I made a new topic because its a different type of error. If you have some time I would love to have some help, thanks a lot! 

The new topic: https://www.mql5.com/en/forum/478627

[cangreburguer]

Fernando Carreiro #:

Yes, that can be seen in the logs. The tester proceeds to collect historical data for the real "USDJPY" symbol (not only the custom symbol "USDJPY Dukascopy FTMO").

Something in the code is referencing the normal "USDJPY" symbol instead of your custom symbol, forcing the tester to carry out the collection of the extra historical data.

Hello Again! The problem came back...

As I previously mentioned on comment #26 deleting part of my code worked as a solution to the bunch of errors about "failed modifying the stop", and the memory was free to run again, but it came back in other way, the modify stops error is not here but the optimization doesn't advance. 

The problem was high disk usage, but now it's not related to the disk anymore, now the optimization doesnt advance, no matter disk usage, more details below.

When I start an optimization on USDJPY or EURUSD from 2015 to 2024, the optimization starts but doesn't advance, only doing the first generation. When I do a simple backtest (no optimization) I found some errors (log attached) at the start of the backtest that say:

"real ticks absent for 12 minutes out of 24 total minute bars within a day" - "real ticks discarded for 12 minutes out of 24 total minute bars within a day" - "1156 tick prices mismatch for 12 minute bars"

This error disappears when I test from 2016 to 2022, and every other try that excludes the 2015 and 2023 years.

IMPORTANT: Looks like its a data problem from the 2015 and 2023 years buuut the WEIRD thing is that in the LOG FILES of the EURUSD and the USDJPY the ERROR DATES are exactly the same (from 2023/11/16 to 2023/12/07). Isn't that weird? Maybe theres a problem elsewhere and not in the data?

What do you guys think?