Hitting memory limitations

New comment
 

Hi everyone,

I've recently moved to futures trading using third party data. The issue I'm currently facing is, during optimizations where I test 6 instruments together for a few years (D1 timeframe, open price only), memory explodes.

For each instrument I'm running a back adjusted series where my indicators resides, and I only query the front/next contracts. But unfortunately, it seems there is no way to tell MT5 that I'm done with a contract when it expired (SymbolSelect(...,false) does nothing). So memory keeps growing on long backtests. I made sure I had no leaks, and that my dynamic allocations we're kept to a minimum. If I run without querying more than 2 contracts, I have no issues.

I'm thinking of maybe doing everything on the back adjusted series, but then I lose the capability to use Carry as a trading strategy, and I cannot reproduce the roll over mechanics.

I'm really open to any creative suggestions. My testing computer runs on win11, 74 core xeon with 140gb ram.

Comparative Analysis of 10 Trend Strategies
Comparative Analysis of 10 Trend Strategies
  • 2017.05.24
  • www.mql5.com
The article provides a brief overview of ten trend following strategies, as well as their testing results and comparative analysis. Based on the obtained results, we draw a general conclusion about the appropriateness, advantages and disadvantages of trend following trading.
 
Mike D-rock:


maybe you are using a beta version of mt5? several of the recent betas have had serious errors in both strategy tester AND metatester.

also nice system btw. however, my ea is rather basic, and yet all my 16 cores will often use up my full 128gb when testing 6 settings. so maybe -- and wow i hardly believe that i am saying this -- maybe 140gb is not enuf for what you are doing?

 
Mike D-rock:

Hi everyone,

I've recently moved to futures trading using third party data. The issue I'm currently facing is, during optimizations where I test 6 instruments together for a few years (D1 timeframe, open price only), memory explodes.

For each instrument I'm running a back adjusted series where my indicators resides, and I only query the front/next contracts. But unfortunately, it seems there is no way to tell MT5 that I'm done with a contract when it expired (SymbolSelect(...,false) does nothing). So memory keeps growing on long backtests. I made sure I had no leaks, and that my dynamic allocations we're kept to a minimum. If I run without querying more than 2 contracts, I have no issues.

I'm thinking of maybe doing everything on the back adjusted series, but then I lose the capability to use Carry as a trading strategy, and I cannot reproduce the roll over mechanics.

I'm really open to any creative suggestions. My testing computer runs on win11, 74 core xeon with 140gb ram.

Please provide more technical details about what you are doing. "Memory explodes" what that means ?

What data are you requesting by symbol ? How much symbols ? How much agents are you running ?

 
Alain Verleyen #:

Please provide more technical details about what you are doing. "Memory explodes" what that means ?

What data are you requesting by symbol ? How much symbols ? How much agents are you running ?


4 symbols, querying a total of ~420 contracts over a period of 20 years. Daily ticks only.

5gb ram per agents which prevents me from scaling properly on all my cores with 50+ permutations for grid search
 
Michael Charles Schefe #:

maybe you are using a beta version of mt5? several of the recent betas have had serious errors in both strategy tester AND metatester.

also nice system btw. however, my ea is rather basic, and yet all my 16 cores will often use up my full 128gb when testing 6 settings. so maybe -- and wow i hardly believe that i am saying this -- maybe 140gb is not enuf for what you are doing?

Do you trade futures and how many symbols do you include in your optimization per pass?

It seems to be ram needed by the backtester, the fact that i cannot unload past contracts once you touch the symbol with SymbolInfo* or CopyRates feels like a flaw.

I decided to add a parameter to decide if I use contracts or trade on the back-adjusted series when not needing contracts precision.


 
Mike D-rock #:
Do you trade futures and how many symbols do you include in your optimization per pass?

It seems to be ram needed by the backtester, the fact that i cannot unload past contracts once you touch the symbol with SymbolInfo* or CopyRates feels like a flaw.

I decided to add a parameter to decide if I use contracts or trade on the back-adjusted series when not needing contracts precision.


fx and 3 total. but only because i know that if i try more, then i will get errors. note that i think that 5G is ok for what you have described, however, many others will tell you that it is "out there". Have you tried shortening the period to 6 months to a year, and checking for server/trading errors in the logs? These are most often the cause of all "memory blowouts" in my experience.
 
Michael Charles Schefe #:
fx and 3 total. but only because i know that if i try more, then i will get errors. note that i think that 5G is ok for what you have described, however, many others will tell you that it is "out there". Have you tried shortening the period to 6 months to a year, and checking for server/trading errors in the logs? These are most often the cause of all "memory blowouts" in my experience.
I’m surprised you use your full 128gb for 3 fx pairs, with 3 symbols everything scales well, with 400+ it’s problematic if i want to use all the cores.

I spent a few hours revising all allocations (new+ArrayResize). No errors in consoles. No leaks. Everything is tight. Used memory deltas to see what calls cause constant ram growth. The moment you touch the contract symbol, even if it’s only to get latest rate with SymbolInfoTick, it will load all its data and never release it.
As you said if I test shorter periods it hides the problem, but since I’m using D1 timeframe, i need at least sample of 6 years for my specific case.
 
Mike D-rock #:
I’m surprised you use your full 128gb for 3 fx pairs, with 3 symbols everything scales well, with 400+ it’s problematic if i want to use all the cores.

I spent a few hours revising all allocations (new+ArrayResize). No errors in consoles. No leaks. Everything is tight. Used memory deltas to see what calls cause constant ram growth. The moment you touch the contract symbol, even if it’s only to get latest rate with SymbolInfoTick, it will load all its data and never release it.
As you said if I test shorter periods it hides the problem, but since I’m using D1 timeframe, i need at least sample of 6 years for my specific case.

Is it 6 years or 20 years ?

Anyway you will need to use less agents, or less...something. As there is currently no way to release memory with the Strategy Tester.

 
Mike D-rock #:
I’m surprised you use your full 128gb for 3 fx pairs, with 3 symbols everything scales well, with 400+ it’s problematic if i want to use all the cores.

my testing period of 4 - 6 years is what makes the memory jump above 65% of 128.
 
Mike D-rock:

I've recently moved to futures trading using third party data. The issue I'm currently facing is, during optimizations where I test 6 instruments together for a few years (D1 timeframe, open price only), memory explodes.

For each instrument I'm running a back adjusted series where my indicators resides, and I only query the front/next contracts. But unfortunately, it seems there is no way to tell MT5 that I'm done with a contract when it expired (SymbolSelect(...,false) does nothing). So memory keeps growing on long backtests. I made sure I had no leaks, and that my dynamic allocations we're kept to a minimum. If I run without querying more than 2 contracts, I have no issues.

I'm thinking of maybe doing everything on the back adjusted series, but then I lose the capability to use Carry as a trading strategy, and I cannot reproduce the roll over mechanics.

I'm really open to any creative suggestions. My testing computer runs on win11, 74 core xeon with 140gb ram.

I suppose you can try to create 2 custom symbols for every stack of specific instrument, that is combine odd contracts into the 1-st custom symbol and combine all even contracts into the 2-nd. Then you'll have only 2 symbols per instruments instead of total number of contracts. At any time your EA will have access to 2 contracts and use carry/rollover trade.
 
Stanislav Korotky #:
I suppose you can try to create 2 custom symbols for every stack of specific instrument, that is combine odd contracts into the 1-st custom symbol and combine all even contracts into the 2-nd. Then you'll have only 2 symbols per instruments instead of total number of contracts. At any time your EA will have access to 2 contracts and use carry/rollover trade.

That's actually pretty smart, thanks for the idea!

Michael Charles Schefe #:
my testing period of 4 - 6 years is what makes the memory jump above 65% of 128.

I'm surprised that longer backtest grows your memory significantly if you are only touching 3 symbols during your backtest. You might have a leak or something is constantly growing.

12
New comment