How to release memory after massive CopyRates()? - page 3

New comment
 
terminalstat:
anyone?

I would say that your assumption that you can not free the cache, is partly right.
You can not explicitly free it, that's true. 

But, according to documentation, the cache is freed and written to disk after a "long" time where not used:

"To save resources, data on a timeframe are stored and saved in RAM only if necessary. If not called for a long time, they are released from RAM and saved into a file. For each timeframe, data are prepared regardless of whether there are ready data for other timeframes or not. Rules of forming and accessing data are the same for all timeframes. I.e., despite the fact that the unit data stored in HCC is one minute (M1), the availability of HCC data doesn't mean the availability of data on M1 timeframe as HC in the same volume."

Thus, I guess If you add a large enough sleep before the last printing of free memory remained in your script, you will see it was freed (or just wait after running the first time - and watch the memory in task manager until it reduces). That way you can find the time limit and run the scan in time intervals that fits your overall memory. I tried with 15 minutes, and was partly freed (some 19 MB). 
It might also be a good idea to close any open charts of any of those symbols requested to cache while waiting it to be freed.

It will be a nice feature if MT would add a possibility to request that for a symbol.


*-edit: As to the "only if necessary" - Am not sure what it means and what happened or caused it but I closed the charts and then ran it again with 500 symbols over 101000 history bars, and the memory was not influenced at all (taskbar shows MT using 231 MB in the start and in the end of the script, and was not changed the whole running time) - although the script says available memory was reduced.

 
Amir Yacoby:

I would say that your assumption that you can not free the cache, is partly right.
You can not explicitly free it, that's true. 

But, according to documentation, the cache is freed and written to disk after a "long" time where not used:

"To save resources, data on a timeframe are stored and saved in RAM only if necessary. If not called for a long time, they are released from RAM and saved into a file. For each timeframe, data are prepared regardless of whether there are ready data for other timeframes or not. Rules of forming and accessing data are the same for all timeframes. I.e., despite the fact that the unit data stored in HCC is one minute (M1), the availability of HCC data doesn't mean the availability of data on M1 timeframe as HC in the same volume."

Thus, I guess If you add a large enough sleep before the last printing of free memory remained in your script, you will see it was freed (or just wait after running the first time - and watch the memory in task manager until it reduces). That way you can find the time limit and run the scan in time intervals that fits your overall memory. I tried with 15 minutes, and was partly freed (some 19 MB). 
It might also be a good idea to close any open charts of any of those symbols requested to cache while waiting it to be freed.

It will be a nice feature if MT would add a possibility to request that for a symbol.


Wait 15 minutes ? While it's interesting to know, I would not call that a way to free the cache.
 
Amir Yacoby:


*-edit: As to the "only if necessary" - Am not sure what it means and what happened or caused it but I closed the charts and then ran it again with 500 symbols over 101000 history bars, and the memory was not influenced at all (taskbar shows MT using 231 MB in the start and in the end of the script, and was not changed the whole running time) - although the script says available memory was reduced.

Only 231 MB ? So that's suppose to be after the data for 500 symbols were cached ?

For only 70 symbols (first run) I am already at 358.9 MB.

 
Alain Verleyen:

Only 231 MB ? So that's suppose to be after the data for 500 symbols were cached ?

For only 70 symbols (first run) I am already at 358.9 MB.

*-edit2: deleted previous edit, it seems something with closing the MT did not remove it from task bar and somehow the task bar froze with regards to MT5 memory. I now remove the task of MT from taskbar and it does not happen again. 
You are right, this is what I get as well.
 
Alain Verleyen:
Please read the topic, it's not about the Strategy Tester.


Does the script run like an EA test with a data caching mechanism that automatically frees up memory after a period of non-use?

I just feel like it's possible, but I'm not sure.

I simply expressed my own opinion, may I ask how this is offensive to you?


from:  terminalstat 2019.11.29 21:15 #5 

I believe (but definitely not sure) this 1Gb is occupied by copied data, stuck in memory. Next script run would peform very fast. Looks like cache.


He is not sure this kind of situation is really related to the use of CopyRates() function.

Then I mentioned that the caching mechanism of the tester is completely independent of the use of the function.

Did it seem to you that I had said something unforgivable?

Why do you always try to use your own judgment standard to deprive other people's right to be heard?

 
Alain Verleyen:
Wait 15 minutes ? While it's interesting to know, I would not call that a way to free the cache.
Definetly not. And maybe that taskbar issue messed my testing with 15 min because now it stays the same. 
Though it's interesting to know the definition of "long time" and "placed into RAM only if needed" means. 
 
Amir Yacoby:
Definetly not. And maybe that taskbar issue messed my testing with 15 min because now it stays the same. 
Though it's interesting to know the definition of "long time" and "placed into RAM only if needed" means. 

Sure.

I am using TerminalInfoInteger(TERMINAL_MEMORY_USED), which report weird value of 672 when the Windows task manager report 374.5 MB

But it shows well the increase due to cached data from 333 at the start to 672 after 70 symbols.

 
Sadly, looks like my only choice is to use CopyRates() for some group of symbols, then terminal restart and then proceed with next group of symbols.
 

Hi, everybody. Sorry for reviving this old thread, but I'm facing the same problem here.

I loop over more than 15k symbols and, after preliminary checks, usually my EA selects about 200 symbols and execute CopyTicks from last day until now on it.

The final memory usage is always about 1.8-2.0 GB. Is there any updates on freeing CopyTicks memory cache?

Thanks,

Daniel

 

Hi Community.  

I am using CopyTicks to create my own custom charts. 

int received=CopyTicks(sym,ticks,COPY_TICKS_ALL,1,oneMillion);

I have over 1000 symbols with tickhistory databases updating real time(MarketWatch).  I need to update the custom charts that I don't have open once a day which loops over the SymbolTotal and creates the bars. This all works fine. However, as this topic details, is that we face an issue with cached memorythat doesn't release unless we close the platform down.  After closing the platform down, it takes about 1 minute to release the 10GB of ram and puts 99% load on SSD. 


All rates[] arrays are ArrayFree at each loop interval.  All ticks[] arrays are ArrayFree and resized depending on the amount of ticks exist in the CopyTicks function. I've also tried the ZeroMemory after each internal.

Files:
ouyx6kKICK.gif  100 kb
1234
New comment