How to release memory after massive CopyRates()? - page 5

New comment
 
Flohti #:

Hi there,


I've just learn that fact,

3182 symbols, a CopyRates of 50 days for each, even not saved at all in any variable, stay in memory and do some process for that apparently,

Loaded approximately 5 Gig RAM and about 20% use on all 32 cores,


What is that ?


And also about 10 15 seconds to release memory on closing MT5, before closing, fans run nosily of course


Since, if there anyway to discharge that noising thing ?

It's not clear what are you requesting as data to use that much memory ?

MqlRates is 60 bytes, 50 days will be 3000 Bytes. It's nothing. Please provide code to reproduce your issue.

We are talking about a code running on a live chart right ? Not with the Tester ?

 
Alain Verleyen #:

It's not clear what are you requesting as data to use that much memory ?

MqlRates is 60 bytes, 50 days will be 3000 Bytes. It's nothing. Please provide code to reproduce your issue.

We are talking about a code running on a live chart right ? Not with the Tester ?

Hi,

It's not MqlRates that takes ressources, it's history data loaded when requesting, if you have unlimited amount of bars set in your parameters like I had, yes I've change now, You will load a lot of historical data in RAM, even if you request only one MqlRates or other type of data,

The code is the simpler possible, a function that do a loop in all symbols, a request of anything about history, MqlThing locally declared in the function, so discharged when function is done..

I haven't kept the code, it's something like the one I've posted earlier with a CopyRates in place of the CopyBuffer..

Yes on a live chart, if you want to test, do that simplest piece of code with unlimited bars, you will see,

I'm actually coding, I haven't really seen if setting to 100 000 bars has resolved the problem, I think probably a little, maybe I will have to delete all the historical files too, I'll see

 
Flohti #:

I eventually can load my market scanner on an instance of Metatrader and trade on another, and close the scanner when I don't need anymore, eventually,

I'm now accustomed on Metatrader "things" that has to be taken into account,

I would do that unless someone know how to deal better with that CPU RAM usage and share his informations

CopyTicks does not have such a memory problem.
 
fxsaber #:
CopyTicks does not have such a memory problem.

Not that quite,

Note that CopyTicks gives you 2 000 ticks only, you have to do a loop to take more,

I've just tried loading all the ticks for 50 days, that take 20+Gio RAM, without keeping any data in execution,

Even downloaded before, it's time consuming so much,

But, Yes, no CPU usage, only RAM, a lot of RAM for loading 50 days of 3000+ symbols, and I repeat, I don't have kept the MqlTick returned by the CopyTicks function

int s, S = SymbolsTotal (false), t, T;
ulong from_time = (TimeCurrent () - 70 * 24 * 60 * 60) * 1000; // Yes it is 70 days for crypto, don't care, not that much a difference
string symbol;
MqlTick ticks[];
for (s = 0; s < S && !IsStopped (); s++) {
        symbol = SymbolName (s, false);
        T = CopyTicksRange (symbol, ticks, COPY_TICKS_ALL, from_time); // goes very faster when ticks already in RAM
        if (T < 1) // but sometime does not work
                while (!IsStopped ()) {
                        T = CopyTicks (symbol, ticks, COPY_TICKS_ALL, from_time);
                        if (T > 0) {
                                for (t = 0; t < T; t++)
                                if (ticks[t].time_msc > from_time) // because my broker has sent me some older ticks sometimes, why ?
                                from_time = ticks[t].time_msc + 1;
                                continue;
                        }
                        break;
                }
}

This is approximately what I've done,

Alain Verleyen if you want a copy-past for a Init block, this is the same thing

int s, S = SymbolsTotal (false);
string symbol;
MqlRates rates[];
for (s = 0; s < S && !IsStopped (); s++) {
        symbol = SymbolName (s, false);
        CopyRates (symbol, PERIOD_D1, 0, 50, rates);
        // don't do anything else, you don't care
}
 
Flohti #:

Note that CopyTicks gives you 2 000 ticks only, you have to do a loop to take more,

Read the documentation.

I've just tried loading all the ticks for 50 days, that take 20+Gio RAM, without keeping any data in execution,

Even downloaded before, it's time consuming so much,

But, Yes, no CPU usage, only RAM, a lot of RAM for loading 50 days of 3000+ symbols, and I repeat, I don't have kept the MqlTick returned by the CopyTicks function

The market scanner doesn't need to store data for all symbols simultaneously. Moreover, if you need bars, you can generate them from the received ticks.

 
fxsaber #:

Read the documentation.


"Not more than 2000"

do a Print (T)

 
fxsaber #:

The market scanner doesn't need to store data for all symbols simultaneously. Moreover, if you need bars, you can generate them from the received ticks.

So, why I want to read historical datas for, eventually 10 or 20 or 50 days, because I want to calculate relative volume for each symbol with my scanner

Finally, I don't do all the calculation for all symbols, I do that for these that are in the time range of their market morning opening hours principally,

And I have to consider that graphical objects to display all the informations take some CPU usage to, much more than keeping historical data in RAM eventually, that's other thing, so much things to consider in coding for metatrader,

Even with a so capable machine, MetaTrader work very badly, it is so incredible,

Massive calculation, massive RAM occupation with massive PPO, matrix, even big arrays, not a problem,

Some graphical objects, historical datas, what else, I can't remember now, so bad at that,

 
Flohti #:

"Not more than 2000"

do a Print (T)

Read better :

If the from and count parameters are not specified, all available recent ticks (but not more than 2000) will be written to ticks_array[].
You can get a lot more ticks at once.
 

Flohti #:

...

Alain Verleyen if you want a copy-past for a Init block, this is the same thing

This is what I said. An array of 50 days MqlRates is 3000 bytes. Even using M1 data for 50 days you are around 5 MB by symbols. So 15 GB for 3000 symbols.

The bars data are cached, but the cache is release after 30 minutes, you can play with that.

And additionally working with ticks give you a way to work without the cache, though it's a bit more complicated to code.

MT5, though not perfect, works very well.

 
Alain Verleyen #:

This is what I said. An array of 50 days MqlRates is 3000 bytes. Even using M1 data for 50 days you are around 5 MB by symbols. So 15 GB for 3000 symbols.

The bars data are cached, but the cache is release after 30 minutes, you can play with that.

And additionally working with ticks give you a way to work without the cache, though it's a bit more complicated to code.

MT5, though not perfect, works very well.

I've just finished coding my tool, that seems to work not that bad, I'll see more when I'll have quotation on european markets and this afternoon on american's too


Have you tried Print the number of ticks returned by the function, I have, with the from parameter set, it returns 2000 ticks, CopyTicksRange, if returns effectively something return all the ticks required, not CopyTicks, I had an all-in-one function to do the job in all circumstance, CopyTicksRange first, if no ticks, a loop with CopyTicks, and in that loop, from and to was set, and only 2000 ticks was returned per loop, it is said in the doc "not more than 2000", it's not specific to that case on not specifying from and count, or to_msc


Thanks for talk

123456
New comment