How to release memory after massive CopyRates()? - page 4

[Dave Bieleveld]

I currently have the same issue. I'm loading 900 symbols, and terminal64.exe eats as much memory as available. That's 47 gigabytes in my case.

I've found that it is not a critical issue, because the memory will be swapped to the pagefile when appropriate.

Still, I would like to know how to free up the cache (which is a good thing) manually, which allows better resource management for my programs.

(edit: I monitor memory-/and disk usage with the resource monitor during loading)

(edit2: I also use spacesniffer to watch how the history is used. Only after closing the terminal, terminal64.exe remains active for a while, dumping it's cache to the history files, causing the history files to grow in size. This possibly indicates that the data in memory is not persistent until closing the terminal. Which is not a favorable situation in case the terminal closes due to abnormal termination.)

[Alain Verleyen]

Dave Bieleveld #:

I currently have the same issue. I'm loading 900 symbols, and terminal64.exe eats as much memory as available. That's 47 gigabytes in my case.

I've found that it is not a critical issue, because the memory will be swapped to the pagefile when appropriate.

Still, I would like to know how to free up the cache (which is a good thing) manually, which allows better resource management for my programs.

(edit: I monitor memory-/and disk usage with the resource monitor during loading)

(edit2: I also use spacesniffer to watch how the history is used. Only after closing the terminal, terminal64.exe remains active for a while, dumping it's cache to the history files, causing the history files to grow in size. This possibly indicates that the data in memory is not persistent until closing the terminal. Which is not a favorable situation in case the terminal closes due to abnormal termination.)

It's unfortunately not possible. The cache is cleared after 30 minutes.

[Manuel Gonzales]

Hello, good day, I have the same problem with "CopyRates" but this time taken to metatester.ex5

On the host, ram usage is normal and is released appropriately.

but on the servers, "extra agents" the ram gets stuck.

I'm on the latest build.

Host:

 

Local Farm:

[Witold Mergist]

Alain Verleyen #:
It's unfortunately not possible. The cache is cleared after 30 minutes.

I also have the same problem and any other "trader/programmer" will have the same problem as well.

This looks like a deliberate move by MetaQuotes to prevent users from doing "mass" calculations.

Creating a function to clean up the EA memory is not a problem. Such a function could be called from the program when the symbol changes, which would cause the memory to be almost empty all the time.

Statements in the documentation such as: "To save resources, data on a timeframe are stored and saved in RAM only if necessary. If not called for a long time..." mean nothing and are senseless.

What does "if necesary...." mean? "Necessary" for whom? I guess the user/programmer knows better than MetaQuotes when he no longer needs historical data in RAM for a particular symbol. True...?

Moreover, "...for a long time..." also makes no sense. For me, the data storage time in RAM  ends when the symbol changes and I DO NOT NEED TO KEEP THEM IN RAM "...for a long time...". Let them go to HDD, I've enough terabytes on it.

The existence of this problem for so long proves that MetaQuotes acted on purpose, for the purposes I wrote about at the beginning.

By the way: my RAM didn't clean itself even after 40-50 minutes of stopping CopyTimes(...) and I'm not going to prove it. It's just as I wrote.

[Flohti]

Hi there,

I've just learn that fact,

3182 symbols, a CopyRates of 50 days for each, even not saved at all in any variable, stay in memory and do some process for that apparently,

Loaded approximately 5 Gig RAM and about 20% use on all 32 cores,

What is that ?

And also about 10 15 seconds to release memory on closing MT5, before closing, fans run nosily of course

Since, if there anyway to discharge that noising thing ?

[Flohti]

I've Just tested something :

#property indicator_chart_window
#property indicator_buffers 8
#property indicator_plots 8

#property indicator_label1 "time"
#property indicator_type1 DRAW_NONE

#property indicator_label2 "open"
#property indicator_type2 DRAW_NONE

#property indicator_label3 "high"
#property indicator_type3 DRAW_NONE

#property indicator_label4 "low"
#property indicator_type4 DRAW_NONE

#property indicator_label5 "close"
#property indicator_type5 DRAW_NONE

#property indicator_label6 "tick"
#property indicator_type6 DRAW_NONE

#property indicator_label7 "volume"
#property indicator_type7 DRAW_NONE

#property indicator_label8 "spread"
#property indicator_type8 DRAW_NONE

double Times[], Opens[], Highs[], Lows[], Closes[], Ticks[], Volumes[], Spreads[];

void OnInit () {
        IndicatorSetString (INDICATOR_SHORTNAME, "CustomRates");
        SetIndexBuffer (0, Times, INDICATOR_DATA);
        SetIndexBuffer (1, Opens, INDICATOR_DATA);
        SetIndexBuffer (2, Highs, INDICATOR_DATA);
        SetIndexBuffer (3, Lows, INDICATOR_DATA);
        SetIndexBuffer (4, Closes, INDICATOR_DATA);
        SetIndexBuffer (5, Ticks, INDICATOR_DATA);
        SetIndexBuffer (6, Volumes, INDICATOR_DATA);
        SetIndexBuffer (7, Spreads, INDICATOR_DATA);
}

void OnDeinit (const int reason) {
}

int OnCalculate (const int rates_total, const int prev_calculated, const datetime &time[], const double &open[], const double &high[],
                const double &low[], const double &close[], const long &tick_volume[], const long &volume[], const int &spread[]) {
        ArrayCopy (Times, time);
        ArrayCopy (Opens, open);
        ArrayCopy (Highs, high);
        ArrayCopy (Lows, low);
        ArrayCopy (Closes, close);
        ArrayCopy (Ticks, tick_volume);
        ArrayCopy (Volumes, volume);
        ArrayCopy (Spreads, spread);
        return rates_total;
}

And 

int s, S = SymbolsTotal (false), handle;
for (s = 0; s < S && !IsStopped (); s++) {
        datetime times[];
        symbol = SymbolName (s, false);
        handle = iCustom (NameSymbol, PERIOD_D1, "Flohti\\CustomRates");
        CopyBuffer (handle, 0, 0, 50, times);
        IndicatorRelease (handle);
}

Same Result,

IndicatorRelease Should do the job, but seems just don't ? Am I wrong ?

Very not normal, annoying 

edit : even without CopyBuffer, same

[Ryan L Johnson]

Flohti #:
IndicatorRelease Should do the job, but seems just don't ? Am I wrong ?

Regarding IndicatorRelease(), "The function removes an indicator handle and releases the calculation block of the indicator, if it's not used by anyone else."

Applying that statement to the letter, your buffers are indirectly defined by your constants in OnCalculate(). To what extent OnCalculate() is "anyone," I have no idea. Possibly. Maybe "anything" would've been a better word.

[Ryan L Johnson]

Witold Mergist #:
This looks like a deliberate move by MetaQuotes to prevent users from doing "mass" calculations...

The existence of this problem for so long proves that MetaQuotes acted on purpose, for the purposes I wrote about at the beginning.

Oh yes. Of course. All of that Python support, DLL access, native machine learning functions, and writing to files is obviously there to prevent mass calculations. 🤦‍♂️

Also...The Earth is flat, and traffic laws don't apply to me because I'm not driving--I'm traveling as sovereign citizen.

[Flohti]

So the indicator I've created to test, the OnCalculate block in general, do exactly the same thing as the functions CopySomething, read historical data and keep it in RAM, to access it more rapidly I suppose,

My problem is that I need to read massive information on the market OnInit, I don't need to read that datas anymore, I don't need keeping that RAM && CPU usage, mostly that f.. CPU usage, 20% 32 threads 4GHz CPU , seriously ?

I have a mostly capable machine, I don't appreciate that, even little, fan loudness when doing nothing I've intended

So my solution to what I want to do is having personal data saving, read write files, I've already done thing like that, it's fast enough for my need,

If anyone knows something about leaving CopySomething unused anymore datas from RAM and CPU usage, please don't let us ignorant

[Flohti]

Ok, I was wrong,

I've just tested only 1 candles asked for each 3000+ symbols, and it loads exactly the same amount of data in RAM,

That means that it loads The history file of the symbol and that's it, that means that we should be able to do more things with the historical datas without charging more RAM and CPU usage since it should already be done..

So writing and reading personal historical files won't relieve CPU and RAM usage if I want to update datas,

I eventually can load my market scanner on an instance of Metatrader and trade on another, and close the scanner when I don't need anymore, eventually,

I'm now accustomed on Metatrader "things" that has to be taken into account,

I would do that unless someone know how to deal better with that CPU RAM usage and share his informations,

Thanks in advance