Memory Waste by indicators - how can I prevent???

New comment
 

Hi everybody,


let us have timeframe M1 and the following simple indicator:

#property indicator_buffers 1
double What_a_waste[];
int OnInit() {
  SetIndexBuffer (0, What_a_waste);
  return 0;
}
int OnCalculate(const int rates_total,
                const int prev_calculated,
                const datetime &time[],
                const double &open[],
                const double &high[],
                const double &low[],
                const double &close[],
                const long &tick_volume[],
                const long &volume[],
                const int &spread[]) {
  printf ("rates_total = %d, prev_calculated = %d, ArraySize(What_a_waste) = %d, first bar %s",
          rates_total, prev_calculated, ArraySize(What_a_waste), TimeToString(time[0]));
  return(rates_total);
}


When applied to a GBPUSD M1 chart, I get the following output:

2018.10.31 14:53:06.236 MemoryWaste (GBPUSD,M1) rates_total = 6966026, prev_calculated = 0, ArraySize(What_a_waste) = 6966026, first bar 1993.05.12 00:00
2018.10.31 14:53:06.564 MemoryWaste (GBPUSD,M1) rates_total = 6966026, prev_calculated = 6966026, ArraySize(What_a_waste) = 6966026, first bar 1993.05.12 00:00
2018.10.31 14:53:06.751 MemoryWaste (GBPUSD,M1) rates_total = 6966026, prev_calculated = 6966026, ArraySize(What_a_waste) = 6966026, first bar 1993.05.12 00:00
2018.10.31 14:53:06.959 MemoryWaste (GBPUSD,M1) rates_total = 6966026, prev_calculated = 6966026, ArraySize(What_a_waste) = 6966026, first bar 1993.05.12 00:00
2018.10.31 14:53:07.162 MemoryWaste (GBPUSD,M1) rates_total = 6966026, prev_calculated = 6966026, ArraySize(What_a_waste) = 6966026, first bar 1993.05.12 00:00

The array of What_a_waste is huge!! 

I'm not interested in more than 6 millions values but only in the last say 100. Otherwise it is a big waste of memory especially if you have several buffers within an indicator or several indicators.


What can I do?

Thanks


Matthias

 
Dr Matthias Hammelsbeck:

I'm not interested in more than 6 millions values but only in the last say 100.

Well, it would seem you have a lot of M1 bars in your history for the GBPUSD instrument. You could delete that history and see how much MT5 downloads. It might be much less than 6,966,026 bars. This isn't a great solution, but it is expedient.

A better way is to put some sort of "maximum # of bars" governor in your code. Perhaps something like this:

int OnCalculate(...)
{
    //--- the main loop of calculations
    int numBars = rates_total - 1;
    if ( numBars > 100 ) { numBars = 100; }
    for(int i=pos;i<numBars && !IsStopped();i++)
    {
    }

    return (numBars);
}

When limiting in this way, I think you need all your buffers in reverse direction, i.e.

ArraySetAsSeries(buffer, true);

 

Tools/Options/Max bars in chart = 5000

This should limit the bars to 50-100 000.

100 000 bars are ~2MB of RAM and 0.1 CPU load for 1/10 of a second once in a while to refresh them, so don't overthink it.

 
Dr Matthias Hammelsbeck:

Hi everybody,


let us have timeframe M1 and the following simple indicator:


When applied to a GBPUSD M1 chart, I get the following output:

The array of What_a_waste is huge!! 

I'm not interested in more than 6 millions values but only in the last say 100. Otherwise it is a big waste of memory especially if you have several buffers within an indicator or several indicators.


What can I do?

Thanks


Matthias

Limit the number of bars on your chart as noted by kypa.

Or don't use your array as an indicator buffer. A buffer always size the number of bars in a chart.

 

Hi Anthony and kypa,

thanks for your answers.


My point is not to limit the number of iterations (though important of course) but to reduce the usage of memory by the buffers defined within the code for the indicator.

The memory for the buffer What_a_waste seems to be reserved at the first call of

OnCalculate


Do I have any influence on this?

kypa: I set Tools/Options/Max bars to 5000.

After restart I get better results , nevertheless a little bit strange. Applying the indicator to GBPUSD M1 gives the following output:

2018.10.31 16:47:07.167 MemoryWaste (GBPUSD,M1) rates_total = 5001, prev_calculated = 0, ArraySize(What_a_waste) = 5001, first bar 2018.10.26 07:13
2018.10.31 16:47:07.481 MemoryWaste (GBPUSD,M1) rates_total = 5001, prev_calculated = 5001, ArraySize(What_a_waste) = 5001, first bar 2018.10.26 07:13
2018.10.31 16:47:07.575 MemoryWaste (GBPUSD,M1) rates_total = 5001, prev_calculated = 5001, ArraySize(What_a_waste) = 5001, first bar 2018.10.26 07:13
2018.10.31 16:47:07.684 MemoryWaste (GBPUSD,M1) rates_total = 5001, prev_calculated = 5001, ArraySize(What_a_waste) = 5001, first bar 2018.10.26 07:13
2018.10.31 16:47:08.074 MemoryWaste (GBPUSD,M1) rates_total = 5001, prev_calculated = 5001, ArraySize(What_a_waste) = 5001, first bar 2018.10.26 07:13

That is perfect :-)


But applying the indicator to EURUSD M1 gives:

2018.10.31 16:48:36.820 MemoryWaste (EURUSD,M1) rates_total = 372623, prev_calculated = 0, ArraySize(What_a_waste) = 372623, first bar 2017.10.31 00:00
2018.10.31 16:48:36.962 MemoryWaste (EURUSD,M1) rates_total = 372623, prev_calculated = 372623, ArraySize(What_a_waste) = 372623, first bar 2017.10.31 00:00
2018.10.31 16:48:37.664 MemoryWaste (EURUSD,M1) rates_total = 372623, prev_calculated = 372623, ArraySize(What_a_waste) = 372623, first bar 2017.10.31 00:00
2018.10.31 16:48:37.884 MemoryWaste (EURUSD,M1) rates_total = 372623, prev_calculated = 372623, ArraySize(What_a_waste) = 372623, first bar 2017.10.31 00:00
2018.10.31 16:48:38.492 MemoryWaste (EURUSD,M1) rates_total = 372623, prev_calculated = 372623, ArraySize(What_a_waste) = 372623, first bar 2017.10.31 00:00
2018.10.31 16:48:38.570 MemoryWaste (EURUSD,M1) rates_total = 372623, prev_calculated = 372623, ArraySize(What_a_waste) = 372623, first bar 2017.10.31 00:00
2018.10.31 16:48:39.196 MemoryWaste (EURUSD,M1) rates_total = 372623, prev_calculated = 372623, ArraySize(What_a_waste) = 372623, first bar 2017.10.31 00:00


That is a funny output, is'nt it?

The question remains: how can I limit the memory usage in a reliable manner? Best with software statements?


Matthias

 
Alain Verleyen:

Limit the number of bars on your chart as noted by kypa.

Or don't use your array as an indicator buffer. A buffer always size the number of bars in a chart.

Alain: thank you for your answer. 

Now my idea for a solution of the problem is the following:

  • explicitly reserve memory for buffers used for internal calculations
  • usage of indicator buffers only for graphical display and for accessing and evaluation by external programs (e.g. EAs)
  • limiting the number of bars in the chart by Tools/Options/Max bars in chart = 5000 (that seems to be the minimum)


Matthias


 
Dr Matthias Hammelsbeck:

My point is not to limit the number of iterations

Ah, right. My answer addressed this and not your original point on memory.

Looks like you are on the right track now.

New comment
Reason: