Discussion of article "MQL5 Cookbook - Creating a ring buffer for fast calculation of indicators in a sliding window"
Forum on Trading, Automated Trading Systems and Testing Trading Strategies
fxsaber, 2016.09.16 14:32
void OnStart() { double Index = -345.23; double Size = -432.98; double Array[]; // Any size (and not integer) will work. ArrayResize(Array, (int)Size < 0 ? (int)MathAbs(Size) : (int)Size); // At any index (and not integer) ALWAYS (except zero - if the array size is zero) will be executed without errors. // Thus the array becomes an infinite copy of itself in both directions Array[(int)Index < 0 ? ArraySize(Array) + ((int)Index % ArraySize(Array)) : (int)Index % ArraySize(Array)] = 1; }
int OnCalculate(const int rates_total, const int prev_calculated, const int begin, const double &price[]) { //--- bool calc = false; for(int i = prev_calculated; i < rates_total; i++) { Sma.AddValue(price[i]); buff[i] = Sma.SMA(); calc = true; } if(!calc) { Sma.ChangeValue(MaPeriod-1, price[rates_total-1]); buff[rates_total-1] = Sma.SMA(); } return(rates_total-1); }
int OnCalculate(const int rates_total, const int prev_calculated, const int begin, const double &price[]) { //--- if (prev_calculated < rates_total) for(int i = prev_calculated; i < rates_total; i++) { Sma.AddValue(price[i]); buff[i] = Sma.SMA(); } else { Sma.ChangeValue(MaPeriod-1, price[rates_total-1]); buff[rates_total-1] = Sma.SMA(); } return(rates_total-1); }
ZY If prev_calculated is zeroed, there will be errors.
Very good article on a very important topic.
Thering buffer is a crucial mechanism.
In addition to the article I would like to add my idea how the use of ring buffers can be developed further:
If, in parallel with recording the values of a parameter in a ring buffer, a second ring buffer is created for this parameter, which will record the time between the changes of values in the first buffer, a curve of the value changes in the current period can be constructed from the data of both ring buffers. Further, with the help of mathematical operations it is possible to extract the character of the parameter value change, i.e. its current signature, and to work not with specific values, but with the context of the parameter change for the whole current period.
P.S. It would be great if someone could implement this as an independent mechanism. Try it?
Thanks for the article! I have a few questions and caustic remarks )
По возможности избегайте запросов по получению данных множества таймфреймов. Вместо этого для расчетов воспользуйтесь одним (наименьшим) таймфреймом. Например, если вам требуется рассчитать два индикатора на M1 и H1, получите данные M1, сконвертируйте их в H1 и затем подайте эти данные для расчета индикатора на H1. Такой подход сложнее, но позволит существенно сэкономит память.
What is the saving due to?
The H1 time-series generated by the terminal takes more memory than the same time-series generated inside the Expert Advisor?
However, we still needed almost 3 GB of RAM. Is there any other way to reduce this figure? We can, if we optimise the number of timeframes. Let's try to change the testing code a bit and use only one timeframe instead of 21 - PERIOD_M1. The number of indicators will remain the same, just some of them will be duplicated:
Now the same 504 indicators in the internal calculation mode occupy 548 MB of RAM.
I don't understand this move at all. How can you compare the calculation of indicators for 21 timeframes with the calculation for 1 TF? The results of calculations are quite different, what difference does it make how much memory is used?
It is hard to find a more relevant application of ring buffers than in trading. It is all the more surprising that this data construction algorithm has not been covered in the MQL community until now.
Konstantin Gruzdev posted his class and some examples back in 2012. A search will find them.
In general, of course, the technique is good. One disadvantage is that all indicators need to be rewritten.
Thanks for the article! I have a few questions and caustic comments )
If possible, avoid queries to retrieve data from multiple timeframes. Instead, use one (smallest) timeframe for calculations. For example, if you need to calculate two indicators on M1 and H1, get the M1 data, convert it to H1 and then feed this data to calculate the indicator on H1. This approach is more complicated, but it will save memory.
What will savememory?
The H1 time-series generated by the terminal occupies more memory than the same time-series generated inside the Expert Advisor?
Unfortunately, yes. And much more. And it does not matter whether you have requested one bar or the whole available history. All data for the specified timeframe will be copied to the internal memory. I do not know exactly how much will be copied, but according to my memory measurements, it was seen that almost everything is copied.
However, we still needed almost 3 GB of RAM. Is there any way to reduce this figure? You can, if you optimise the number of timeframes. Let's try to change the testing code a bit and use only one timeframe instead of 21 - PERIOD_M1. The number of indicators will remain the same, just some of them will be duplicated:
Now the same 504 indicators in the internal calculation mode occupy 548 MB of RAM.
I don't understand this move at all. How can you compare the calculation of indicators for 21 timeframes with the calculation for 1 TF? The results of calculations are quite different, what difference does it make how much memory is used?
If you use only one smallest timeframe to calculate several indicators on different timeframes, you will save memory. Suppose there are two indicators, one counts values on M1 and the other on H1. We can load quotes for M1 and for H1. We can load quotes for each indicator and get values from them. However, only due to the fact that H1 will be loaded, the memory usage will increase a lot. Therefore, if we request M1, then convert M1 to H1 and feed this data to the indicator on H1, the memory will be significantly saved. This saving is achieved due to the fact that MetaTrader internal buffers allocate much more memory for storing H1 quotes than if these quotes were stored inside the Expert Advisor.
There is another interesting feature that was not mentioned in the article. The model of memory allocation in the strategy tester is different and much more economical. It allocates much less memory when multiple timeframes are used, but everything is calculated normally.
...
In general, there were three goals in writing the article:
- To create fast algorithms for calculations of indicators inside the Expert Advisor (completed).
- Create a convenient interface for calculations in the ring buffer (completed).
- Create a memory-saving calculation (not implemented).
Very good article on a very important topic.
The ring buffer is a crucial mechanism.
In addition to the article I would like to add my idea how the use of ring buffers can be developed further:
If, in parallel with recording the values of a parameter in a ring buffer, a second ring buffer is created for this parameter, which will record the time between the changes of values in the first buffer, a curve of the value changes in the current period can be constructed from the data of both ring buffers. Further, with the help of mathematical operations it is possible to extract the character of the parameter value change, i.e. its current signature, and to work not with specific values, but with the context of the parameter change for the whole current period.
P.S. It would be great if someone could implement this as an independent mechanism. Try it?
What you describe is just an indicator. Create a class, let's say CTradeChange. Inside it place two synchronous ring buffers: one stores N last prices, the other N last time values:
CTradeChange change;
...
change.Add(value, TimeCurrent());Next, in the Add method, calculate the difference between the current and previous time value.
In general, there were three goals in writing the article:
- Create fast algorithms for calculations of indicators inside the Expert Advisor (completed).
- Create a convenient interface for calculations in the ring buffer (completed).
- Create a memory-saving calculation (not implemented).
Thank you for your answers.
When developing the panel mentioned in the article, I came to the necessity to limit the number of bars displayed on the charts (5000). Precisely because of memory...
In MT4 it was much easier: firstly, each TF was loaded independently, so it didn't require loading M1, and secondly, indicator buffers took as much space as you fill them (even 100 bars, if you don't need more).
In 5, it has become more universal and integral due to generation of TFs from M1, but memory can be difficult.
What you describe is just an indicator. Create a class, let's say CTradeChange. Inside it place two synchronous ring buffers: one stores N last prices, the other N last time values:
Then in the Add method, calculate the difference between the current and previous time value.
You gave a strange answer. It seems that you have not understood the idea at all. I will quote it again:
"If parallel with the recording of parameter values in the ring buffer, create for this parameter a second ring buffer, which will record the time between the changes of values in the first buffer, then on the basis of data from both ring buffers you can build a curve of value changes in the current period. Further, using mathematical operations, you can extract the nature of the change in the parameter value- i.e. its current signature, and work not with specific values, but with the context of the parameter change over the entire current period ."
Note, I did not ask how to build two synchronous ring buffers, as their mechanism is quite simple, but suggested to try to develop the scope of application of ring buffers further than described in the article.
The area of application proposed by you in the article is obtaining specific values by indices inside the current period.
My proposed extension of the area is to obtain signatures of changes in values within the current period. This can be done by plotting a curve (not on a graph) based on data from two ring buffers:
1. buffer with values of the current period.
2. buffer with time intervals between the values of the first buffer.
By combining the data it is possible to plot the curve mathematically (not necessarily on a graph) and algorithmically represent and study it inside the programme. To do this, it is necessary to scan both buffers and consider the signature of the parameter change.
This solution opens up the possibility to operate not only with specific values of the parameter within a period, but with the signature of its change over the entire period.
For example, it will be possible to specify to the programme:
if(Характер_изменения_значения_параметра_за_период == BIG_WAVE)Лот += 10;
The constant BIG_WAVE is the signature, which expresses the character of the value change for the current period.
For example, we can build 5 patterns of signatures:
FLAT, RISING, BIG_WAVE, FALLING, SMALL_WAVES.
Each of these constants is a pattern of a certain character of the parameter change within the current period.
We need to develop the format of the signature record, and create these templates. Then, the algorithm will read the current signature and compare it with the templates, finding the closest matches (there can be no absolute match) between the nature of the current change and one of the templates.
The advantages of this approach over the standard use of specific values are obvious.
The use of signatures makes it possible to base decisions on the context extracted from the data set and not try to extract that context from single values.
P.S. I hope you understood my point this time.
That's a strange answer you gave. You don't seem to have understood the point at all. I'll quote it again:
...
My proposed extension of the domain is to obtain signatures of changes in values within the current period. This can be done by plotting a curve (not on a graph) based on the data from the two ring buffers:
P.S. I hope you understood my point this time.
I understood your point perfectly well the first time too.
What is a "value change signature"? It is some value changing in dynamics. Therefore it is an indicator. It is not necessary to develop a ring buffer for this purpose, but it is enough just to create on the basis of several of these ring indicators an algorithm for calculating the very "character of change" you are talking about.
- Free trading apps
- Over 8,000 signals for copying
- Economic news for exploring financial markets
You agree to website policy and terms of use
New article MQL5 Cookbook - Creating a ring buffer for fast calculation of indicators in a sliding window has been published:
The ring buffer is the simplest and the most efficient way to arrange data when performing calculations in a sliding window. The article describes the algorithm and shows how it simplifies calculations in a sliding window and makes them more efficient.
At the start of the calculation, the indicator simply adds new values to the ring buffer of the moving average. You do not have to control the number of added values. All calculations and removals of obsolete elements occur automatically. If the indicator is called when changing the price of the last bar, the last moving average value should be replaced with a new one. The ChangeValue method is responsible for that.
The graphical display of the indicator is equivalent to the standard moving average:
Fig. 1. The simple moving average calculated in the ring buffer
Author: Vasiliy Sokolov