Theory of EA acceleration when using a custom indicator (function - iCustom)

[Aleksey Vyazmikin]

I will start by saying that I am not a programmer and I could be wrong, but I would like to hear a professional's opinion of the idea suggested below, based on calculations.

The use of custom indicators is a topical issue when developing an Expert Advisor in several stages. It is especially important when programmers replace each other, and then it is better to paste some of the logic of the Expert Advisor in the indicator - test the logic (to check the calculations and their relevance) and then work on a new part of the idea. As a result, we get a not very complex Expert Advisor that actually requests information from the indicator and makes a decision by comparing the expected and actual data.

But, this approach has a significant disadvantage - the speed of such an Expert Advisor. The fact is that the more often the data is requested from the custom indicators, the slower the calculation and more resources (memory) are required for optimization.

So far I work in MT4, but the principle of this problem, as I understand it, is the same.

And the problem is not in the calculation speed of the iCustom indicator itself, but in its connection with the Expert Advisor.

Suppose the indicator uses 5 chart buffers and the EA requires information from those buffers, then it needs to use 5 iCustom functions in its code and actually put 5 indicators on the chart instead of one indicator, which requires 5 times more resources. Maybe I'm wrong, and the compiler analyzes this circumstance - it calculates for one indicator and gives all functions the data for the required buffers at once? In that case my further speculations are vain.

But, if it's not so, why not to merge the information from the indicator into one pack?

I propose to make an experiment on this topic with measuring the performance of the Expert Advisor.

This will require taking a custom indicator with more than 1 buffer and adding an additional buffer.

The algorithm is logical (no mathematical):

1. Convert the buffers in the indicator to integers, depending on the digits per number, a total of 3 buffers, was: 1,21101; 1,13; 5, became: 121101;113;5

2. We count how many digits to put after the first number - in our case 4, then in the next number the next one - 1, these values are the degree of the multiplier:

1,21101*10^4=1211010000

1.13*10^1=113

5*10^0=5 (check for 0)

3. Sum the numbers and obtain 1211011135.

4. Write the value to the 4. buffer

5. We request the 4 indicator buffer in the Expert Advisor and decompose the value into components in reverse order and obtain 3 figures that can be further used for the work of the Expert Advisor.

Can someone compare the speed of this approach, is there a rationale behind it?

[Denis Kirichenko]

...Допустим, индикатор использует 5 графических буферов и для работы советника требуется информация с этих буферов, тогда в коде советника необходимо использовать 5 функций iCustom, и фактически вместо одного индикатора наносить на график 5 индикаторов, что требует в 5 раз больше ресурсов. Быть может я не прав и компилятор анализирует это обстоятельство - делая расчет по одному индикатору и дает сразу всем функциям данные по востребованным буферам!? Тогда дальше мои измышления пусты...

The compiler can do many things, but not that :-)

...But, this approach has a significant disadvantage - the speed of such an Expert Advisor. The fact is that the more frequently the data is requested from the custom indicators, the slower the calculation is and the more resources (memory) are needed for optimization ...

And this case can be optimized. First, the indicator itself should be written intelligently (not to recalculate all bars on every tick), second, the indicator should be super heavy, to somehow slow down the EA... To make a long story short...

And here is an interesting article on "Reducing memory consumption of auxiliary indicators".

I had a multicurrency Expert Advisor that received indicator readings for 28 symbols for 8 timeframes.

It was 28*8=224 copies of indicators... I struggled with how to optimize the process. The multicurrency operation consumed almost 700 MB of RAM... I solved it easily - I just set the "Max bars in window" field in terminal settings on the "Charts" tab to a small value... As far as I remember the minimum for this field is 5 thousand bars...

[Dmitry Fedoseev]

I measured the increase in testing time by a factor of 1.2 to 1.3, which is quite insignificant compared to the benefits given by the custom indicators.

The conclusion about 5 buffers is wrong. If parameters are the same, only one indicator will be loaded.

[Denis Kirichenko]

Integer:

The conclusion about the 5 buffers is incorrect. If parameters are the same, only one indicator will be loaded.

Yes, I agree.

There will be 5 calls of the CopyBuffer() function with different arguments (buffer number). And the indicator copy - handle - will be the same.

The author gave a loud title of the topic "Acceleration theory..." but he's clueless on the basic stuff.

-Aleks, there are many articles on this subject!

[Alexey Volchanskiy]

denkir:

Yes, I agree.

There will be 5 calls to CopyBuffer() function with different arguments (buffer number). And there will be only one copy of the indicator - handle.

The author gave a loud title of the topic "Acceleration theory..." but he's clueless on the basic stuff.

-Aleks-, there are a lot of articles on the indicator theme!

I tried to measure the call speed of built-in and analogous indices through iCustom once. Built-in ones are about 20-50% faster. Most likely because they are written in C++ and not in MQL.

[Aleksey Vyazmikin]

Denkir, I'm talking about the work of the EA during optimization, does the number of candlesticks on the chart affect the amount of calculations? I studied the article, I know that there are variants of optimizing indicators. However, in this case, I want to believe that the indicator is perfect and examine the method of passing the data from the indicator to the EA. I am not a programmer and find it hard to check the optimality of the indicator code - at most I can prescribe in TOR - 1 bar lag and limitation of history for calculations.

Integer:

I measured the increase of testing time in 1.2 - 1.3 times, which is absolutely insignificant compared to the benefits provided by the custom indicators.

My conclusion about 5 buffers is not correct. If the parameters are the same, only one indicator will be loaded.

What were you measuring? And 30% is not so little for me.

About the 5-buffers, if I understood you correctly, then with the same input parameters of the indicator, the calculation is performed for only one indicator when the Expert Advisor calls the last one multiple times and gets information from different buffers?

If it is so, then the combination of custom indicators should speed up the work of the Expert Advisor? For example, then in one indicator the logic of different indicators with independent settings can be implemented, when any of them is called, the information from all buffers is received at once and used in the case of information request from another buffer with the same parameters?

VDev:
I tried to measure the speed of calling of embedded indices and their analogs through iCustom. Built-in ones are about 20-50% faster. Most likely because they are written in C++ and not in MQL.

It's a fact.

[Denis Kirichenko]

Extremely interesting article "Price series averaging without additional buffers for intermediate calculations " by GODZILLA.

It was written back in 2010.

There is a section 3. Comparison of the obtained indicator using classes with its analogues with calls of technical and custom indicators in the code

[Aleksey Vyazmikin]

denkir:

Yes, I agree.

There will be 5 calls to CopyBuffer() function with different arguments (buffer number). And there will be only one copy of the indicator - handle.

The author loudly called the topic "Acceleration Theory..." and he's no clue whatsoever about the basic stuff.

-Aleks-, there are a lot of articles on the indicator theme!

And you can simply disprove my hypothesis (on 4, preferably), confirming my words with numbers?

The topic is about my theory - the title is ok :)

[Denis Kirichenko]

-Aleks-:

Denkir, I'm talking about the performance of the EA during optimization, does the number of candlesticks on the chart affect the amount of calculations there?

-Aleks-, about the tester You need to ask the developer ...

But I think there is a problem with the terms in the discussion. What do you want, to reduce the speed of calculations or to save RAM to work with the indicator?

[Aleksey Vyazmikin]

denkir:

-Aleks-, about the tester. You need to ask the developers...

But as I see it, there is a problem with the terms in the discussion. What do you want, to reduce the speed of the calculations or to save RAM to work with the indicator?

It seems to me that my method will solve both of these problems. I may be wrong.

When optimizing, speed is important, but once RAM is clogged, again, according to my observations, processing speed per pass decreases.

[Andrey Khatimlianskii]

-Aleks-:

Can anyone compare the speed of this approach, is there a rationale behind it?

This approach will reduce indicator's memory consumption (approximately a multiple of the difference in the number of buffers before and after), but will increase the load on the processor (both "assembling" and "decomposition" must be done constantly).

And if the indicator has 5-buffers, then the first call of iCustom will calculate all buffers, the subsequent calls and requests of other buffers will only read the ready information (until the appearance of new data for calculation).

If you have a really heavy indicator, think up your own cache system, so that at first call the data is calculated and written to the file, and at next calls it is read only. I've done it this way, it's much faster.

But in 95% of cases there's no need for all that, the tester is fast enough as it is, and in 5 you can also connect the cloud.

Good luck!