Optimization slow when using custom indicator

[Pasi Hakamaki]

Hi,

Is it typical that optimization gets very slow when using a custom made indicator? I made a simple indicator and use it in my EA and the optimization is soo slow :( I also made EA that uses two simple moving averages and optimizing it is much faster. My custom indicator is not complex and I've done some profiling and figured out that it should have no bottle necks. I've done the same with my EA but still so slow optimization.

Br, Candles

[Jinsong Zhang]

i think the problem is from your indicator

[phi nuts]

song_song:
i think the problem is from your indicator

... and so,  Candles, re-write the code.

For example, some indicators are using Copy function. For the first tick, it is OK to copy the whole bars on chart, but for the next tick, just copy the most current data. 

[Pasi Hakamaki]

phi.nuts:

... and so,  Candles, re-write the code.

For example, some indicators are using Copy function. For the first tick, it is OK to copy the whole bars on chart, but for the next tick, just copy the most current data. 

Yes, I was thinking about that last night. My indicator copies rates from the whole chart i.e. all the bars every time OnCalculate() is called. That has got to be the reason. Will fix it NOW.

Br, Candles

[Pasi Hakamaki]

Did not fix it right away, but now getting there still have some problem see below:

   if(rates_total>prev_calculated)
     {
      //do calculations for whole chart again here :(
     }

Now the optimization is fast as it should, great :)  The below code should be even faster as it only does calculations for the current new bar but still strangely this did not work:

   if(rates_total>prev_calculated)
     {
      for(int i=prev_calculated;i<rates_total;i++)
        {
         //do calculations for current bar only, here :)
        }
      }

So now the expert still repeats the calculation for the whole chart when a new bar comes because I must use this:

if(rates_total>prev_calculated) //if we have new bar
     {
      for(int i=1;i<rates_total;i++)
        {
         //do calculation for the whole chart again here :(
        }
      }

Hope you understand my problem here :)

Br, Candles

[phi nuts]

Maybe try this

   int i, limit;

   if(prev_calculated == 0)     
       limit = prev_calculated;      //--- first calculation
     else 
       limit = prev_calculated - 1;  //--- 

   for(i = limit;i < rates_total && !IsStopped(); i++)
      {

      }

 

[Pasi Hakamaki]

phi.nuts:

Maybe try this

 

Of course! This will naturally work :)

Thanks again Phi :)

[serge103]

This problem still exist today after 7 years.

I have tested a simple filter indicator with the iCustom indicator call function.

This unique line of code used at every tick increaded the process time of my EA (already including more than 1000 line) by 23%.

I replaced the custom indicator by an EMA, and this increase was only 5%.

I then managed to modify the same indicator code and included it in my EA for the exact same result and the increase in processing time was less than a second.

I could not see the difference. Which means less than than 1 sec out of 2:17 min in back testing of severals months. It's more than 32 times faster.

I noticed the same thing with all indicators I tried. In some cases, it increases the processing time by a factor of 10 if we try to call it at every tick.

The only way they can be used is at every 15 min or more. In some cases, this is not accurate enough.

I repeat the same question: Why are the custom indicators so slow running?

Is there a way to be faster in backtesting?

What can we do when the original code is not available?

For an EA we are only interested in the 0 index immediate value, we don't care about past values (totally useless).

How can we avoid computing these useless visual past values??

 

Thanks.

[Harish M]

serge103:

This problem still exist today after 7 years.

I have tested a simple filter indicator with the iCustom indicator call function.

This unique line of code used at every tick increaded the process time of my EA (already including more than 1000 line) by 23%.

I replaced the custom indicator by an EMA, and this increase was only 5%.

I then managed to modify the same indicator code and included it in my EA for the exact same result and the increase in processing time was less than a second.

I could not see the difference. Which means less than than 1 sec out of 2:17 min in back testing of severals months. It's more than 32 times faster.

I noticed the same thing with all indicators I tried. In some cases, it increases the processing time by a factor of 10 if we try to call it at every tick.

The only way they can be used is at every 15 min or more. In some cases, this is not accurate enough.

I repeat the same question: Why are the custom indicators so slow running?

Is there a way to be faster in backtesting?

What can we do when the original code is not available?

For an EA we are only interested in the 0 index immediate value, we don't care about past values (totally useless).

How can we avoid computing these useless visual past values??

 

Thanks.

Observing the same issue with iCustom for the indicator I wrote. I have history data for 20 years. That is 500,000+ candles for 15 minute chart.

So, the trick is to limit the number of candles to consider for every calculation instead of all 500,000+ past price actions for every tick.

Use another variable to limit counting for number of past candles with X_history variable

To get the smooth result for your custom indicator, if it takes only 500 past candles, set it accordingly.

e.g.,

X_history = 500;

limit = Bars - count_already_calclulated - 1;

if (limit > X_history) {

      limit = X_history;

}

iMA if used for 5 period average, might be setting the X_history = 6, so it is faster.

[PetziK Kay]

Hi,
I am currently facing the same issue and found your conversation.
Can anyone please put a code framework here? 
I have been trying to implement an ICustom Indicator via a handle into the Expert Advisor.
Apparently it is very slow in backtesting.
I looked at the code of the custom indicator (ALMA v2)  and it calculates all bars with every tick.
thats not necessary. I am working with shifted indicator, so no redraw on the current bar, and I am working with higher timeframes..
it would be total sufficient if the indicator and the EA both update the arrays only once per bar.

I tried this via this piece of code:

at the onInit section:

   barsTotal = iBars(_Symbol, PERIOD_CURRENT);


//-- CHECK ON NEW BAR
   int bars = iBars(_Symbol, PERIOD_CURRENT);
   if(barsTotal != bars)

     {
      barsTotal = bars;


        The content of creating the buffers of the indicator

     }//--- done

but, still, the EA is very very slow in backtesting. 

So, this brings me to the final part:
I'd like to limit down the numbers of bars to be calculated (and plotted) by the indicator. The last 300 bars are more than sufficient.

    

  //--- preliminary calculations
      if(prev_calculated == 0)
        {
         limit  = 0;
         mtflimit = rates_total - 1;
         ArrayInitialize(alma,EMPTY_VALUE);
        }
      else
        {
         limit = rates_total - 1;
         mtflimit = rates_total - prev_calculated + PeriodSeconds(tf)/PeriodSeconds(Period());
        }

      copied = CopyBuffer(Price_handle,0,Time[rates_total-1],Time[0],price);

      if(copied < 0)
        {
         Print("not all prices copied. Will try on next tick Error =",GetLastError(),", copied =",copied);
         return(0);
        }

so, what I am seeing here is, that the copybuffer obviously runs with every tick / on calculate... and it also applies to the complete array of loaded bars in the chart... rates_total-1.

Could I replace the (rates_total - 1) by a constant number, defined like: 100 ?
For backtesting I would only need actually the last 10 bars, 20 if at most!?

Have you guys experiencing with that? 
I read that you were able to increase the Backtesting speed by 30%... that is still not enough. I'd need a performance increase by at least 10x! (one month of data from April 1st til April 29th) - 15 M Chart.
Backtest said, it would take 5 hours to render!
It should only take 5- 10 Minutes, in my opinion.

Please let me know,
otherwise the only option I see would be to replace the iCustome via handle completely and rebuild the ALMA Arrays manually in a FOR () Loop within the EA itself. I did that in the past, this increases speed significantly, but takes a lot of coding effort.

Any suggestions please!
Thank you