Help - performance issue with custom function ichimokuOnArray

New comment
 

Hi all,

I'm having a big performance issue with a problem difficult to explain. Let me introduce the problem.

I want to be able to backtest my strategies over Renko charts. But I want these backtests to have the accuracy of M1 charts.

After lots of hours programming, what I have done is this:

My EA is intended to run over M1 charts. My EA loads in a dynamic array ALL the renko history, stored in a pre-created .hst file in the "history" folder. This is a 2-dimensional array, first dimension's the number of candles in .hst file, and second dimension are the OHLC and datetime of candle. Assume .hst file has over 100 thousand candles: first dimension range is over 100 thousand

This is done correctly, and performance is acceptable: it takes around 5 seconds to load whole history file into the array.

The problem I'm having is when developing custom functions to simulate Custom Indicators. For instance, I want to use Ichimoku. So I have developed a function called IchimokuOnArray. This is the beginning of the function:


int IchimokuOnArray(double &a[][5], double &Tenkan[], double &Kijun[], double &Span_A[], double &Span_B[], 
                     double &Chikou[], double &Span_A2[], double &Span_B2[], int tenkan_sen=9, int kijun_sen=26,
                     int senkou_span_b=52) {
   int length = ArrayRange(a,0);
   ArrayResize(Tenkan,length);   ArrayResize(Kijun,length);    ArrayResize(Span_A,length);
   ArrayResize(Span_B,length);   ArrayResize(Chikou,length);    ArrayResize(Span_A2,length);
   ArrayResize(Span_B2,length);

(....)

My problem now is that ichimoku calculations are very time-cpu-consuming, and it takes a very long time to calculate this indicator: for each candle, this indicator performs several loops, and what is worse, one loop is inside the other, with opposite indexes!! (one checks the array from left to right, and the other the opposite way).

This thread is about brain-storming issue, on how could I improve this development.

If someone would want to help or collaborate, I can paste my code, just wanted to know if someone else has used this approach to deal with this kind of problem: to backtest renko history files through M1 charts. Maybe someone has done it and has found a better way of doing this! In that case, would like to know more about it.

 
store the indicator (data) to a file and read the file instead of calculate every candle
 
qjol:
store the indicator (data) to a file and read the file instead of calculate every candle

Now that you mention it, yes, calculating every candle does have its impact.
 
deysmacro:

Now that you mention it, yes, calculating every candle does have its impact.

Thanks for your answers,


yes, that would be an option, but I would have to store the results of the indicator in a lot of files, depending on the input variables of the indicator, so I would have lots of files to check, and that would not avoid the first calculation of the indicator, in the moment where you store all values into the file.

The thing is: how does MT4 calculate values for indicators? I mean, in this function, I want to achieve exactly the same as with Custom Indicator Ichimoku... but how could I do it the same way as MT4 does it internally, so that performance was the same? If I did this function within a C++ DLL, would that improve performance?

 
yokinfx:


yes, that would be an option, but I would have to store the results of the indicator in a lot of files,

and in your way you are storing the results in a lot of arrays

depending on the input variables of the indicator, so I would have lots of files to check,

in your way you have a lot of arrays to check

and that would not avoid the first calculation of the indicator, in the moment where you store all values into the file.

at least not every candle

The thing is: how does MT4 calculate values for indicators? I mean, in this function, I want to achieve exactly the same as with Custom Indicator Ichimoku... but how could I do it the same way as MT4 does it internally, so that performance was the same?

you insist on doing it your way

so, this is the consequences you have to deal with

the right way to do it is not arrays not files but using plain and simple iIchimoku

If I did this function within a C++ DLL, would that improve performance?

no idea you can try


 

Thanks for your answer, but you are not giving real options:


at least not every candle

There are ways to avoid re-checking the whole array every new candle: You only have to load whole indicator array in OnInit, then every new bar (for example) you re-check not the whole array, just new data.


you insist on doing it your way

so, this is the consequences you have to deal with

the right way to do it is not arrays not files but using plain and simple iIchimoku

Hahaha I am not insisting on anything, I am just asking for oppinions...

Yes, I know the best way is using iIchimoku. But I just want to understand how MT4 calculates these indicator's values so fast. Maybe if I understood it, I could do the same operations in my own way.

You can't use iIchimoku for values stored in your own Array, which is what I was looking for.

Something I usually say is

Nothing is impossible, just easier or harder

Until now, I have developed iIchimokuOnArray - which some people would consider difficult- but now I'd love to get this function work fast, which is the difficult part. I still don't know if I have knowledge enough to do it...


I know of other developers who work with their EAs over Renko Charts in this way, storing anything in their own Arrays. So, it can be done.

 
yokinfx:

Thanks for your answer, but you are not giving real options:


There are ways to avoid re-checking the whole array every new candle: You only have to load whole indicator array in OnInit, then every new bar (for example) you re-check not the whole array, just new data.


Hahaha I am not insisting on anything, I am just asking for oppinions...

Yes, I know the best way is using iIchimoku. But I just want to understand how MT4 calculates these indicator's values so fast. Maybe if I understood it, I could do the same operations in my own way.

You can't use iIchimoku for values stored in your own Array, which is what I was looking for.

Something I usually say is

Nothing is impossible, just easier or harder

Until now, I have developed iIchimokuOnArray - which some people would consider difficult- but now I'd love to get this function work fast, which is the difficult part. I still don't know if I have knowledge enough to do it...


I know of other developers who work with their EAs over Renko Charts in this way, storing anything in their own Arrays. So, it can be done.

Did you check the Ichimoku code ?

By the way, I don't understand why you need to do that, can't you apply the Ichimoku indicator to the renko chart directly ? It's a real question (not a mockery) as I am not accustomed to work with Renko charts.

 
angevoyageur:

Did you check the Ichimoku code ?

By the way, I don't understand why you need to do that, can't you apply the Ichimoku indicator to the renko chart directly ? It's a real question (not a mockery) as I am not accustomed to work with Renko charts.



Yes, I wrote my Ichimoku function according to the Ichimoku code provided by MQ.


Answering your question, I would like to do that because Renko Charts are just another way of artificial chart, or synthetic candle generation. So I would like to have my synthetic candle charts in my own files, and be able to backtest and optimize these EAs not entering in "disconnected" mode.

Now I am doing it like this: I create my .hst file, I enter in "disconnected mode" and import my .hst file into an existing TF, and I do my backtest in this TF. And this way works.

But I would like to optimize my EAs changing also my Renko box size through an input variable. Something like having EURUSD61.hst, EURUSD62.hst, EURUSD63.hst in my history folder, and be able to backtest my EA with these history files, but precission of M1. So, I can not backtest EAs in non-existing TF, BUT I was just trying to create a whole load of functions to do it: loading the synthetic candles chart into arrays, then calling my Personal <IndicatorOnArray> functions, and so on.

I hope now you understand what I am trying to do...

 
yokinfx:

Yes, I wrote my Ichimoku function according to the Ichimoku code provided by MQ.


Answering your question, I would like to do that because Renko Charts are just another way of artificial chart, or synthetic candle generation. So I would like to have my synthetic candle charts in my own files, and be able to backtest and optimize these EAs not entering in "disconnected" mode.

Now I am doing it like this: I create my .hst file, I enter in "disconnected mode" and import my .hst file into an existing TF, and I do my backtest in this TF. And this way works.

But I would like to optimize my EAs changing also my Renko box size through an input variable. Something like having EURUSD61.hst, EURUSD62.hst, EURUSD63.hst in my history folder, and be able to backtest my EA with these history files, but precission of M1. So, I can not backtest EAs in non-existing TF, BUT I was just trying to create a whole load of functions to do it: loading the synthetic candles chart into arrays, then calling my Personal <IndicatorOnArray> functions, and so on.

I hope now you understand what I am trying to do...

Yes thank you it's clear now.

You probably have to optimize the algorithm of your iIchimokuOnArray. I suggest you to check this code, it can give you idea (mql5 inside). Just an idea, don't have time to dive into the code.

New comment
Reason: