# Very slow calculation time for LWMA, I need help optimising MA on Array

Hello everyone,

I'm trying to test an indicator that is really simple, but it take enormous calculation time during my backtest. Before adding this new feature my EA was running fine, so I checked in "profiling on historical data" and its confirmed, this part of my code use a lot of ressource !

I would like to find a way to optimise it, given the fact that this not seems to be a complex calculation, as I'm only using LWMA (and for example on PineScript it is calculated really fast, so I guess there might be a solution).

Could anyone help me doing it ? I guess it will need to replace the iMAOnArray function, or modify it but not sure how to it :/

Oh and by the way, this function is called only on each new candle (not on new ticks !)

Here is my code :

```//shortM = short period
//longM = long period
//price = used price (close, hlc...)
//LWMA = indicator data as output

//iMAOnArray(double& array[], int period, int ma_shift, ENUM_MA_METHOD ma_method, int shift); = code from this topic : https://www.mql5.com/en/articles/81

bool get_LWMA(int shortM, int longM, double & price[], double & LWMA[])
{
int mix0 = shortM;
int mix1 = longM;
int mix2 = shortM + longM;
int mix3 = mix2 + longM;
int mix4 = mix3 + mix2;
int mix5 = mix4 + mix3;

double M1[];
ArrayResize(M1,mix5 + mix4 + mix3 + mix2 + mix1 + mix0 + 3);
ArraySetAsSeries(M1,true);

double M2[];
ArrayResize(M2,mix5 + mix4 + mix3 + mix2 + mix1 + 3);
ArraySetAsSeries(M2,true);

double M3[];
ArrayResize(M3,mix5 + mix4 + mix3 + mix2 + 3);
ArraySetAsSeries(M3,true);

double M4[];
ArrayResize(M4,mix5 + mix4 + mix3 + 3);
ArraySetAsSeries(M4,true);

double M5[];
ArrayResize(M5,mix5 + mix4 + 3);
ArraySetAsSeries(M5,true);

double M6[];
ArrayResize(M6,mix5 + 3);
ArraySetAsSeries(M6,true);

ArrayCopy(M1,price,0,0);
for(int i = 0; i <= mix5 + mix4 + mix3 + mix2 + mix1 + mix0 + 2; i++)
{
M1[i] = iMAOnArray(M1, mix0, 0, MODE_LWMA, i);
}

ArrayCopy(M2,M1,0,0);
for(int i = 0; i <= mix5 + mix4 + mix3 + mix2 + mix1 + 2; i++)
{
M2[i] = iMAOnArray(M2, mix1, 0, MODE_LWMA, i);
}

ArrayCopy(M3,M2,0,0);
for(int i = 0; i <= mix5 + mix4 + mix3 + mix2 + 2; i++)
{
M3[i] = iMAOnArray(M3, mix2, 0, MODE_LWMA, i);
}

ArrayCopy(M4,M3,0,0);
for(int i = 0; i <= mix5 + mix4 + mix3 + 2; i++)
{
M4[i] = iMAOnArray(M4, mix3, 0, MODE_LWMA, i);
}

ArrayCopy(M5,M4,0,0);
for(int i = 0; i <= mix5 + mix4 + 2; i++)
{
M5[i] = iMAOnArray(M5, mix4, 0, MODE_LWMA, i);
}

ArrayCopy(M6,M5,0,0);
for(int i = 0; i <= mix5 + 2; i++)
{
M6[i] = iMAOnArray(M6, mix5, 0, MODE_LWMA, i);
}

ArraySetAsSeries(LWMA,true);
ArrayCopy(LWMA,M6,0,0);

return true;
}
```

Thanks in advance for anyone willing to help me !

winGou-:

Hello everyone,

I'm trying to test an indicator that is really simple, but it take enormous calculation time during my backtest. Before adding this new feature my EA was running fine, so I checked in "profiling on historical data" and its confirmed, this part of my code use a lot of ressource !

I would like to find a way to optimise it, given the fact that this not seems to be a complex calculation, as I'm only using LWMA (and for example on PineScript it is calculated really fast, so I guess there might be a solution).

Could anyone help me doing it ? I guess it will need to replace the iMAOnArray function, or modify it but not sure how to it :/

Oh and by the way, this function is called only on each new candle (not on new ticks !)

Here is my code :

Thanks in advance for anyone willing to help me !

Here is the correct way to calculate LWMA in an efficient way:

```//+------------------------------------------------------------------+
//| Can given value be evaluated to be zero                          |
//+------------------------------------------------------------------+
template <typename T>
const bool MathZero(const T value)
{
return( (value == NULL)
|| (value == (T)EMPTY_VALUE)
|| !(::MathIsValidNumber((double)value)) );
}

#define ARRAY_INVALID_INDEX ((int)0xFFFFFFFE)

//+------------------------------------------------------------------+
//| Validate given index in range of array, shift if needed          |
//+------------------------------------------------------------------+
const int array_index(const int index, const int size)
{
// Exit on zero size
if(size < 1)
{ return(ARRAY_INVALID_INDEX); }

// Predefined cases
switch(index)
{
// Return last element of array
case -1:    return(size - 1);     // WHOLE_ARRAY

// Return end of array
case 0x00:  return((int)NULL);    // NULL
}

// Return validated index
if( (index > NULL)
&& (index < size))
{ return(index); }

// Move backwards on negative index value
else if( ((size + index) < size)
&& ((size + index) > NULL))
{ return((size - 1) + index); }

// Return failed index
return(ARRAY_INVALID_INDEX);
}

//+------------------------------------------------------------------+
//| Get lowest and higest value in range of array                    |
//+------------------------------------------------------------------+
template<typename T>
const T min_max_values(T &min_val, T &max_val, const T &values[], const int _val_index = -1, const int periods = WHOLE_ARRAY, const bool ignore_zeros = true)
{
// Local init
int     min_idx     = NULL;
int     max_idx     = NULL;
const   T       sums        = min_max_idx(min_idx, max_idx, values, _val_index, periods, ignore_zeros);

// Assign values
min_val = values[min_idx];
max_val = values[max_idx];

// Return
return(sums);
}

template <typename T>
const T min_max_idx(int &min_val_idx, int &max_val_idx, const T &values[], const int _val_index = -1, const int periods = WHOLE_ARRAY, const bool ignore_zeros = true)
{
// Precheck input

const int val_index = array_index(_val_index, ArraySize(values));
if(val_index == ARRAY_INVALID_INDEX)
{ return((T)NULL); }

if( (periods == NULL)
|| (periods == 1) )
{
{
min_val_idx = val_index;
max_val_idx = val_index;
return(values[val_index]);
}

// Local init

const   bool    as_series   = ((ArrayIsDynamic(values)) && !(ArrayIsSeries(values)) && (ArrayGetAsSeries(values)) && (ArraySetAsSeries(values, false)));
const   int     end         = ((periods > val_index) || (periods < 1)) ? val_index + 0x01 : periods;
int     cnt         = NULL;
T       sums        = NULL;
T       tmp         = NULL;
min_val_idx = val_index;
max_val_idx = val_index;

// Loop dataseries
while( (cnt < end)
&& (!_StopFlag) )
{
// Max value
tmp         = (values[max_val_idx] < values[val_index - cnt]) ? values[val_index - cnt] : (T)NULL;
max_val_idx = (tmp == NULL) ? max_val_idx : (val_index - cnt);

// Min value
tmp         = ((values[min_val_idx] > values[val_index - cnt]) && (((values[val_index - cnt] != NULL) && ignore_zeros) || !ignore_zeros)) ? values[val_index - cnt] : (T)NULL;
min_val_idx = (tmp == NULL) ? min_val_idx : (val_index - cnt);

// Summ collector
sums       += values[val_index - cnt];

// Loop control
cnt++;
}

// Restore array cfg
(as_series && (ArraySetAsSeries(values, as_series)));
min_val_idx = ((as_series) ? val_index - min_val_idx : min_val_idx);
max_val_idx = ((as_series) ? val_index - max_val_idx : max_val_idx);

// Return
return(sums);
}

//+------------------------------------------------------------------+
//| Calculate mean of given value range                              |
//+------------------------------------------------------------------+
template <typename T>
const bool avg(T &result, const T &values[], const int _val_index = -1, const int periods = WHOLE_ARRAY, const T init_val = NULL, const bool ignore_zeros = false, const double remove_minmax = NULL)
{
result = avg(values, _val_index, periods, init_val, ignore_zeros, remove_minmax);
return(result != NULL);
}

template <typename T>
const T avg(const T &values[], const int _val_index = -1, const int periods = WHOLE_ARRAY, const T init_val = NULL, const bool ignore_zeros = false, const double remove_minmax = NULL)
{
// Precheck input

const int val_index = array_index(_val_index, ArraySize(values));
if(val_index == ARRAY_INVALID_INDEX)
{ return(NULL); }

if( (periods == NULL)
|| (periods == 1) )
{ return(values[val_index]); }

// Local init

const   bool    as_series   = ((ArrayIsDynamic(values)) && !(ArrayIsSeries(values)) && (ArrayGetAsSeries(values)) && (ArraySetAsSeries(values, false)));
const   int     end         = (((periods > val_index) || (periods < 1)) ? (val_index + 1) : periods);
T       build_avg   = NULL;
T       build_avg_b = NULL;
T       tmp         = NULL;
T       raw_val_min = NULL;
T       raw_val_max = NULL;
T       val_min     = NULL;
T       val_max     = NULL;
int     cnt         = NULL;
int     total       = NULL;
int     total_b     = NULL;
int     loop        = NULL;
bool    eval        = false;

// Preinit

// Min max removal
if(remove_minmax > NULL)
{
// Find min max values
min_max_values(raw_val_min, raw_val_max, values, val_index, periods, ignore_zeros);

// Lower bandwidth
val_min += (T)((remove_minmax < 1.0) ? (val_min - (val_min * remove_minmax)) : NULL);
val_min += (T)((remove_minmax > (1.0 - DBL_MIN)) ? DBL_MIN : NULL);

// Upper bandwidth
val_max -= (T)((remove_minmax < 1.0) ? (val_max - (val_max * remove_minmax)) : NULL);
val_max -= (T)((remove_minmax > (1.0 - DBL_MIN)) ? DBL_MIN : NULL);
}

// Loop dataseries

// Runtime optimized loop selection
if(ignore_zeros)
{
if(remove_minmax > NULL)
{
if(init_val != NULL)
{
tmp = values[val_index];
if(!MathZero(tmp) && (tmp < val_max) && (tmp > val_min))
{ return(init_val); }
build_avg   = (T)(((double)init_val * ((val_index < end) ? (val_index - 1) : end)) + (tmp - values[(val_index < end) ? NULL : (val_index - end)]));
total       = end;
}
else
{
while(cnt < end)
{
// Value collector
tmp          = values[val_index - cnt];
eval         = (!MathZero(tmp) && (tmp < val_max) && (tmp > val_min));
build_avg   += (T)((eval) ? tmp : NULL);
total       += eval;

// Backup value collector
eval         = (!MathZero(tmp) && (tmp < raw_val_max) && (tmp > raw_val_min));
build_avg_b += (T)((eval) ? tmp : NULL);
total_b     += eval;

// Loop control
cnt          = (_StopFlag) ? end        : (cnt + 1);
total        = (_StopFlag) ? NULL       : total;
total_b      = (_StopFlag) ? NULL       : total_b;
loop++;
}
}
}
else
{
if(init_val != NULL)
{
if(!MathZero(values[val_index]))
{ return(init_val); }
build_avg   = (T)(((double)init_val * ((val_index < end) ? (val_index - 1) : end)) + (values[val_index] - values[(val_index < end) ? NULL : (val_index - end)]));
total       = end;
}
else
{
while(cnt < end)
{
// Value collector
tmp          = values[val_index - cnt];
eval         = (!MathZero(tmp));
build_avg   += (T)((eval) ? tmp : NULL);
total       += eval;

// Loop control
cnt          = (_StopFlag) ? end        : (cnt + 1);
total        = (_StopFlag) ? NULL       : total;
loop++;
}
}
}
}
else
{
if(remove_minmax > NULL)
{
if(init_val != NULL)
{
if((values[val_index - total] < val_max) && (values[val_index - total] > val_min))
{ return(init_val); }
build_avg   = (T)(((double)init_val * ((val_index < end) ? (val_index - 1) : end)) + (values[val_index] - values[(val_index < end) ? NULL : (val_index - end)]));
total       = end;
}
else
{
while(cnt < end)
{
// Value collector
eval        = ((values[val_index - total] < val_max) && (values[val_index - total] > val_min));
build_avg   += (T)((eval) ? values[val_index - total] : NULL);
total       += eval;

// Backup value collector
eval        = ((values[val_index - total] < raw_val_max) && (values[val_index - total] > raw_val_min));
build_avg_b += (T)((eval) ? values[val_index - cnt] : NULL);
total_b     += eval;

// Loop control
cnt          = (_StopFlag) ? end        : (cnt + 1);
total        = (_StopFlag) ? NULL       : total;
total_b      = (_StopFlag) ? NULL       : total_b;
loop++;
}
}
}
else
{
if(init_val != NULL)
{
build_avg   = (T)(((double)init_val * ((val_index < end) ? (val_index - 1) : end)) + (values[val_index] - values[(val_index < end) ? NULL : (val_index - end)]));
total       = end;
}
else
{
while(cnt < end)
{
// Value collector
build_avg   += values[val_index - total];
total++;

// Loop control
cnt          = (_StopFlag) ? end        : (cnt + 1);
total        = (_StopFlag) ? NULL       : total;
}
}
loop = total;
}
}

// Restore array cfg

(as_series && (ArraySetAsSeries(values, as_series)));

// Select result and calculate return value

if(total > 0)
{ return((T)(build_avg / total)); }
else if(total_b > 0)
{ return((T)(build_avg_b / total_b)); }
else if((raw_val_max > NULL) && (raw_val_min > NULL) && (loop == end))
{ return((T)((raw_val_max + raw_val_min) / 2.0)); }

// Return error
return((T)NULL);
}

//+------------------------------------------------------------------+
//| Calculate linear weighted average of given value range           |
//+------------------------------------------------------------------+
template <typename T>
const bool lwma(T &result, const T &values[], const int val_index = -1, const int periods = WHOLE_ARRAY, const T init_value = NULL)
{
result = lwma(values, val_index, periods, init_value);
return(result != NULL);
}
template <typename T>
const T lwma(const T &values[], const int _val_index = -1, const int periods = WHOLE_ARRAY, const T init_value = NULL)
{
// Precheck input

const int val_index = array_index(_val_index, ArraySize(values));
if(val_index == ARRAY_INVALID_INDEX)
{ return((T)NULL); }

if( (periods == NULL)
|| ((periods == 1) && (init_value == NULL)) )
{ return((T)values[val_index]); }

// Local init

const   bool    as_series   = ((ArrayIsDynamic(values)) && !(ArrayIsSeries(values)) && (ArrayGetAsSeries(values)) && (ArraySetAsSeries(values, false)));
const   int     end         = ((periods > val_index) || (periods < 1)) ? val_index + 0x01 : periods;
const   int     begin       = val_index - (end - 1);
double  lwma_f      = NULL;
T       init_val    = (init_value != NULL) ? init_value : avg(values, begin, periods);
int     cnt         = (init_value != NULL) ? end - 0x01 : 0x01;

// Calculate lwma

while( (cnt < end)
&& (!_StopFlag) )
{
// Next value
lwma_f      = 2.0 / (cnt + 1);
init_val    = (T)((values[begin + cnt] - init_val) * lwma_f) + init_val;

// Loop control
cnt++;
}

// Restore array cfg
(as_series && (ArraySetAsSeries(values, as_series)));

// Return
return(init_val);
}
```

I hope I adjusted everything, as this is a copy from my library and I had to make a few changes to make it stand-alone code....

Not compiled, outline not tested, but the algorithm is well tested, so the calculations are correct.

Usage is quite simple, you call the functions on the buffer (open, high, low, close) and receive the lwma value according to the period you requested.

EDIT: Fixed a few typos....

Dominik Christian Egert #:

Here is the correct way to calculate LWMA in an efficient way:

I hope I adjusted everything, as this is a copy from my library and I had to make a few changes to make it stand-alone code....

Not compiled, outline not tested, but the algorithm is well tested, so the calculations are correct.

Usage is quite simple, you call the functions on the buffer (open, high, low, close) and receive the lwma value according to the period you requested.

Wow !! Thank you for your response.

It will probably take me some time to understand your code and be able to implement it, as I'm still a rookie :)

winGou- #:

Wow !! Thank you for your response.

It will probably take me some time to understand your code and be able to implement it, as I'm still a rookie :)

No problem, if you take the challenge on, you will learn from it, I am sure.

Dominik Christian Egert #:
No problem, if you take the challenge on, you will learn from it, I am sure.

As I'm still trying to understand your code, could you please confirm me that it could be used to calculate the LWMA of an existing array (which is already an LWMA of an existing Array...).

Just to be sure I'm trying to calculate : LWMA of LWMA of LWMA [...] of LWMA (of used_price).

What is slowing the process is the recalculation of each past candles, and I didn't manage to avoid this for now.

Would there be any method to do it, starting from the code I provided ?

Your code is very clean and well documented (and a little bit too complicated for me atm), and It's really important for me to understand what I'm doing in my own code, so I want to avoid the copy/paste without understanding, so if there is an easier workaround starting from the provided code, I would love to know it :)

winGou- #:

As I'm still trying to understand your code, could you please confirm me that it could be used to calculate the LWMA of an existing array (which is already an LWMA of an existing Array...).

Just to be sure I'm trying to calculate : LWMA of LWMA of LWMA [...] of LWMA (of used_price).

What is slowing the process is the recalculation of each past candles, and I didn't manage to avoid this for now.

Would there be any method to do it, starting from the code I provided ?

Your code is very clean and well documented (and a little bit too complicated for me atm), and It's really important for me to understand what I'm doing in my own code, so I want to avoid the copy/paste without understanding, so if there is an easier workaround starting from the provided code, I would love to know it :)

It takes in an array, you specify the index, and you specify the periods.

You will get the lwma value corresponding to the index you provided, based on the periods you specified.

What you now need to do is, call this function subsequently for each index of your array.

On subsequent calls, you just increment the index by one, and you pass in as "init_val" the previous result. This will speed up calculations enormous.

Once you did that for each element of your source array, storing each result in a destination array, you have a fully populated result array.

For the next "iteration", your lwma of lwma, you repeat the same process as described above, but now you use the result array as your source array.

It's quite straight forward, actually.

EDIT:

Concerning the "copy paste without understanding", this is the main problem, because you need to understand the actual formula to calculate the lwma value. Apparently you do not at the moment, so you are actually already "copy/pasting" anyways...

Dominik Christian Egert #:

It takes in an array, you specify the index, and you specify the periods.

You will get the lwma value corresponding to the index you provided, based on the periods you specified.

What you now need to do is, call this function subsequently for each index of your array.

On subsequent calls, you just increment the index by one, and you pass in as "init_val" the previous result. This will speed up calculations enormous.

Once you did that for each element of your source array, storing each result in a destination array, you have a fully populated result array.

For the next "iteration", your lwma of lwma, you repeat the same process as described above, but now you use the result array as your source array.

It's quite straight forward, actually.

EDIT:

Concerning the "copy paste without understanding", this is the main problem, because you need to understand the actual formula to calculate the lwma value. Apparently you do not at the moment, so you are actually already "copy/pasting" anyways...

First thing first : Thank you very much for you help and your precious time !

I will give it a try, the lwma code part is the only I'm able to understand for now, I will try to understand the rest of your code after if I manage to get the same results !

I managed to write a workaround with the same idea behind : using my function to initialise the arrays, shift them and use the stored result for calculating only new candles values, but in terms of code its... HORRIBLE !

I have no check for error, so if I get an erronous value during the calculation it will be used multiple times :/

Sorry for my english btw, not my native language !

winGou- #:

First thing first : Thank you very much for you help and your precious time !

I will give it a try, the lwma code part is the only I'm able to understand for now, I will try to understand the rest of your code after if I manage to get the same results !

I managed to write a workaround with the same idea behind : using my function to initialise the arrays, shift them and use the stored result for calculating only new candles values, but in terms of code its... HORRIBLE !

I have no check for error, so if I get an erronous value during the calculation it will be used multiple times :/

Sorry for my english btw, not my native language !

Yes.

Some sample-code:

```{
int lwma_periods = 25;
double price[]

// First lwma
double first_lwma[];
calc_lwma(prev_calculated, price, first_lwma);

// Second lwma
double second_lwma[];
calc_lwma(prev_calculated, first_lwma, second_lwma);

// Third lwma
double third_lwma[];
calc_lwma(prev_calculated, second_lwma, third_lwma);

}

void calc_lwma(const int prev_calculated, const double& source[], double& result[])
{
const int source_size = ArraySize(source);
ArrayResize(result, source_size);
for(int idx = prev_calculated; (idx < source_size) && !_StopFlag; idx++)
{ result[idx] = lwma(source, idx, lwma_periods, (idx > NULL) ? result[idx - 1] : NULL); }
}

```

Not compiled, not tested....

Dominik Christian Egert #:

Yes.

Some sample-code:

Not compiled, not tested....

Not sure to understand what should be in "prev_calculated" value as i'm working directly on an EA with all my arrays as series.

I know this is used mostly in indicators but how should I define it ?

EDIT : Sorry I realised my first post wasn't clear enough, i'm not writing an indicator, i'm working on an EA and directly putting the calculation inside the EA code. So i don't have any 'prev_calculated' variable :/

winGou- #:

Not sure to understand what should be in "prev_calculated" value as i'm working directly on an EA with all my arrays as series.

I know this is used mostly in indicators but how should I define it ?

Whatever you like, it's the index of the last value you calculated the lwma for.

I am not sure how you handle your buffers, so I left it open...
You should know from how you do it.

You load the data from some Copy** function, right? So how do you call that to get your data?

Then, let's say you have 2000 values in your close array. Now you calculate your values starting at 0 or at 0 + lwma_periods.

Next time you load data from your Copy** function, you might have 2001 values. So you start your calculations at index 1999, right?

Dominik Christian Egert #:
Whatever you like, it's the index of the last value you calculated the lwma for.

I am not sure how you handle your buffers, so I left it open...
You should know from how you do it.

You load the data from some Copy** function, right? So how do you call that to get your data?

Then, let's say you have 2000 values in your close array. Now you calculate your values starting at 0 or at 0 + lwma_periods.

Next time you load data from your Copy** function, you might have 2001 values. So you start your calculations at index 1999, right?

Sorry I realised my first post wasn't clear enough, i'm not writing an indicator, i'm working on an EA and directly putting the calculation inside the EA code. So i don't have any 'prev_calculated' variable :/

It means that I'm loosing every data on new candle, except if I create arrays outside of the OnTick and OnInit functions, is that what you suggest to do ?