Very slow calculation time for LWMA, I need help optimising MA on Array

winGou- 2023.07.30 09:43

Hello everyone,

I'm trying to test an indicator that is really simple, but it take enormous calculation time during my backtest. Before adding this new feature my EA was running fine, so I checked in "profiling on historical data" and its confirmed, this part of my code use a lot of ressource !

I would like to find a way to optimise it, given the fact that this not seems to be a complex calculation, as I'm only using LWMA (and for example on PineScript it is calculated really fast, so I guess there might be a solution).

Could anyone help me doing it ? I guess it will need to replace the iMAOnArray function, or modify it but not sure how to it :/

Oh and by the way, this function is called only on each new candle (not on new ticks !)

Here is my code :

//shortM = short period
//longM = long period
//price = used price (close, hlc...)
//LWMA = indicator data as output

//iMAOnArray(double& array[], int period, int ma_shift, ENUM_MA_METHOD ma_method, int shift); = code from this topic : https://www.mql5.com/en/articles/81 


bool get_LWMA(int shortM, int longM, double & price[], double & LWMA[])
  {
   int mix0 = shortM;         
   int mix1 = longM;          
   int mix2 = shortM + longM; 
   int mix3 = mix2 + longM;   
   int mix4 = mix3 + mix2;    
   int mix5 = mix4 + mix3;    

   double M1[];
   ArrayResize(M1,mix5 + mix4 + mix3 + mix2 + mix1 + mix0 + 3);
   ArraySetAsSeries(M1,true);

   double M2[];
   ArrayResize(M2,mix5 + mix4 + mix3 + mix2 + mix1 + 3);
   ArraySetAsSeries(M2,true);

   double M3[];
   ArrayResize(M3,mix5 + mix4 + mix3 + mix2 + 3);
   ArraySetAsSeries(M3,true);

   double M4[];
   ArrayResize(M4,mix5 + mix4 + mix3 + 3);
   ArraySetAsSeries(M4,true);

   double M5[];
   ArrayResize(M5,mix5 + mix4 + 3);
   ArraySetAsSeries(M5,true);

   double M6[];
   ArrayResize(M6,mix5 + 3);
   ArraySetAsSeries(M6,true);

   ArrayCopy(M1,price,0,0);
   for(int i = 0; i <= mix5 + mix4 + mix3 + mix2 + mix1 + mix0 + 2; i++)
     {
      M1[i] = iMAOnArray(M1, mix0, 0, MODE_LWMA, i);
     }

   ArrayCopy(M2,M1,0,0);
   for(int i = 0; i <= mix5 + mix4 + mix3 + mix2 + mix1 + 2; i++)
     {
      M2[i] = iMAOnArray(M2, mix1, 0, MODE_LWMA, i);
     }

   ArrayCopy(M3,M2,0,0);
   for(int i = 0; i <= mix5 + mix4 + mix3 + mix2 + 2; i++)
     {
      M3[i] = iMAOnArray(M3, mix2, 0, MODE_LWMA, i);
     }

   ArrayCopy(M4,M3,0,0);
   for(int i = 0; i <= mix5 + mix4 + mix3 + 2; i++)
     {
      M4[i] = iMAOnArray(M4, mix3, 0, MODE_LWMA, i);
     }

   ArrayCopy(M5,M4,0,0);
   for(int i = 0; i <= mix5 + mix4 + 2; i++)
     {
      M5[i] = iMAOnArray(M5, mix4, 0, MODE_LWMA, i);
     }

   ArrayCopy(M6,M5,0,0);
   for(int i = 0; i <= mix5 + 2; i++)
     {
      M6[i] = iMAOnArray(M6, mix5, 0, MODE_LWMA, i);
     }

   ArraySetAsSeries(LWMA,true);
   ArrayCopy(LWMA,M6,0,0);

   return true;
  }

Thanks in advance for anyone willing to help me !

Dominik Egert 2023.07.30 11:03 #1

winGou-:

Hello everyone,

Could anyone help me doing it ? I guess it will need to replace the iMAOnArray function, or modify it but not sure how to it :/

Oh and by the way, this function is called only on each new candle (not on new ticks !)

Here is my code :

Thanks in advance for anyone willing to help me !

Please dont even make me comment on your code.

Here is the correct way to calculate LWMA in an efficient way:

//+------------------------------------------------------------------+
//| Can given value be evaluated to be zero                          |
//+------------------------------------------------------------------+
template <typename T>
const bool MathZero(const T value)
{
    return( (value == NULL)
         || (value == (T)EMPTY_VALUE)
         || !(::MathIsValidNumber((double)value)) );
}



#define ARRAY_INVALID_INDEX ((int)0xFFFFFFFE)

//+------------------------------------------------------------------+
//| Validate given index in range of array, shift if needed          |
//+------------------------------------------------------------------+
const int array_index(const int index, const int size)
{
    // Exit on zero size
    if(size < 1)
    { return(ARRAY_INVALID_INDEX); }
    
    // Predefined cases
    switch(index)
    {
        // Return last element of array
        case -1:    return(size - 1);     // WHOLE_ARRAY

        // Return end of array
        case 0x00:  return((int)NULL);    // NULL
    }

    // Return validated index
    if( (index > NULL)
     && (index < size))
    { return(index); }

    // Move backwards on negative index value
    else if( ((size + index) < size)
          && ((size + index) > NULL))
    { return((size - 1) + index); }

    // Return failed index
    return(ARRAY_INVALID_INDEX);
}


//+------------------------------------------------------------------+
//| Get lowest and higest value in range of array                    |
//+------------------------------------------------------------------+
template<typename T>
const T min_max_values(T &min_val, T &max_val, const T &values[], const int _val_index = -1, const int periods = WHOLE_ARRAY, const bool ignore_zeros = true)
{
    // Local init
            int     min_idx     = NULL;
            int     max_idx     = NULL;
    const   T       sums        = min_max_idx(min_idx, max_idx, values, _val_index, periods, ignore_zeros);

    // Assign values
    min_val = values[min_idx];
    max_val = values[max_idx];

    // Return
    return(sums);
}

template <typename T>
const T min_max_idx(int &min_val_idx, int &max_val_idx, const T &values[], const int _val_index = -1, const int periods = WHOLE_ARRAY, const bool ignore_zeros = true)
{
    // Precheck input

        const int val_index = array_index(_val_index, ArraySize(values));
        if(val_index == ARRAY_INVALID_INDEX)
        { return((T)NULL); }

        if( (periods == NULL)
         || (periods == 1) )
        { 
        { 
            min_val_idx = val_index;
            max_val_idx = val_index;
            return(values[val_index]); 
        }


    // Local init

        const   bool    as_series   = ((ArrayIsDynamic(values)) && !(ArrayIsSeries(values)) && (ArrayGetAsSeries(values)) && (ArraySetAsSeries(values, false)));
        const   int     end         = ((periods > val_index) || (periods < 1)) ? val_index + 0x01 : periods;
                int     cnt         = NULL;
                T       sums        = NULL;
                T       tmp         = NULL;
                        min_val_idx = val_index;
                        max_val_idx = val_index;


    // Loop dataseries
        while( (cnt < end)
            && (!_StopFlag) )
        {
            // Max value
            tmp         = (values[max_val_idx] < values[val_index - cnt]) ? values[val_index - cnt] : (T)NULL;
            max_val_idx = (tmp == NULL) ? max_val_idx : (val_index - cnt);

            // Min value
            tmp         = ((values[min_val_idx] > values[val_index - cnt]) && (((values[val_index - cnt] != NULL) && ignore_zeros) || !ignore_zeros)) ? values[val_index - cnt] : (T)NULL;
            min_val_idx = (tmp == NULL) ? min_val_idx : (val_index - cnt);

            // Summ collector
            sums       += values[val_index - cnt];

            // Loop control
            cnt++;
        }


    // Restore array cfg
    (as_series && (ArraySetAsSeries(values, as_series)));
    min_val_idx = ((as_series) ? val_index - min_val_idx : min_val_idx);
    max_val_idx = ((as_series) ? val_index - max_val_idx : max_val_idx);

    // Return
    return(sums);
}




//+------------------------------------------------------------------+
//| Calculate mean of given value range                              |
//+------------------------------------------------------------------+
template <typename T>
const bool avg(T &result, const T &values[], const int _val_index = -1, const int periods = WHOLE_ARRAY, const T init_val = NULL, const bool ignore_zeros = false, const double remove_minmax = NULL)
{
    result = avg(values, _val_index, periods, init_val, ignore_zeros, remove_minmax);
    return(result != NULL);
}

template <typename T>
const T avg(const T &values[], const int _val_index = -1, const int periods = WHOLE_ARRAY, const T init_val = NULL, const bool ignore_zeros = false, const double remove_minmax = NULL)
{
    // Precheck input

        const int val_index = array_index(_val_index, ArraySize(values));
        if(val_index == ARRAY_INVALID_INDEX)
        { return(NULL); }

        if( (periods == NULL)
         || (periods == 1) )
        { return(values[val_index]); }


    // Local init

        const   bool    as_series   = ((ArrayIsDynamic(values)) && !(ArrayIsSeries(values)) && (ArrayGetAsSeries(values)) && (ArraySetAsSeries(values, false)));
        const   int     end         = (((periods > val_index) || (periods < 1)) ? (val_index + 1) : periods);
                T       build_avg   = NULL;
                T       build_avg_b = NULL;
                T       tmp         = NULL;
                T       raw_val_min = NULL;
                T       raw_val_max = NULL;
                T       val_min     = NULL;
                T       val_max     = NULL;
                int     cnt         = NULL;
                int     total       = NULL;
                int     total_b     = NULL;
                int     loop        = NULL;
                bool    eval        = false;


    // Preinit

        // Min max removal
        if(remove_minmax > NULL)
        {
            // Find min max values
            min_max_values(raw_val_min, raw_val_max, values, val_index, periods, ignore_zeros);

            // Lower bandwidth
            val_min += (T)((remove_minmax < 1.0) ? (val_min - (val_min * remove_minmax)) : NULL);
            val_min += (T)((remove_minmax > (1.0 - DBL_MIN)) ? DBL_MIN : NULL);

            // Upper bandwidth
            val_max -= (T)((remove_minmax < 1.0) ? (val_max - (val_max * remove_minmax)) : NULL);
            val_max -= (T)((remove_minmax > (1.0 - DBL_MIN)) ? DBL_MIN : NULL);
        }


    // Loop dataseries

        // Runtime optimized loop selection
        if(ignore_zeros)
        {
            if(remove_minmax > NULL)
            {
                if(init_val != NULL)
                { 
                    tmp = values[val_index];
                    if(!MathZero(tmp) && (tmp < val_max) && (tmp > val_min))
                    { return(init_val); }
                    build_avg   = (T)(((double)init_val * ((val_index < end) ? (val_index - 1) : end)) + (tmp - values[(val_index < end) ? NULL : (val_index - end)])); 
                    total       = end;
                }
                else
                {
                    while(cnt < end)
                    {
                        // Value collector
                        tmp          = values[val_index - cnt];
                        eval         = (!MathZero(tmp) && (tmp < val_max) && (tmp > val_min));
                        build_avg   += (T)((eval) ? tmp : NULL);
                        total       += eval;
    
                        // Backup value collector
                        eval         = (!MathZero(tmp) && (tmp < raw_val_max) && (tmp > raw_val_min));
                        build_avg_b += (T)((eval) ? tmp : NULL);
                        total_b     += eval;
    
                        // Loop control
                        cnt          = (_StopFlag) ? end        : (cnt + 1);
                        total        = (_StopFlag) ? NULL       : total;
                        total_b      = (_StopFlag) ? NULL       : total_b;
                        loop++;
                    }
                }
            }
            else
            {
                if(init_val != NULL)
                { 
                    if(!MathZero(values[val_index]))
                    { return(init_val); }
                    build_avg   = (T)(((double)init_val * ((val_index < end) ? (val_index - 1) : end)) + (values[val_index] - values[(val_index < end) ? NULL : (val_index - end)])); 
                    total       = end;
                }
                else
                {
                    while(cnt < end)
                    {
                        // Value collector
                        tmp          = values[val_index - cnt];
                        eval         = (!MathZero(tmp));
                        build_avg   += (T)((eval) ? tmp : NULL);
                        total       += eval;
    
                        // Loop control
                        cnt          = (_StopFlag) ? end        : (cnt + 1);
                        total        = (_StopFlag) ? NULL       : total;
                        loop++;
                    }
                }
            }
        }
        else
        {
            if(remove_minmax > NULL)
            {
                if(init_val != NULL)
                { 
                    if((values[val_index - total] < val_max) && (values[val_index - total] > val_min))
                    { return(init_val); }
                    build_avg   = (T)(((double)init_val * ((val_index < end) ? (val_index - 1) : end)) + (values[val_index] - values[(val_index < end) ? NULL : (val_index - end)])); 
                    total       = end;
                }
                else
                {
                    while(cnt < end)
                    {
                        // Value collector
                        eval        = ((values[val_index - total] < val_max) && (values[val_index - total] > val_min));
                        build_avg   += (T)((eval) ? values[val_index - total] : NULL);
                        total       += eval;
    
                        // Backup value collector
                        eval        = ((values[val_index - total] < raw_val_max) && (values[val_index - total] > raw_val_min));
                        build_avg_b += (T)((eval) ? values[val_index - cnt] : NULL);
                        total_b     += eval;
    
                        // Loop control
                        cnt          = (_StopFlag) ? end        : (cnt + 1);
                        total        = (_StopFlag) ? NULL       : total;
                        total_b      = (_StopFlag) ? NULL       : total_b;
                        loop++;
                    }
                }
            }
            else
            {
                if(init_val != NULL)
                { 
                    build_avg   = (T)(((double)init_val * ((val_index < end) ? (val_index - 1) : end)) + (values[val_index] - values[(val_index < end) ? NULL : (val_index - end)])); 
                    total       = end;
                }
                else
                {
                    while(cnt < end)
                    {
                        // Value collector
                        build_avg   += values[val_index - total];
                        total++;
    
                        // Loop control
                        cnt          = (_StopFlag) ? end        : (cnt + 1);
                        total        = (_StopFlag) ? NULL       : total;
                    }
                }
                loop = total;
            }
        }


    // Restore array cfg

        (as_series && (ArraySetAsSeries(values, as_series)));


    // Select result and calculate return value

        if(total > 0)
        { return((T)(build_avg / total)); }
        else if(total_b > 0)
        { return((T)(build_avg_b / total_b)); }
        else if((raw_val_max > NULL) && (raw_val_min > NULL) && (loop == end))
        { return((T)((raw_val_max + raw_val_min) / 2.0)); }


    // Return error
    return((T)NULL);
}



//+------------------------------------------------------------------+
//| Calculate linear weighted average of given value range           |
//+------------------------------------------------------------------+
template <typename T>
const bool lwma(T &result, const T &values[], const int val_index = -1, const int periods = WHOLE_ARRAY, const T init_value = NULL)
{
    result = lwma(values, val_index, periods, init_value);
    return(result != NULL);
}
template <typename T>
const T lwma(const T &values[], const int _val_index = -1, const int periods = WHOLE_ARRAY, const T init_value = NULL)
{
    // Precheck input

        const int val_index = array_index(_val_index, ArraySize(values));
        if(val_index == ARRAY_INVALID_INDEX)
        { return((T)NULL); }

        if( (periods == NULL)
         || ((periods == 1) && (init_value == NULL)) )
        { return((T)values[val_index]); }


    // Local init

        const   bool    as_series   = ((ArrayIsDynamic(values)) && !(ArrayIsSeries(values)) && (ArrayGetAsSeries(values)) && (ArraySetAsSeries(values, false)));
        const   int     end         = ((periods > val_index) || (periods < 1)) ? val_index + 0x01 : periods;
        const   int     begin       = val_index - (end - 1);
                double  lwma_f      = NULL;
                T       init_val    = (init_value != NULL) ? init_value : avg(values, begin, periods);
                int     cnt         = (init_value != NULL) ? end - 0x01 : 0x01;


    // Calculate lwma

        while( (cnt < end)
            && (!_StopFlag) )
        {
            // Next value
            lwma_f      = 2.0 / (cnt + 1);
            init_val    = (T)((values[begin + cnt] - init_val) * lwma_f) + init_val;

            // Loop control
            cnt++;
        }

    // Restore array cfg
    (as_series && (ArraySetAsSeries(values, as_series)));

   // Return
    return(init_val);
}

I hope I adjusted everything, as this is a copy from my library and I had to make a few changes to make it stand-alone code....

Not compiled, outline not tested, but the algorithm is well tested, so the calculations are correct.

Usage is quite simple, you call the functions on the buffer (open, high, low, close) and receive the lwma value according to the period you requested.

EDIT: Fixed a few typos....

[ARCHIVE!] Any rookie question, Buy/Sell Pressure Indicator please did you know

winGou- 2023.07.30 11:12 #2

Dominik Christian Egert #:

Please dont even make me comment on your code.

Here is the correct way to calculate LWMA in an efficient way:

I hope I adjusted everything, as this is a copy from my library and I had to make a few changes to make it stand-alone code....

Not compiled, outline not tested, but the algorithm is well tested, so the calculations are correct.

Usage is quite simple, you call the functions on the buffer (open, high, low, close) and receive the lwma value according to the period you requested.

Wow !! Thank you for your response.

It will probably take me some time to understand your code and be able to implement it, as I'm still a rookie :)

Dominik Egert 2023.07.30 11:32 #3

winGou- #:

Wow !! Thank you for your response.

It will probably take me some time to understand your code and be able to implement it, as I'm still a rookie :)

No problem, if you take the challenge on, you will learn from it, I am sure.

winGou- 2023.07.30 12:24 #4

Dominik Christian Egert #:
No problem, if you take the challenge on, you will learn from it, I am sure.

As I'm still trying to understand your code, could you please confirm me that it could be used to calculate the LWMA of an existing array (which is already an LWMA of an existing Array...).

Just to be sure I'm trying to calculate : LWMA of LWMA of LWMA [...] of LWMA (of used_price).

What is slowing the process is the recalculation of each past candles, and I didn't manage to avoid this for now.

Would there be any method to do it, starting from the code I provided ?

Your code is very clean and well documented (and a little bit too complicated for me atm), and It's really important for me to understand what I'm doing in my own code, so I want to avoid the copy/paste without understanding, so if there is an easier workaround starting from the provided code, I would love to know it :)

MQL5 How can I How to know the Multi Timeframe Indicators

Dominik Egert 2023.07.30 12:43 #5

winGou- #:

As I'm still trying to understand your code, could you please confirm me that it could be used to calculate the LWMA of an existing array (which is already an LWMA of an existing Array...).

Just to be sure I'm trying to calculate : LWMA of LWMA of LWMA [...] of LWMA (of used_price).

What is slowing the process is the recalculation of each past candles, and I didn't manage to avoid this for now.

Would there be any method to do it, starting from the code I provided ?

Start with lwma () from my code.

It takes in an array, you specify the index, and you specify the periods.

You will get the lwma value corresponding to the index you provided, based on the periods you specified.

What you now need to do is, call this function subsequently for each index of your array.

On subsequent calls, you just increment the index by one, and you pass in as "init_val" the previous result. This will speed up calculations enormous.

Once you did that for each element of your source array, storing each result in a destination array, you have a fully populated result array.

For the next "iteration", your lwma of lwma, you repeat the same process as described above, but now you use the result array as your source array.

It's quite straight forward, actually.

EDIT:

Concerning the "copy paste without understanding", this is the main problem, because you need to understand the actual formula to calculate the lwma value. Apparently you do not at the moment, so you are actually already "copy/pasting" anyways...

MT4 iMAOnArray and iBandsOnArray I'm trying to convert Does anyone have MovingAverage

winGou- 2023.07.30 15:10 #6

Dominik Christian Egert #:

Start with lwma () from my code.

It takes in an array, you specify the index, and you specify the periods.

You will get the lwma value corresponding to the index you provided, based on the periods you specified.

What you now need to do is, call this function subsequently for each index of your array.

On subsequent calls, you just increment the index by one, and you pass in as "init_val" the previous result. This will speed up calculations enormous.

Once you did that for each element of your source array, storing each result in a destination array, you have a fully populated result array.

For the next "iteration", your lwma of lwma, you repeat the same process as described above, but now you use the result array as your source array.

It's quite straight forward, actually.

EDIT:

First thing first : Thank you very much for you help and your precious time !

I will give it a try, the lwma code part is the only I'm able to understand for now, I will try to understand the rest of your code after if I manage to get the same results !

I managed to write a workaround with the same idea behind : using my function to initialise the arrays, shift them and use the stored result for calculating only new candles values, but in terms of code its... HORRIBLE !

I have no check for error, so if I get an erronous value during the calculation it will be used multiple times :/

Sorry for my english btw, not my native language !

Help needed with my Help understanding difference between MT4 iMAOnArray and iBandsOnArray

Dominik Egert 2023.07.30 15:39 #7

winGou- #:

First thing first : Thank you very much for you help and your precious time !

I will give it a try, the lwma code part is the only I'm able to understand for now, I will try to understand the rest of your code after if I manage to get the same results !

I have no check for error, so if I get an erronous value during the calculation it will be used multiple times :/

Sorry for my english btw, not my native language !

Yes.

Some sample-code:

{
    int lwma_periods = 25;
    double price[]

    // First lwma
    double first_lwma[];
    calc_lwma(prev_calculated, price, first_lwma);
    


    // Second lwma
    double second_lwma[];
    calc_lwma(prev_calculated, first_lwma, second_lwma);
    

    // Third lwma
    double third_lwma[];
    calc_lwma(prev_calculated, second_lwma, third_lwma);
    
}

void calc_lwma(const int prev_calculated, const double& source[], double& result[])
{
    const int source_size = ArraySize(source);
    ArrayResize(result, source_size);
    for(int idx = prev_calculated; (idx < source_size) && !_StopFlag; idx++)
    { result[idx] = lwma(source, idx, lwma_periods, (idx > NULL) ? result[idx - 1] : NULL); }
}

Not compiled, not tested....

winGou- 2023.07.30 21:09 #8

Dominik Christian Egert #:

Yes.

Some sample-code:

Not compiled, not tested....

Not sure to understand what should be in "prev_calculated" value as i'm working directly on an EA with all my arrays as series.

I know this is used mostly in indicators but how should I define it ?

EDIT : Sorry I realised my first post wasn't clear enough, i'm not writing an indicator, i'm working on an EA and directly putting the calculation inside the EA code. So i don't have any 'prev_calculated' variable :/

Creating an EA with [Solved] Help - Global How to I put

Dominik Egert 2023.07.30 21:21 #9

winGou- #:

Not sure to understand what should be in "prev_calculated" value as i'm working directly on an EA with all my arrays as series.

I know this is used mostly in indicators but how should I define it ?

Whatever you like, it's the index of the last value you calculated the lwma for.

I am not sure how you handle your buffers, so I left it open...

You should know from how you do it.

You load the data from some Copy** function, right? So how do you call that to get your data?

Then, let's say you have 2000 values in your close array. Now you calculate your values starting at 0 or at 0 + lwma_periods.

Next time you load data from your Copy** function, you might have 2001 values. So you start your calculations at index 1999, right?

Array issue... i dont My EA program loops ArraySetAsSeries()

winGou- 2023.07.30 21:22 #10

Dominik Christian Egert #:

Whatever you like, it's the index of the last value you calculated the lwma for.

I am not sure how you handle your buffers, so I left it open...

You should know from how you do it.

You load the data from some Copy** function, right? So how do you call that to get your data?

Then, let's say you have 2000 values in your close array. Now you calculate your values starting at 0 or at 0 + lwma_periods.

Next time you load data from your Copy** function, you might have 2001 values. So you start your calculations at index 1999, right?

Sorry I realised my first post wasn't clear enough, i'm not writing an indicator, i'm working on an EA and directly putting the calculation inside the EA code. So i don't have any 'prev_calculated' variable :/

It means that I'm loosing every data on new candle, except if I create arrays outside of the OnTick and OnInit functions, is that what you suggest to do ?

Improving Speed of Code Init() starting program logic. Help Needed for Order

1 2 3 4

New comment