Having wired Moving Average problem while creating EA.. - page 3

[Alain Verleyen]

angreeee:
i noticed again even when i run it from 2009 at the current date (04.2014) the difference between the MA on the chart and ima in backtest is still 0.10, so i guess the issue persists. I will do my own iMa replacement function if all other failed. icustom still returns only zeros on D1 chart even when starting at 2009 and working fine on H4 chart.

I confirmed earlier there is a problem with SMMA mode, I think you already report that to Service Desk ? Other modes seem ok to me.

[Grzegorz Korycki]

analyzing how the custom moving average indicator works i understand why there are problems like that.

Every moving average is counted in non-crude fashion counting next frame of moving average from previous frame of moving average instead of just counting necessary (in this case 370) frames for the equation. That way if 1 frame of MA is incorrect all that follow will be also. The further from error-frame the smaller the error will be. It could even work properly if all frames from beginning were counted properly, but i noticed sometimes iMA at start report zeros (i have a procedure to discard faulty ma readings, but the iMA itself does not) and those zeros are probably also taken under consideration when counting next frames of iMA from the previous frames of iMA.
That is why when i started back-tests in 2013 the difference was greatest, in 2012 it was smaller than when starting in 2013, start=2011 even smaller, and so on. Difference was still visible even when i started back-test from 2009, so it is an serious issue.

[Grzegorz Korycki]

angevoyageur:
I confirmed earlier there is a problem with SMMA mode, I think you already report that to Service Desk ? Other modes seem ok to me.

  im finishing writing my own iMA replacement procedure so i can fully document and understand the issue. (not just by visual comparison like done in this thread). 

  The results of ima, my ima replacement and custom moving average results will be available in the log for comparison.

  PS. if you confirm the error I am reporting it now. (i will add new comment in this thread when the issue is reported). The site (or my internet) works very slowly at the moment so im having issues even getting to service desk page.

[Grzegorz Korycki]

ok it is reported.

[Grzegorz Korycki]

i thought other MA types are affected too but i made more tests and SSMA seems the only one affected, like you have said.

but i noticed iCustom issue. Script to test SSMA and iCustom issues:

#property version     "1.1"
#property description "MA TEST"

#include <Trade/Trade.mqh>

   #define MAGICMA  20131002

double Bid;
double Ask;

   int ma_temp;
   int custom_ma_temp;
   double ma_buffer[20];
   double ima2_buffer[510];
   double ima2[510];

input int ma_period = 370;
input ENUM_TIMEFRAMES ma_tf = PERIOD_D1;
input ENUM_MA_METHOD ma_method = MODE_SMMA;
input ENUM_APPLIED_PRICE ma_price = PRICE_OPEN;
   
   
   
double OnTester()
{
    double custommax = TesterStatistics(STAT_EQUITY_DDREL_PERCENT);
    return (custommax);
}
   CTrade  trade;
   
  void OnTick()
  {
   MqlTick last_tick;
   if(SymbolInfoTick(Symbol(),last_tick))
     {
      Bid = last_tick.bid;
      Ask = last_tick.ask;
     }
   start();
  }
  
int OnInit()
  {
   trade.SetExpertMagicNumber(MAGICMA);
   int deviation=99;
   trade.SetDeviationInPoints(deviation);
   trade.SetAsyncMode(false);

   return(0);
  }  
  
      
      
      
void trend1()
{

   double ma;

   ma_temp=iMA(Symbol(),ma_tf,ma_period,0,ma_method,ma_price);
   CopyBuffer(ma_temp,0,0,1,ma_buffer);
   ma=ma_buffer[0];
   Alert("ma=", ma);

   custom_ma_temp=iCustom(Symbol(),ma_tf,"Examples\\Custom Moving Average", ma_period, 0, ma_method,ma_price);
   CopyBuffer(custom_ma_temp,0,0,1,ma_buffer);
   ma=ma_buffer[0];  
   Alert("custom ma=", ma);

}

      


void start()
{
         trend1();
}

[Alain Verleyen]

angreeee:

analyzing how the custom moving average indicator works i understand why there are problems like that.

Every moving average is counted in non-crude fashion counting next frame of moving average from previous frame of moving average instead of just counting necessary (in this case 370) frames for the equation. That way if 1 frame of MA is incorrect all that follow will be also. The further from error-frame the smaller the error will be. It could even work properly if all frames from beginning were counted properly, but i noticed sometimes iMA at start report zeros (i have a procedure to discard faulty ma readings, but the iMA itself does not) and those zeros are probably also taken under consideration when counting next frames of iMA from the previous frames of iMA.
That is why when i started back-tests in 2013 the difference was greatest, in 2012 it was smaller than when starting in 2013, start=2011 even smaller, and so on. Difference was still visible even when i started back-test from 2009, so it is an serious issue.

I don't see the problem. iMA is well coded for performance. If an iMA report a 0 it's because you don't have data (or enough data). By the way there is only a problem with SMMA, I don't know why, but it cannot be due to this "incremental" implementation as it's working well for other mode.

[Grzegorz Korycki]

angevoyageur:
I don't see the problem. iMA is well coded for performance. If an iMA report a 0 it's because you don't have data (or enough data). By the way there is only a problem with SMMA, I don't know why, but it cannot be due to this "incremental" implementation as it's working well for other mode.

Yeah you are right it would be much slower. But the fact is, that this way of counting plus some empty frames (zeros) at the beginning would result problems like that. (that is why iMA SSMA is always smaller, not bigger than the accual SSMA) I know i don't check what iMA returns thats why i get zeros at start instead of handling lack of necessary information properly, but that is a different issue.

With smaller TFs there are much more frames analyzed and the problem may be diluted with time. With every new frame the ima SSMA gets closer and closer to the real SSMA, so the more bars passed, the less visible is the problem, thats why i think no-one noticed it before. The same issue i found with 370 SSMA and PERIOD_12H PERIOD_8H, etc.

If you have time please look into the iCustom function. I attached source code to easily test it. Check any lower TFs that PERIOD_D1 - iCustom works fine and return same values as iMA. In PERIOD_D1 it is always zeros for iCustom, while iMA still reports some values.

The curious thing is when you use SSMA on max PERIOD_H12 both iMA and iCustom "custom moving average" indicator report faulty values. (test from 2014.01 to see the difference) 

The error has to be somewhere here: (form custom moving average)

void CalculateSmoothedMA(int rates_total,int prev_calculated,int begin,const double &price[])
  {
   int i,limit;
//--- first calculation or number of bars was changed
   if(prev_calculated==0)
     {
      limit=InpMAPeriod+begin;
      //--- set empty value for first limit bars
      for(i=0;i<limit-1;i++) ExtLineBuffer[i]=0.0;
      //--- calculate first visible value
      double firstValue=0;
      for(i=begin;i<limit;i++)
         firstValue+=price[i];
      firstValue/=InpMAPeriod;
      ExtLineBuffer[limit-1]=firstValue;
     }
   else limit=prev_calculated-1;
//--- main loop
   for(i=limit;i<rates_total && !IsStopped();i++)
      ExtLineBuffer[i]=(ExtLineBuffer[i-1]*(InpMAPeriod-1)+price[i])/InpMAPeriod;
//---
  }

Notice the firstValue is counted the same as in SMA procedure:

void CalculateSimpleMA(int rates_total,int prev_calculated,int begin,const double &price[])
  {
   int i,limit;
//--- first calculation or number of bars was changed
   if(prev_calculated==0)// first calculation
     {
      limit=InpMAPeriod+begin;
      //--- set empty value for first limit bars
      for(i=0;i<limit-1;i++) ExtLineBuffer[i]=0.0;
      //--- calculate first visible value
      double firstValue=0;
      for(i=begin;i<limit;i++)
         firstValue+=price[i];
      firstValue/=InpMAPeriod;
      ExtLineBuffer[limit-1]=firstValue;
     }
   else limit=prev_calculated-1;
//--- main loop
   for(i=limit;i<rates_total && !IsStopped();i++)
      ExtLineBuffer[i]=ExtLineBuffer[i-1]+(price[i]-price[i-InpMAPeriod])/InpMAPeriod;
//---
  }

as firstvalue = (x1 + x2 + x3 + ... + xn) / n

which is the "crude" equasion for Simple Moving average, but not Smoothed Moving Average

 

maybe that is the reason for the issue? 

[Grzegorz Korycki]

from that site:

https://mahifx.com/indicators/smoothed-moving-average-smma 

 

The first value for the Smoothed Moving Average is calculated as a Simple Moving Average (SMA):

SUM1=SUM (CLOSE, N)

SMMA1 = SUM1/ N

The second and subsequent moving averages are calculated according to this formula:

SMMA (i) = (SUM1 – SMMA1+CLOSE (i))/ N

Where:

SUM1 – is the total sum of closing prices for N periods;
SMMA1 – is the smoothed moving average of the first bar;
SMMA (i) – is the smoothed moving average of the current bar (except the first one);
CLOSE (i) – is the current closing price;
N – is the smoothing period.

 

So i guess the iMa and custom moving averages do it in the right manner. But calculation like that produces the errors like we have encountered resulting in huge differences depending on the tested periods. It is totally unacceptable for an EA that is basing its trades around moving average. I guess I will have to eliminate Smoothed MA for that reason from my EA as it produces faulty results while backtesting. Even if test it from year 2000 and get it right, the customers may test it from 2013 and say "it wipes out the account" because they will get other SSMA than i did.

One more quote:
The SMMA gives recent prices an equal weighting to historic prices. The calculation takes all available data series into account rather than referring to a fixed period.

that is why it is different every time the back-test period changes. 

[Alain Verleyen]

angreeee:

Please don't reply inside the quote. As you see, when I want to quote you it's now empty.

The algorithm is good for SMMA, you have to begin somewhere. But you point the origin of the problem, as SMMA is build on the previous value, it's sensitive to the start condition. As you don't have the same start candle on a live chart and with the Strategy tester that explains the different values.

The same is true for EMA, after a second checking (on D1) I now have different values too, as for SMMA.

[Alain Verleyen]

angreeee:

from that site:

https://mahifx.com/indicators/smoothed-moving-average-smma 

 

The first value for the Smoothed Moving Average is calculated as a Simple Moving Average (SMA):

SUM1=SUM (CLOSE, N)

SMMA1 = SUM1/ N

The second and subsequent moving averages are calculated according to this formula:

SMMA (i) = (SUM1 – SMMA1+CLOSE (i))/ N

Where:

SUM1 – is the total sum of closing prices for N periods;
SMMA1 – is the smoothed moving average of the first bar;
SMMA (i) – is the smoothed moving average of the current bar (except the first one);
CLOSE (i) – is the current closing price;
N – is the smoothing period.

 

So i guess the iMa and custom moving averages do it in the right manner. But calculation like that produces the errors like we have encountered resulting in huge differences depending on the tested periods. It is totally unacceptable for an EA that is basing its trades around moving average. I guess I will have to eliminate Smoothed MA for that reason from my EA as it produces faulty results while backtesting. Even if test it from year 2000 and get it right, the customers may test it from 2013 and say "it wipes out the account" because they will get other SSMA than i did.

Thank you for the link. That confirm my previous post. It's very interesting as I never pay attention to such things.

By checking the algorithm of EMA it's also sensitive to such issue, I don't know why my first test got the same values.