Self-regressive Median Coefficient - page 3

New comment
 
Alain Verleyen:

You are wrong, send me the original and I will show you. I will not build the indicator from the snippets posted here.

If you are not interested it's ok.

//+------------------------------------------------------------------+
//|                                                    Quantiles.mqh |
//|                                                    Arthur Albano |
//|                            https://www.facebook.com/arthuralbano |
//+------------------------------------------------------------------+
#property copyright "Arthur Albano"
#property link      "https://www.facebook.com/arthuralbano"
#include <Math\Stat\Math.mqh>
//+------------------------------------------------------------------+
//| Computes the interpolated percentile value of an array[]         |
//| todo: check for Mendenhall & Sicich (C=0)                        |
//+------------------------------------------------------------------+
double MathPercentileValue(double &array[], const double &percent_rank, const int C=0)
  {
//---
    double percentile_value = 0.0;

//---
    int size=ArraySize(array);

//--- check data range
    if(size==0)
        return(QNaN);

//--- check for percent rank [0..1]
    if(percent_rank<0 || percent_rank>1)
        return(QNaN);
        
//--- simple case: one element
    if(size==1)
        return(array[0]);

   double rank;
   rank = (size - 1) * percent_rank;
   
   int left_rank;
   left_rank = (int)MathCeil(rank);
   double left_weight;
   left_weight = (rank + 1 ) - double(left_rank);
   
   int right_rank;
   right_rank = (int)MathFloor(rank);
   double right_weight;
   right_weight = 1 - left_weight;
   
//--- prepare sorted values
   double sorted_values[];
   ArrayResize(sorted_values,size);
   if(ArrayCopy(sorted_values,array,0,0,WHOLE_ARRAY)!=size)
      return(QNaN);
   ArraySort(sorted_values);

   //--- calculation
    percentile_value = sorted_values[left_rank]*left_weight + sorted_values[right_rank]*right_weight;
   
   return(percentile_value);
}


//+------------------------------------------------------------------+
//| Quantile Linear Regression Coefficient                           |
//+------------------------------------------------------------------+
double MathQuantLRCoeff(double &x_array[], double &y_array[], const double percent_rank = 0.5)
{
   int size = ArraySize(x_array);
   if (size<2)                   return(QNaN);
   if (size != ArraySize(y_array)) return(QNaN);

   double CoeffBuffer[];

   int coeffSize = (size*(size-1))/2; if (ArraySize(CoeffBuffer)!=coeffSize) ArrayResize(CoeffBuffer,coeffSize); 

   for(int i=0,k=0;i<(size-1);i++)
   for(int j=(i+1);j<(size);j++,k++) CoeffBuffer[k] = (y_array[j]-y_array[i])/(x_array[j]-x_array[i]); ArraySort(CoeffBuffer);
                              
   double QuantRegCoeff = MathPercentileValue(CoeffBuffer,percent_rank,0);
   return(QuantRegCoeff);
}
 

The code of the complete indicator :

//------------------------------------------------------------------
#property copyright   "mladen"
#property link        "mladenfx@gmail.com"
//------------------------------------------------------------------
#property indicator_separate_window
#property indicator_buffers 5
#property indicator_plots   2
#property indicator_label1  "Test value"
#property indicator_type1   DRAW_FILLING
#property indicator_color1  clrHotPink,clrLightGreen
#property indicator_label2  "Test value"
#property indicator_type2   DRAW_COLOR_LINE
#property indicator_color2  clrCrimson,clrLimeGreen
#property indicator_width2  2


input int                inpPeriod = 15;          // Period
input ENUM_APPLIED_PRICE inpPrice  = PRICE_CLOSE; // Price
double val[],valc[],prices[],fillu[],filld[];
//------------------------------------------------------------------
//
//------------------------------------------------------------------
int OnInit()
{
   IndicatorSetInteger(INDICATOR_DIGITS,8);
   SetIndexBuffer(0,fillu,INDICATOR_DATA);
   SetIndexBuffer(1,filld,INDICATOR_DATA);
   SetIndexBuffer(2,val,INDICATOR_DATA);
   SetIndexBuffer(3,valc,INDICATOR_COLOR_INDEX);
   SetIndexBuffer(4,prices,INDICATOR_CALCULATIONS);
   PlotIndexSetInteger(0,PLOT_SHOW_DATA,false);
   IndicatorSetString(INDICATOR_SHORTNAME,"Self-regressive Median Coefficient ("+(string)inpPeriod+")");
   return(INIT_SUCCEEDED);
}
void OnDeinit(const int reason) { return; }

//------------------------------------------------------------------
//
//------------------------------------------------------------------
int OnCalculate(const int rates_total,
                const int prev_calculated,
                const datetime& time[],
                const double& open[],
                const double& high[],
                const double& low[],
                const double& close[],
                const long& tick_volume[],
                const long& volume[],
                const int& spread[])
{                
   if (Bars(_Symbol,_Period)<rates_total) return(0);
   
   //
   //---
   //

   for (int i=(int)MathMax(prev_calculated-1,0); i<rates_total && !_StopFlag; i++)
   {
      prices[i] = getPrice(inpPrice,open,close,high,low,i,rates_total); 
      val[i]    = MathQuantRegCoeff(i,inpPeriod,time,prices);
      valc[i]   = (val[i]>0) ? 1 : 0;
      fillu[i]  = 0;
      filld[i]  = val[i];
   }                  
   return(rates_total);
}


//------------------------------------------------------------------
//
//------------------------------------------------------------------
double CoeffBuffer[]; 
double MathQuantRegCoeff(int i, int _period, const datetime &time[], double &price[])
{
   if (i<_period) return(0);
   int coeffSize = (_period*(_period-1))/2; if (ArraySize(CoeffBuffer)!=coeffSize) ArrayResize(CoeffBuffer,coeffSize); 
    
   for(int l=0,k=0; l<(_period-1); l++)
   for(int j=(l+1); j<(_period  ); j++,k++) CoeffBuffer[k] = (price[i-j]-price[i-l])/(time[i-j]-time[i-l]); ArraySort(CoeffBuffer);
                              
   double  QuantRegCoeff = (coeffSize%2==1) ? (CoeffBuffer[coeffSize/2]) : (0.5*(CoeffBuffer[(coeffSize-1)/2]+CoeffBuffer[(coeffSize+1)/2]));
   return( QuantRegCoeff );
}
//
//---
//    
double getPrice(ENUM_APPLIED_PRICE tprice,const double &open[],const double &close[],const double &high[],const double &low[],int i,int _bars)
{
   switch(tprice)
     {
      case PRICE_CLOSE:     return(close[i]);
      case PRICE_OPEN:      return(open[i]);
      case PRICE_HIGH:      return(high[i]);
      case PRICE_LOW:       return(low[i]);
      case PRICE_MEDIAN:    return((high[i]+low[i])/2.0);
      case PRICE_TYPICAL:   return((high[i]+low[i]+close[i])/3.0);
      case PRICE_WEIGHTED:  return((high[i]+low[i]+close[i]+close[i])/4.0);
     }
   return(0);
  }

It draws this :


The code used in this indicator is different (different, faster,  way of preparing and passing array elements but the results are the same)  The 2 lines of code that need optimization remain the same (as it is obvious, since, practically, it is all that remains in the function calculation code)

 
Alain Verleyen:

You are wrong, send me the original and I will show you. I will not build the indicator from the snippets posted here.

If you are not interested it's ok.

Hi Alain,

I'm just curious if it's possible to write the part of code (I've marked it in the code) in more efficient way. Therefore I've written my own indicator and I'll be glad if you show me your approach.

Thank you in advance.

Files:
NTC_LinearPrediction.mq4  6 kb
 
Mladen Rakic:

The code of the complete indicator :

It draws this :


The code used in this indicator is different (different, faster,  way of preparing and passing array elements but the results are the same)  The 2 lines of code that need optimization remain the same (as it is obvious, since, practically, it is all that remains in the function calculation code)

Thanks. I will work on that.
 
Petr Nosek:

Hi Alain,

I'm just curious if it's possible to write the part of code (I've marked it in the code) in more efficient way. Therefore I've written my own indicator and I'll be glad if you show me your approach.

Thank you in advance.

I will work on Mladen code, which is mql5. You will just have to apply it to your code if I succeed ;-)
 
Alain Verleyen:
I will work on Mladen code, which is mql5. You will just have to apply it to your code if I succeed ;-)

It's OK. The code is pretty similar. The difference is that I don't use time[] as an axis "x" because time axis is discontinuous in MT (because of e.g. weekends an so on). But I'm sure that I'll be able to understand your MQL5 code.

If you're successful I'll call you "Master of efficiency" ;-)

Good luck.

 

So at my surprise, I didn't succeed. The main optimization idea I had was to reduce the number of division operations needed : 

   for(int l=0,k=0; l<(_period-1); l++)
      for(int j=(l+1); j<(_period); j++,k++) CoeffBuffer[k]=(price[i-j]-price[i-l])/(time[i-j]-time[i-l]);

As this operation is done in a loop for every candle i, there are a lot of repetitions of the exact same operation. So the idea was to do all the operations once and to memorize them (see attached how I did). However it doesn't improve this speed, even while the numbers of operations was reduced by a factor 16 !

From 64 millions to 4 millions division operations, but no change in execution time. I didn't expect that. That means double arithmetic CPU is very efficient and cached very well all the results.

Also, though this imbricated loop with division operations is time consuming, the main bottleneck in the ArraySort(), the speed impact is more than 3 times the one of the loops. So even if the "division" optimization had worked that global impact would have been low (~20% max).

That was an interesting exercise, even if it failed.

Attached the code (as we are on week-end I didn't pay attention to "live" update).

Files:
275327_AV.mq5  10 kb
 
You don't need to calculate all. Just bars-1. And then find the median for the given period. So, with 129122 bars, calculate 129121 coefficients, and then pick up the median for period. Something like a global coefficient buffer. So coefficients are calculated only at new bars. Is it possible?
 
Alain Verleyen:

So at my surprise, I didn't succeed. The main optimization idea I had was to reduce the number of division operations needed :

As this operation is done in a loop for every candle i, there are a lot of repetitions of the exact same operation. So the idea was to do all the operations once and to memorize them (see attached how I did). However it doesn't improve this speed, even while the numbers of operations was reduced by a factor 16 !

From 64 millions to 4 millions division operations, but no change in execution time. I didn't expect that. That means double arithmetic CPU is very efficient and cached very well all the results.

Also, though this imbricated loop with division operations is time consuming, the main bottleneck in the ArraySort(), the speed impact is more than 3 times the one of the loops. So even if the "division" optimization had worked that global impact would have been low (~20% max).

That was an interesting exercise, even if it failed.

Attached the code (as we are on week-end I didn't pay attention to "live" update).

Thank you for your effort. Don't be sad I won't call you "Master of efficiency" :D

I really appreciate your approach. Even though you've failed you are still honest (unlike others).

BTW I think that the better approach for OP's needs is using Linear Regression or Kalman filter or something similar.

 
Arthur Albano:
You don't need to calculate all. Just bars-1. And then find the median for the given period. So, with 129122 bars, calculate 129121 coefficients, and then pick up the median for period. Something like a global coefficient buffer. So coefficients are calculated only at new bars. Is it possible?
Well that's not so simple. Considering a period of 32, for 129122 bars, there are 31*129122 coefficients, not 129121. (Neglecting the fact that older bars doesn't have enough data to be calculated correctly).
1234567
New comment