Indicator returns historically correct values, but not on new data

Scott 2008.10.30 03:26

Hi all,

Long time reader, first post.

I have a problem similar to ser01 (see : https://forum.mql4.com/16576). I have used a temporary array in my indicator (as suggested in that thread) but the results I get from my indicator are not "right".

When I first apply the indicator to the chart, the values returned are correct and get a "nice" curve. As new data arrives, the indicator is recomputed but the return values are not correct. If I recompile the indicator or change the indicator properties (or force the indicator to recompute entirely) the correct values are displayed. If I remove from the code, the use of IndicatorCounted() and limits, and force the indicator to recompute for all data on every tick (very slow and CPU intensive) I get the right results.

Hopefully someone here can help get this indicator right and might even find it useful for their own purposes?

(Hopefully my code and pics will appear below?)

//+------------------------------------------------------------------+
//|                                          ZeroParameterFilter.mq4 |
//|                                   Copyright © 2008 wabbit.com.au |
//|                                         http://www.wabbit.com.au |
//+------------------------------------------------------------------+
#property copyright "Copyright © 2008 wabbit.com.au"
#property link      "http://www.wabbit.com.au"

/*
From: http://www.nrcm.de/php/zpf.php

The zero-parameter filter (ZPF_NRCM) is an adaptive filter, the only input needed is the time series to be filtered.
For the calculation of the filter will increase (Slope) a local linear regression function F(x) determined by the method of least squares.
The continuity of the filter curve is a pairing of current and previous regression lines obtained.
The adaptive smoothing measure of the filter and the length of the regression line are dependent on the determination (B) or the square of the correlation coefficients.
Thus the sensitivity of the filter depends on the quality of the regression function in relation to the regression data.
If the data variability is explained completely by the involution function (B=1), then the sensitivity of the filter is high.
In this case, the delay has a minimal value of 1 (LAG) and the regression line a minimum length of N = 3 (sample size).
The current value of the filter is located in the vicinity of the mean of the regression line:

Filter(t) =  F(x=2)

If the data variability is small (B=0), then the sensitivity of the filter is also small and the momentary filter value is in the proximity of the initial value of the regression line:

Filter(t) = F(x=N),

whose length with each time step increases as a unit of time.

The application of the filter is not limited to the smoothing of open - high - low - close - or tick - data.
In principle, any arbitrary time series (such as indicators, capital curve, etc.) be used as input data.

The zero-parameter filter was developed in collaboration with Dr. Andreas Uhl.
*/

#property indicator_chart_window
#property indicator_buffers 1
#property indicator_color1 Violet
#property indicator_width1 3
#property indicator_style1 DRAW_LINE

double ZPF[];
double temp[];

int bar;
double data;
double almostZero;

int index = 0;
int N;
double slope;
double rope = 1;
double tau0;
double dtau;
double wavg;
double sst;
double sx0;
double ssx0;
double cod;

int counted_bars = 0;
int limit;

int init()
{
   SetIndexBuffer(0,ZPF);
   ArraySetAsSeries(temp,true);
   
   //very small number to replace zero
   almostZero=MathPow(10,-20);
   
	return(0);
}

int deinit()
{
	return(0);
}

int start()
{
	ArrayResize(temp,Bars);
	
	int limit = Bars - IndicatorCounted() - 1;
	int i, x;
	
	for(i=limit; i>=0; i--)
	{
		sx0 = 0.0;
		ssx0 = 0.0;
		wavg = 0.0;
		sst = 0.0;

		tau0 = rope + 1;
		N = MathFloor(tau0);

		if(i==Bars-1)
		{
			temp[i] = ( Open[i] + High[i] + Low[i] + Close[i] ) / 4;
		}
		else
		{
			for(x=0; x<N; x++)
			{
				bar = i + x;
				data = ( Open[bar] + High[bar] + Low[bar] + Close[bar] ) / 4;

				sx0 += (tau0 - x);
				ssx0 += MathPow(tau0 - x,2);
				wavg += (tau0 - x) * data;
				sst += MathPow(data - temp[i+1], 2);	
			}

			//it's better to avoid errors than catch them...
			if(sx0 == 0.0) sx0 = almostZero;
			if(ssx0 == 0.0) ssx0 = almostZero;
			if(sst == 0.0) sst = almostZero;

			wavg = wavg/sx0;
			slope = (wavg - temp[i+1]) * sx0 / ssx0;
			cod = MathPow(slope, 2) * ssx0 / sst;
			dtau = cod * (tau0 - 2);
			rope = tau0 - dtau;
			temp[i] = temp[i+1] + dtau * slope;
		}
	}
	
	for(i=limit; i>=0; i--)
		ZPF[i] = temp[i];

	return(0);
}

Here is a pic when the indicator was first applied to the chart and left to run for a few minutes:

First application of indicator to chart

Here is a pic after the indicator has been "re-initialized" or fully recomputed:

Fully re-evaluated indicator

Notice how the indicator looks different after the first arrow.

I hope this helps someone to solve this issue.

wabbit :D

Ray 2008.10.30 04:32 #1

Insert this code at the end:

for(i=limit; i>=0; i--)
ZPF[i] = temp[i];

// insert
static int oldTime;
if(oldTime != Time[0]){
   oldTime = Time[0];
   Print("NEW BAR -- temp array size = ", ArraySize(temp));
   Print("temp[5] = ", temp[5]);
   Print("temp[4] = ", temp[4]);
   Print("temp[3] = ", temp[3]);
   Print("temp[2] = ", temp[2]);
   Print("temp[1] = ", temp[1]);
   Print("temp[0] = ", temp[0]);
   Print("----------------");
}
// insert end
return(0);

Tell me what you see after a few minutes.

Scott 2008.10.30 05:55 #2

phy:

Insert this code at the end:

Tell me what you see after a few minutes.

Hi phy,

Thanks for the interest.

I am doing some stress testing to see how long all of the processing takes, so please ignore the large array sizes (I don't normally have 1.3 million data points loaded!)

I added the relevant code snippets and the results are below:

13:49:16 ZeroParameterFilter EURUSDm,M1: loaded successfully
13:49:26 ZeroParameterFilter EURUSDm,M1: initialized
13:49:28 ZeroParameterFilter EURUSDm,M1: NEW BAR -- temp array size = 1384837
13:49:28 ZeroParameterFilter EURUSDm,M1: temp[5] = 1.3213
13:49:28 ZeroParameterFilter EURUSDm,M1: temp[4] = 1.3215
13:49:28 ZeroParameterFilter EURUSDm,M1: temp[3] = 1.3216
13:49:28 ZeroParameterFilter EURUSDm,M1: temp[2] = 1.3217
13:49:28 ZeroParameterFilter EURUSDm,M1: temp[1] = 1.3217
13:49:28 ZeroParameterFilter EURUSDm,M1: temp[0] = 1.3215
13:49:28 ZeroParameterFilter EURUSDm,M1: ----------------
13:49:31 ZeroParameterFilter EURUSDm,M1: NEW BAR -- temp array size = 1384838
13:49:31 ZeroParameterFilter EURUSDm,M1: temp[5] = 1.3213
13:49:31 ZeroParameterFilter EURUSDm,M1: temp[4] = 1.3215
13:49:31 ZeroParameterFilter EURUSDm,M1: temp[3] = 1.3216
13:49:31 ZeroParameterFilter EURUSDm,M1: temp[2] = 1.3217
13:49:31 ZeroParameterFilter EURUSDm,M1: temp[1] = 1.3215
13:49:31 ZeroParameterFilter EURUSDm,M1: temp[0] = 1.321
13:49:31 ZeroParameterFilter EURUSDm,M1: ----------------
13:50:05 ZeroParameterFilter EURUSDm,M1: NEW BAR -- temp array size = 1384839
13:50:05 ZeroParameterFilter EURUSDm,M1: temp[5] = 1.3213
13:50:05 ZeroParameterFilter EURUSDm,M1: temp[4] = 1.3215
13:50:05 ZeroParameterFilter EURUSDm,M1: temp[3] = 1.3216
13:50:05 ZeroParameterFilter EURUSDm,M1: temp[2] = 1.3217
13:50:05 ZeroParameterFilter EURUSDm,M1: temp[1] = 1.3212
13:50:05 ZeroParameterFilter EURUSDm,M1: temp[0] = 1.3209
13:50:05 ZeroParameterFilter EURUSDm,M1: ----------------
13:51:02 ZeroParameterFilter EURUSDm,M1: NEW BAR -- temp array size = 1384840
13:51:02 ZeroParameterFilter EURUSDm,M1: temp[5] = 1.3213
13:51:02 ZeroParameterFilter EURUSDm,M1: temp[4] = 1.3215
13:51:02 ZeroParameterFilter EURUSDm,M1: temp[3] = 1.3216
13:51:02 ZeroParameterFilter EURUSDm,M1: temp[2] = 1.3217
13:51:02 ZeroParameterFilter EURUSDm,M1: temp[1] = 1.3211
13:51:02 ZeroParameterFilter EURUSDm,M1: temp[0] = 1.3207
13:51:02 ZeroParameterFilter EURUSDm,M1: ----------------
13:52:01 ZeroParameterFilter EURUSDm,M1: NEW BAR -- temp array size = 1384841
13:52:01 ZeroParameterFilter EURUSDm,M1: temp[5] = 1.3213
13:52:01 ZeroParameterFilter EURUSDm,M1: temp[4] = 1.3215
13:52:01 ZeroParameterFilter EURUSDm,M1: temp[3] = 1.3216
13:52:01 ZeroParameterFilter EURUSDm,M1: temp[2] = 1.3217
13:52:01 ZeroParameterFilter EURUSDm,M1: temp[1] = 1.3211
13:52:01 ZeroParameterFilter EURUSDm,M1: temp[0] = 1.321
13:52:01 ZeroParameterFilter EURUSDm,M1: ----------------
13:53:04 ZeroParameterFilter EURUSDm,M1: NEW BAR -- temp array size = 1384842
13:53:04 ZeroParameterFilter EURUSDm,M1: temp[5] = 1.3213
13:53:04 ZeroParameterFilter EURUSDm,M1: temp[4] = 1.3215
13:53:04 ZeroParameterFilter EURUSDm,M1: temp[3] = 1.3216
13:53:04 ZeroParameterFilter EURUSDm,M1: temp[2] = 1.3217
13:53:04 ZeroParameterFilter EURUSDm,M1: temp[1] = 1.3212
13:53:04 ZeroParameterFilter EURUSDm,M1: temp[0] = 1.3211
13:53:04 ZeroParameterFilter EURUSDm,M1: ----------------

What specifically are you looking for?

wabbit :D

Bar Bar Data Upon Restart ASCTrend system

Ray 2008.10.30 06:01 #3

Don't you think it odd that temp 2-3-4-5 never change?

with each new bar I would expect the values to move 'into the past"

Scott 2008.10.30 06:23 #4

phy:

Don't you think it odd that temp 2-3-4-5 never change?

with each new bar I would expect the values to move 'into the past"

Yes I noticed that, but I haven't noticed anything that is going to tell me why these values are not changing.

Having read the MQL documentation quite a few times, I am still intrigued why the authors chose to implement the series array structures they way they have. I would have thought it much more efficient to index the data from 0 from the left so that as each new bar is formed it takes the preceding index + 1 as its index value. Having the latest bar as index 0 then shuffling all the previous values before it along seems quite odd to me? Maybe this has something to do with the performance of this indicator?

wabbit :D

Indicator buffer issue with new indicator structure .. static array ?

Ray 2008.10.30 06:35 #5

Having the latest bar as index 0 then shuffling all the previous values before it along seems quite odd to me?

You would have to do the shuffling yourself with a series array.

Try this. Make temp a second indicator index:

int init()
{
   IndicatorBuffers(2);
   SetIndexBuffer(0,ZPF);
   SetIndexBuffer(1,temp);
   //ArraySetAsSeries(temp,true);

and

   //ArrayResize(temp,Bars);

Ray 2008.10.30 07:01 #6

Hmmm... even that goes sour sometimes...

Scott 2008.10.30 07:17 #7

phy:

You would have to do the shuffling yourself with a series array.

Try this. Make temp a second indicator index:

and

Success?

It appears the data is being shuffled through the temp array now that ArrayBuffers() has been initialised. (I guess I will have to do some more study into the differences between array and buffers?)

Anyway, here is the relevant log extract:

15:03:01 ZeroParameterFilter EURUSDm,M1: NEW BAR -- temp array size = 1384912
15:03:01 ZeroParameterFilter EURUSDm,M1: temp[5] = 1.3224
15:03:01 ZeroParameterFilter EURUSDm,M1: temp[4] = 1.3227
15:03:01 ZeroParameterFilter EURUSDm,M1: temp[3] = 1.323
15:03:01 ZeroParameterFilter EURUSDm,M1: temp[2] = 1.3232
15:03:01 ZeroParameterFilter EURUSDm,M1: temp[1] = 1.3232
15:03:01 ZeroParameterFilter EURUSDm,M1: temp[0] = 1.3232 <--
15:03:01 ZeroParameterFilter EURUSDm,M1: ----------------
15:04:01 ZeroParameterFilter EURUSDm,M1: NEW BAR -- temp array size = 1384913
15:04:01 ZeroParameterFilter EURUSDm,M1: temp[5] = 1.3227
15:04:01 ZeroParameterFilter EURUSDm,M1: temp[4] = 1.323
15:04:01 ZeroParameterFilter EURUSDm,M1: temp[3] = 1.3232
15:04:01 ZeroParameterFilter EURUSDm,M1: temp[2] = 1.3232
15:04:01 ZeroParameterFilter EURUSDm,M1: temp[1] = 1.3232 <----
15:04:01 ZeroParameterFilter EURUSDm,M1: temp[0] = 1.3231
15:04:01 ZeroParameterFilter EURUSDm,M1: ----------------
15:05:00 ZeroParameterFilter EURUSDm,M1: NEW BAR -- temp array size = 1384914
15:05:00 ZeroParameterFilter EURUSDm,M1: temp[5] = 1.323
15:05:00 ZeroParameterFilter EURUSDm,M1: temp[4] = 1.3232
15:05:00 ZeroParameterFilter EURUSDm,M1: temp[3] = 1.3232
15:05:00 ZeroParameterFilter EURUSDm,M1: temp[2] = 1.3232 <----
15:05:00 ZeroParameterFilter EURUSDm,M1: temp[1] = 1.3227
15:05:00 ZeroParameterFilter EURUSDm,M1: temp[0] = 1.3221
15:05:00 ZeroParameterFilter EURUSDm,M1: ----------------
15:06:00 ZeroParameterFilter EURUSDm,M1: NEW BAR -- temp array size = 1384915
15:06:00 ZeroParameterFilter EURUSDm,M1: temp[5] = 1.3232
15:06:00 ZeroParameterFilter EURUSDm,M1: temp[4] = 1.3232
15:06:00 ZeroParameterFilter EURUSDm,M1: temp[3] = 1.3232 <----
15:06:00 ZeroParameterFilter EURUSDm,M1: temp[2] = 1.3227
15:06:00 ZeroParameterFilter EURUSDm,M1: temp[1] = 1.322
15:06:00 ZeroParameterFilter EURUSDm,M1: temp[0] = 1.3211
15:06:00 ZeroParameterFilter EURUSDm,M1: ----------------
15:07:00 ZeroParameterFilter EURUSDm,M1: NEW BAR -- temp array size = 1384916
15:07:00 ZeroParameterFilter EURUSDm,M1: temp[5] = 1.3232
15:07:00 ZeroParameterFilter EURUSDm,M1: temp[4] = 1.3232 <----
15:07:00 ZeroParameterFilter EURUSDm,M1: temp[3] = 1.3227
15:07:00 ZeroParameterFilter EURUSDm,M1: temp[2] = 1.322
15:07:00 ZeroParameterFilter EURUSDm,M1: temp[1] = 1.3211
15:07:00 ZeroParameterFilter EURUSDm,M1: temp[0] = 1.3203
15:07:00 ZeroParameterFilter EURUSDm,M1: ----------------

15:08:02 ZeroParameterFilter EURUSDm,M1: NEW BAR -- temp array size = 1384917
15:08:02 ZeroParameterFilter EURUSDm,M1: temp[5] = 1.3232 <----
15:08:02 ZeroParameterFilter EURUSDm,M1: temp[4] = 1.3227
15:08:02 ZeroParameterFilter EURUSDm,M1: temp[3] = 1.322
15:08:02 ZeroParameterFilter EURUSDm,M1: temp[2] = 1.3211
15:08:02 ZeroParameterFilter EURUSDm,M1: temp[1] = 1.3203
15:08:02 ZeroParameterFilter EURUSDm,M1: temp[0] = 1.3198
15:08:02 ZeroParameterFilter EURUSDm,M1: ----------------

Thank you very much (phy) for your excellent guidance.

I still ponder though, why do we have to use temporary arrays/buffers in the first place? Could this line:

temp[i] = temp[i+1] + dtau * slope;
not be

ZPF[i] = ZPF[i+1] + dtau * slope;

Or is this just another one of those "mysteries of the deep?" I might have to do some more playing around to see what other effects I can cause/see what happens?

wabbit :D

Looking for an EA Help, can't understand MT4 Simple Indicator from CSV

Ray 2008.10.30 08:02 #8

Don't know, have to stare at the code some more.

I hate trying to figure out what other perople wrote.

New comment