Indicator shifts when terminal adds/removes chart bars

[VSc]

Problem:

While it is running, MT4 will load or remove bars in a chart (to achieve/maintain the "Max Bars In Chart" value), and when it does, my "separate window" indicator shifts by the number of bars that are added/removed. My indicator is my own tick chart, and therefore the indicator indexes don’t relate directly to the corresponding bars in the chart window, plus they are based on individual ticks, so I can't simply redraw/recalculate.

I have attached the tick chart mql file for reference. (It simply uses histogram bars for the tick chart, and has a shift function that moves all indicator bars over to the left if ever the next tick chart bar is about to be drawn at Time[0]. This shift function is NOT related to the problem).

Example:

MT4 is set to have max 20000 bars.

Once more bars have been added to the chart (564 in this instance, but it varies, and you don't know the value until it happens), the terminal deletes 128 bars (this can be seen by the change to the value of the Bars variable). At this time, the indicator value that I have drawn at (e.g.) buffer index[135] will now appear at index[7] - the indicator values all "shift" to the right in the indicator window. Likewise the value drawn at index[5] will disappear from the indicator window, but it has actually shifted to the right (into negative index values) and will gradually reappear as the chart progresses, which makes the indicator a mess. Then every time "Bars" reaches 20,564 again, the process repeats.

The following screens show the indicator over a period of ~0.8 seconds, at the time the chart adjusts the bars:

Conversely, if you increase the "maximum bars" value, it shifts to the left when MT4 downloads more history.

I have tried MT4 installers from 2 different brokers, using each broker’s accounts and trade servers, and on 3 different PC's/Windows versions (8, 10, server 2012 VPS), using different values for “Max bars in chart” (20,000/2,100) and they all do the same thing. I am using Build 1220 right now, but this has been a problem I have been aware of for maybe a year, I have only just isolated the catalyst.

I have searched the forums and documentation and I am now asking for help please.

So what I need to know is – 

is this an error in MT4, or my indicator?

Or is it neither and actually the standard behaviour in MT4?

Is there a fix, a property I can set to overcome it, or do I have to code around it?

Thanks for taking the time to read, and TIA for any assistance.

Vaughan

[Alain Verleyen]

VSc:
...

So what I need to know is – 

is this an error in MT4, or my indicator?

Or is it neither and actually the standard behaviour in MT4?

It is the normal behaviour.

Is there a fix, a property I can set to overcome it, or do I have to code around it?

You need to code your indicator properly to deal with it. Suggestion : use a normal buffer to keep your values, and then ArrayCopy() to set the values of your indicator buffer (which is managed automatically by MT4, so you don't have control  on it).

Thanks for taking the time to read, and TIA for any assistance.

Vaughan

Please write correct English, what is the meaning of TIA ?

[VSc]

Alain Verleyen:
It is the normal behaviour.

You need to code your indicator properly to deal with it. Suggestion : use a normal buffer to keep your values, and then ArrayCopy() to set the values of your indicator buffer (which is managed automatically by MT4, so you don't have control  on it).

Please write correct English, what is the meaning of TIA ?

Hi Alain,

Thanks for the explanation and guidance. Knowing that it's not a bug is half the battle.

Because I am doing tick-by-tick comparisons for my indicator, I need the code as lean as possible so I don't miss any ticks. Additional processing is undesirable, and I was hoping it was a behaviour that could be overcome by a setting in the terminal rather than with code. I will try ArrayCopy() and see if I can make it run only when the chart shifts.

TIA = "Thanks In Advance".

Vaughan

[VSc]

Please help me now to clear the indicator buffer properly, or determine that I can't.

Firstly I have modified my tick chart indicator (as suggested by @Alain Verleyen) so that it loads a regular array and then does an ArrayCopy into the indicator buffer once each tick bar is "full". When the chart shifts due to history being loaded/removed, it does another ArrayCopy to re-draw the indicator. My problem now is that I don't know how to fully clear the indicator array before I redraw it, after this shift has taken place.

I have experimented with various functions, but none of them clear the values that have moved into the negative index positions of the indicator buffer. 

To demonstrate, refer to the second image above (reproduced here):

It's not only the indicator values inside the red box that move to the right - all of the "bars" in the indicator window move to the right, but those outside the red box move off the screen - you can't see them but the values are still held in the indicator buffer. As new M1 bars appear and the terminal moves the chart and indicator windows along one position, these "bars" reappear in the indicator window, at Time[0], one-by-one.

Here's a screenshot showing my updated indicator after the chart has shifted due to bars being removed from history. As the terminal advances the chart window, it's bringing indicator values from negative index positions back into the 0+ indexes where they are visible. You can even see highlighted red where these bars originally came from and have been redrawn by me, to prove they are not random values.

I have tried ArrayFree() but it doesn't appear to do what I thought it might. Either that or I am not using it properly. When I do this:

ArrayFree(IndBuffer1);

All of the values drawn in the indicator window (for this buffer) disappear, but when you ArrayCopy() back into it, nothing shows in the indicator window until you do this:

SetIndexBuffer(0, IndBuffer1);

And then it shows that the indicator has held its values and wasn't actually "freed". The way I read the doco it actually freed the mapping to physical memory. Maybe it does, and SetIndexBuffer() remaps it - I need to blow away what is in memory because I can't seem to get the values out of the indicator buffer any other way.

And ArrayInitialize() doesn't initialise the negative index values.

Finally, as I mentioned above, I do a tick-by-tick comparison, so missing ticks is highly undesirable, that's why I want this code to be as lean as possible, and I don't want to have to run IndBuffer1[0] = EMPTY_VALUE every time a new M1 chart bar is formed, if only I could clear out those shifted values. But if I can't find a way to reset the entire indicator buffer in one go, I don't see any other choice?

Thanks,

Vaughan

[Alain Verleyen]

VSc:
...

I am not sure I understand your problem. Why isn't your tick values not drawn from bar 0 ? (on your second screenshot).

I can't find a way to reset the entire indicator buffer in one go

ArrayInitialize(IndBuffer1,EMPTY_VALUE);

"I don't want to have to run IndBuffer1[0] = EMPTY_VALUE every time a new M1 chart bar is formed"

You always have to set a buffer value to a new candle, otherwise it will be random value.

[VSc]

Alain Verleyen:

I am not sure I understand your problem. Why isn't your tick values not drawn from bar 0 ? (on your second screenshot).

Unless your tick bars are very large, they will (at times) be drawn faster than the M1 bars. So every time a tick bar completes at bar 0, the indicator values have to shift along one position to draw the next one at bar 0. But I want lean code, so why run code to shift the indicator on every tick bar if I don't have to? Instead when the latest bar is drawn at bar 0, I shift the indicator maybe 15 positions, then draw the next bar at 14, 13 and so on - that way the code that shifts the indicator is not running every tick bar (only every 15 at least). Then for each tick bar other than every 15, I am merely decrementing an int that holds the current position being drawn - a much faster solution.

ArrayInitialize(IndBuffer1, EMPTY_VALUE)

This doesn't initialise IndBuffer1[-1], IndBuffer1[-2] etc. When the chart removes history, it does it 128 bars at a time, so the terminal has pushed values all they way up (down?) to IndBuffer[-128], and ArrayInitialize doesn't reach these negative indicator buffer values. That's what I thought ArrayFree might achieve but it doesn't.

You always have to set a buffer value to a new candle, otherwise it will be random value.

While this may be the case, I have never seen an indicator buffer have a value other than EMPTY_VALUE on a new candle.

My indicator is fast becoming full of code that is written just to handle the chart shifting, which will compromise its accuracy as it misses ticks. I guess the reality is I am just trying to use MT4 indicators in a way they are not designed. All because when the terminal removes history, it shifts the indicator values somewhere that they can't be reached, and has to be redrawn.

Does anyone know if this same behaviour happens in MT5?

Thanks again @Alain Verleyen, your assistance is greatly appreciated.

Vaughan

[carling1]

Hi VSC

A bit belatedly.

I encountered a similar problem where after every 128 bars the drawn values of an indicator have been shifted by 256 bars to the right.

I would love to know what is happening here.

My current solution is to detect that this error occurred in which case the indicator is forced to refresh from the start.

The method for detecting the condition is to save the current value and time of one point of a (changing) buffer. At the end of each bar that value is looked up again using iBarShift, if the value has changed then a redraw is initiated.

[William Roeder]

carling1: I would love to know what is happening here.

Do you really expect an answer? There are no mind readers here and our crystal balls are cracked.

We can't see your broken code.
How To Ask Questions The Smart Way. 2004
          Be precise and informative about your problem

Fix your broken code.

With the information you've provided — we can only guess. And you haven't provided any information for that.

[carling1]

William Roeder:

Do you really expect an answer? There are no mind readers here and our crystal balls are cracked.

We can't see your broken code.
How To Ask Questions The Smart Way. 2004
          Be precise and informative about your problem

Fix your broken code.

With the information you've provided — we can only guess. And you haven't provided any information for that.

I am so happy to get a response.  

My description of the problem is probably not clear.  So, here is a more detailed "visual" explanation.  The following picture shows the discrepancy where (for some unknown reason) MT4 redraws the chart with a section of 128 bars effectively removed.

Note that the problem only occurs after 128 bars where the chart has been "undisturbed", no timeframe change.  On M1 this means 2 hours and 8 minutes.  On M5 it will be 5 times longer, etc.

The indicator drawn at the bottom is actually very simple (in this case) it is merely the close of the chart, but here it goes:

(It includes my test whether and when to redraw).

//+------------------------------------------------------------------+
//|                                                    TestClose.mq4 |
//|                                                          carling |
//|                                                              cgn |
//+------------------------------------------------------------------+
#property copyright "carling"
#property link      "cgn"
#property version   "1.00"
#property strict
#property indicator_separate_window
#property indicator_buffers 1

#include <cgnFunctions.mqh>
extern color   Color_0           = clrRed ;
extern int     History           = 10000;
//+------------------------------------------------------------------+
//| Custom indicator initialization function                         |
//+------------------------------------------------------------------+
double testbuffer0[];
datetime dte_lastevent;
int OnInit()
  {
//--- indicator buffers mapping
         SetIndexBuffer(0,testbuffer0);
         SetIndexLabel(0,"testbuffer0"); 
         SetIndexStyle(0,DRAW_LINE,STYLE_SOLID,2,Color_0);
   
//---
   return(INIT_SUCCEEDED);
  }
//+------------------------------------------------------------------+
//| Custom indicator iteration function                              |
//+------------------------------------------------------------------+
int OnCalculate(const int rates_total,
                const int prev_calculated,
                const datetime &time[],
                const double &open[],
                const double &high[],
                const double &low[],
                const double &close[],
                const long &tick_volume[],
                const long &volume[],
                const int &spread[])
  {
//---
   static datetime time_lastbar = 0;
   static bool initialised = false;
   int counted_bars = IndicatorCounted(); // this is one less than the prev_calculated passed to OnCalculate

  // process bar(s)
  if(!NewBar(dte_lastevent,0,NULL, Period())) return(rates_total); // Do an update every period - calculate for each bar on the chart to set the buffer values accordingly


   if(initialised) CheckRedraw(testbuffer0);    // the buffer needs to have some data before we can check it
   //if(initialised && CheckRedraw(testbuffer0))
   //{
   //   // a redraw is required
   //   counted_bars = 0;
   //   time_lastbar = 0;   
   //}
   int limit = Bars - counted_bars;
   if (limit>History)                 // If too many bars ..
      limit=History;                  // ..calculate for specific amount.
   
   for(int i=limit-1; i>=1; i--)    // not updating the current bar
   {
      if(Time[i] <= time_lastbar)   continue;     // may not reprocess a bar 
      
      // calculate for this bar
      // .. not much here
      testbuffer0[i] = close[i];
   }
   
   if(!initialised)     
      CheckRedraw(testbuffer0);
   initialised = true;   
   time_lastbar = Time[1];
   
//--- return value of prev_calculated for next call
   return(rates_total);
  }
//+------------------------------------------------------------------+

bool CheckRedraw(double &buf[])
{
   static datetime timelastcheck=0;
   static double valuelastcheck = 0;
   bool ret = false;
   // check if the drawn indicator has been screwed up
   //if(cdata.test)
   {
      if(timelastcheck == 0)
      {
         timelastcheck = Time[1];
         valuelastcheck = buf[1];
         PrintFormat("Started redraw check for times %s, close %f ******", TimeToStr(timelastcheck),buf[1]);
         return false;
      }
   
      // check if the values have changed
      int ibar = iBarShift(NULL,Period(),timelastcheck,true);
      if(valuelastcheck != buf[ibar])
      {
         PrintFormat("Close mismatch ********************************");         
         PrintFormat("Restart new check from time %s, to time %s, buf %f",TimeToStr(timelastcheck), TimeToStr(Time[1]), buf[1]);
         timelastcheck = Time[1];
         valuelastcheck = buf[1];

         // perform redraw
         ret = true;
      
      }
   }
   return(ret);
} 

[Keith Watford]

Please edit your post and

use the code button (Alt+S) when pasting code