Strange error returning 'rates_total' in OnCalculate function on an indicator

New comment
 

Hello everybody,

I was trying to solve this well-known problem explained by Fabio Cavalloni and others, when I have discovered some really strange behaviour in OnCalculate. 

I have written a small sample program that reads the closing value in a different timeframe. Inside onCalculate, I check whether iBarShift is ready or not:

  • If iBarShift returns -1, then set up a 1 second timer that refreshes the chart using this trick explained by Alain Verleyen.
  • If iBarShift doesn't return -1, then kill the Timer, because there's no need to refresh the chart anymore.

Using the 1 second timer, and refreshing the chart after that, works like switching to a different timeframe manually and returning back. 

This is the code, which I have tried on american stocks outside market hours:

#property copyright "Copyright 2020, Carlos"
#property indicator_chart_window
#property indicator_buffers 1    // buffers used
#property indicator_plots   1    // buffers displayed

//--- Indicator Properties
#property indicator_label1  "Close Value in Other TF"
#property indicator_type1   DRAW_LINE
#property indicator_color1  Blue
#property indicator_style1  STYLE_SOLID
#property indicator_width1  2

#define SECONDS_TO_WAIT 1

//--- input parameters
input ENUM_TIMEFRAMES other_TimeFrame = PERIOD_D1;   

//--- Indicator buffer
double otherTF_Buffer[];

int offset;

int OnInit()
{
   //--- indicator buffers mapping
   SetIndexBuffer(0, otherTF_Buffer, INDICATOR_DATA);
   IndicatorSetInteger(INDICATOR_DIGITS,_Digits);
   PlotIndexSetDouble(0, PLOT_EMPTY_VALUE, EMPTY_VALUE);
   
   double tf1 = PeriodSeconds(_Period);
   double tf2 = PeriodSeconds(other_TimeFrame);
   offset = (int)ceil(tf2/tf1);

   return(INIT_SUCCEEDED);
}


bool check_ibarshift_in_Other_TF(datetime t)
{
   int res;
   
   res = iBarShift(NULL, other_TimeFrame, t);
   Print("check_ibarshift(date = ", t, ") ==> iBarShift = ", res); 
   
   // If iBarShift is not ready and returns error, then we set a 1 second timer 
   // to let the history load. This will let  us refresh the chart in absence 
   // of OnCalculate events (as for example, during weekends, or for some stocks
   // that close during the day).
   if(res == WRONG_VALUE)
   {
      Print("Setting Timer");
      EventSetTimer(SECONDS_TO_WAIT);
      Comment("LOADING HISTORY ... WAIT, PLEASE");
      return false;
   }
   
   // If enough bars are loaded, kill the timer
   Comment(""); 
   Print("Killing Timer");
   EventKillTimer();  
   return true;
}

// After 1 second has gone by, the Timer event is generated, and 
// this function is called. During that second, mt5 probably has 
// had enough time to load the history for the other timeframe. 
void OnTimer(void)
{
   Print("Inside OnTimer, calling ChartSetSymbolPeriod to refresh the chart.");
   ChartSetSymbolPeriod(0,NULL,0);
}


int OnCalculate(const int rates_total,
                const int prev_calculated,
                const datetime &time[],
                const double &open[],
                const double &high[],
                const double &low[],
                const double &close[],
                const long &tick_volume[],
                const long &volume[],
                const int &spread[])
{
   int    start; 
   int    shift;  
   double c;     // close value in the other timeframe
   
   Print("On Calculate ==> Rates total = ", rates_total, "; Prev_calculated = ", prev_calculated);
   //--- Detect start position
   if(prev_calculated == 0) 
   {
      if(check_ibarshift_in_Other_TF(time[rates_total-1]) == false) return 0;
      
      start = 1;
      ArrayInitialize(otherTF_Buffer,EMPTY_VALUE);
   }
   else
   {
      start = prev_calculated - offset;
   }
   
   //--- Main loop
   Print("start = ", start);
   for(int i = start; i < rates_total && !IsStopped(); i++)
   {     
      shift = iBarShift(NULL, other_TimeFrame, time[i], false);
      if(shift < 0)
      {
         Print("iBarShift error #", GetLastError());
         return i;
      }
      
      c = iClose(NULL, other_TimeFrame, shift);
      if(c < 0)
      {
         Print("iClose error #", GetLastError());
         return i;
      }
      
      // Save the close value from the other timeframe
      otherTF_Buffer[i] = c;
   }

   //--- return value of prev_calculated for next call
   Print("Loop end, returning rates_total = ", rates_total);
   return(rates_total);
}

//+------------------------------------------------------------------+

The problem with this code is that sometimes it works, sometimes it doesn't, and when it fails, I have tried to trace back the error, and I have surprisingly found that I'm returning rates_total = 36037 bars, but the very next call to OnCalculate, prev_calculated is equal to 0!!! 

How is this even possible??? It doesn't only happen with ADBE.NAS, it happens randomly with other stocks or symbols when the respective market is closed. But it doesn't happen always, other times my code works like a charm. That's why I think it's a bug. 

This is the example log where you can see the bug:

2020.05.19 11:28:37.464 OnCalculate_Fail (ADBE.NAS,M1)  On Calculate ==> Rates total = 36037; Prev_calculated = 0
2020.05.19 11:28:37.464 OnCalculate_Fail (ADBE.NAS,M1)  check_ibarshift(date = 2020.05.18 22:54:00) ==> iBarShift = -1
2020.05.19 11:28:37.464 OnCalculate_Fail (ADBE.NAS,M1)  Setting Timer
2020.05.19 11:28:38.458 OnCalculate_Fail (ADBE.NAS,M1)  Inside OnTimer, calling ChartSetSymbolPeriod to refresh the chart.
2020.05.19 11:28:38.483 OnCalculate_Fail (ADBE.NAS,M1)  On Calculate ==> Rates total = 36037; Prev_calculated = 0
2020.05.19 11:28:38.483 OnCalculate_Fail (ADBE.NAS,M1)  check_ibarshift(date = 2020.05.18 22:54:00) ==> iBarShift = 0
2020.05.19 11:28:38.483 OnCalculate_Fail (ADBE.NAS,M1)  Killing Timer
2020.05.19 11:28:38.483 OnCalculate_Fail (ADBE.NAS,M1)  start = 1
2020.05.19 11:28:38.494 OnCalculate_Fail (ADBE.NAS,M1)  Loop end, returning rates_total = 36037
2020.05.19 11:28:38.516 OnCalculate_Fail (ADBE.NAS,M1)  On Calculate ==> Rates total = 36037; Prev_calculated = 0
2020.05.19 11:28:38.516 OnCalculate_Fail (ADBE.NAS,M1)  check_ibarshift(date = 2020.05.18 22:54:00) ==> iBarShift = -1
2020.05.19 11:28:38.516 OnCalculate_Fail (ADBE.NAS,M1)  Setting Timer
2020.05.19 11:28:39.520 OnCalculate_Fail (ADBE.NAS,M1)  Inside OnTimer, calling ChartSetSymbolPeriod to refresh the chart.
2020.05.19 11:28:39.532 OnCalculate_Fail (ADBE.NAS,M1)  On Calculate ==> Rates total = 36037; Prev_calculated = 0
2020.05.19 11:28:39.532 OnCalculate_Fail (ADBE.NAS,M1)  check_ibarshift(date = 2020.05.18 22:54:00) ==> iBarShift = 0
2020.05.19 11:28:39.532 OnCalculate_Fail (ADBE.NAS,M1)  Killing Timer
2020.05.19 11:28:39.532 OnCalculate_Fail (ADBE.NAS,M1)  start = 1
2020.05.19 11:28:39.539 OnCalculate_Fail (ADBE.NAS,M1)  Loop end, returning rates_total = 36037
2020.05.19 11:28:39.572 OnCalculate_Fail (ADBE.NAS,M1)  On Calculate ==> Rates total = 36037; Prev_calculated = 0
I hope somebody can confirm whether it's a bug or at least can shed some light on the matter.

I have tried the code both on the latest MT5 Release version, and I have even installed the Beta 2431, but the problem remains. 
OnCalculate during weekend MT5
OnCalculate during weekend MT5
  • 2020.04.11
  • www.mql5.com
Hello, I always had issues when creating indicators in MT5 that access data from different symbols and timeframes rather than current...
Files:
OnCalculate_Fail.mq5  8 kb
 

If I run the very same code that I sent above in other symbols where the market is closed, it works perfectly.

This is a sample log that works flawlessly for GOOG.NAS:

2020.05.19 12:33:58.532 OnCalculate_Fail (GOOG.NAS,M1)  On Calculate ==> Rates total = 57612; Prev_calculated = 0
2020.05.19 12:33:58.532 OnCalculate_Fail (GOOG.NAS,M1)  check_ibarshift(date = 2020.05.18 22:54:00) ==> iBarShift = -1
2020.05.19 12:33:58.532 OnCalculate_Fail (GOOG.NAS,M1)  Setting Timer
2020.05.19 12:33:59.563 OnCalculate_Fail (GOOG.NAS,M1)  Inside OnTimer, calling ChartSetSymbolPeriod to refresh the chart.
2020.05.19 12:33:59.585 OnCalculate_Fail (GOOG.NAS,M1)  On Calculate ==> Rates total = 57612; Prev_calculated = 0
2020.05.19 12:33:59.585 OnCalculate_Fail (GOOG.NAS,M1)  check_ibarshift(date = 2020.05.18 22:54:00) ==> iBarShift = 0
2020.05.19 12:33:59.585 OnCalculate_Fail (GOOG.NAS,M1)  Killing Timer
2020.05.19 12:33:59.585 OnCalculate_Fail (GOOG.NAS,M1)  start = 1
2020.05.19 12:33:59.597 OnCalculate_Fail (GOOG.NAS,M1)  Loop end, returning rates_total = 57612

When the loop ends, rates_total  returns 57612, and OnCalculate is not called anymore. 

 
carlicus: and I have surprisingly found that I'm returning rates_total = 36037 bars, but the very next call to OnCalculate, prev_calculated is equal to 0!!! 

How is this even possible???

You changed the chart's symbol and period via ChartSetSymbolPeriod(0,NULL,0) so everything must be redone just as if you changed only TF.

Perhaps you should read the manual:
The call of ChartSetSymbolPeriod with the same symbol and timeframe can be used to update the chart (similar to the terminal's Refresh command). In its turn, the chart update triggers re-calculation of the indicators attached to it. Thus, you are able to calculate an indicator on the chart even if there are no ticks (e.g., on weekends).
          Chart Operations / ChartSetSymbolPeriod - Reference on algorithmic/automated trading language for MetaTrader 5
 

Good luck if you want to convince Metaquotes developers it's a bug. It's how it works, you have no control on the number of times OnCalculate() will be called.

Using the 1 second timer, and refreshing the chart after that, works like switching to a different timeframe manually and returning back.

That's not exact, using ChartSetSymbolPeriod(0,NULL,0) is not the same as changing the timeframe. It's similar to manually click on the Refresh() command in the chart popup menu, and this can lead to SEVERAL calls of OnCalculate(). This command refresh the symbol's data and "reset" ALL the charts on the considered symbol.

Beside that, you don't need a timer at all, it's useless. Just use ChartSetSymbolPeriod().

 
William Roeder:

You changed the chart's symbol and period via ChartSetSymbolPeriod(0,NULL,0) so everything must be redone just as if you changed only TF.

Perhaps you should read the manual:

Ok, the title of my post is not the most appropriate since it's clear what you explain.  And it makes sense that prev_calculated is 0 after the chart is refreshed. 

However my code is ready to deal with that. Maybe you haven't noticed the problem in my log. I will try to be more clear. The problem is with iBarShift: 

  1. The 1st time onCalculate is called, I check iBarShift in the other TF to see if it's ready.  It's not ready, so it returns -1. This means the history in the other TF is not available, and that's why I set a timer, in order to let it load. 

  2. A second after that, CharSetSymbolPeriod is called, so onCalculate is called again, and iBarshift in the other TF is checked again.
    This time iBarShift returns 0, meaning that the history in the other TF has been FULLY loaded, because iBarShift it was called on the most current date available at the moment.
    The timer is killed so no more calls to ChartSetSymbolPeriod are made. 

  3. The indicator is fully calculated. 

  4. OnCalculate is suddenly called again, according to Alain (that knows much more than me, of course), because it can cause several calls to OnCalculate. 

  5. iBarShift is called again to check if its ready and it returns an error!! How does that happen, if it previously returned no errors for the most current date? 

 
Alain Verleyen:

Good luck if you want to convince Metaquotes developers it's a bug. It's how it works, you have no control on the number of times OnCalculate() will be called.

That's not exact, using ChartSetSymbolPeriod(0,NULL,0) is not the same as changing the timeframe. It's similar to manually click on the Refresh() command in the chart popup menu, and this can lead to SEVERAL calls of OnCalculate(). This command refresh the symbol's data and "reset" ALL the charts on the considered symbol.

Beside that, you don't need a timer at all, it's useless. Just use ChartSetSymbolPeriod().

Alain, it's not useless. I have tried the very same code without a timer, as you explained it to Favio, and it didn't work. Setting up a timer avoids calling ChartSetSymbolPeriod() a thousand times.

When the market closes again, I'll show you a log without the timer so you can believe me. 

Anyway, as I replied to William, the main problem is: how is it possible that iBarShift fails after the history in the other TF was fully loaded? I mean, after I waited for one second and iBarShift returned no error for the most current date?
 

Alain, this is my code as you suggested to Favio, without timer:

#property copyright "Copyright 2020, Carlos"
#property indicator_chart_window
#property indicator_buffers 1    // buffers used
#property indicator_plots   1    // buffers displayed

//--- Indicator Properties
#property indicator_label1  "Close Value in Other TF"
#property indicator_type1   DRAW_LINE
#property indicator_color1  Blue
#property indicator_style1  STYLE_SOLID
#property indicator_width1  2

#define SECONDS_TO_WAIT 1

//--- input parameters
input ENUM_TIMEFRAMES other_TimeFrame = PERIOD_D1;   

//--- Indicator buffer
double otherTF_Buffer[];

int OnInit()
{
   Print("Entering onInit");
   //--- indicator buffers mapping
   SetIndexBuffer(0, otherTF_Buffer, INDICATOR_DATA);
   IndicatorSetInteger(INDICATOR_DIGITS,_Digits);
   PlotIndexSetDouble(0, PLOT_EMPTY_VALUE, EMPTY_VALUE);

   return(INIT_SUCCEEDED);
}


int OnCalculate(const int rates_total,
                const int prev_calculated,
                const datetime &time[],
                const double &open[],
                const double &high[],
                const double &low[],
                const double &close[],
                const long &tick_volume[],
                const long &volume[],
                const int &spread[])
{
   int    start; 
   int    shift;  
   double c;     // close value in the other timeframe
   double tf1, tf2;
   int offset;
   
   Print("On Calculate ==> Rates total = ", rates_total, "; Prev_calculated = ", prev_calculated);
   //--- Detect start position
   if(prev_calculated == 0) 
   {  
      start = 1;
      ArrayInitialize(otherTF_Buffer,EMPTY_VALUE);
      
      tf1 = PeriodSeconds(_Period);
      tf2 = PeriodSeconds(other_TimeFrame);
      offset = (int)ceil(tf2/tf1);
   }
   else
   {
      start = prev_calculated-offset;
   }
   
   //--- Main loop
   Print("start = ", start);
   for(int i = start; i < rates_total && !IsStopped(); i++)
   {     
      shift = iBarShift(NULL, other_TimeFrame, time[i], false);
      if(shift < 0)
      {
         Print("iBarShift error #", GetLastError());
         ChartSetSymbolPeriod(0,NULL,0);
         return 0;
      }
      
      c = iClose(NULL, other_TimeFrame, shift);
      if(c < 0)
      {
         Print("iClose error #", GetLastError());
         ChartSetSymbolPeriod(0,NULL,0);
         return 0;
      }
      
      // Save the close value from the other timeframe
      otherTF_Buffer[i] = c;
   }

   //--- return value of prev_calculated for next call
   Print("Loop end, returning rates_total = ", rates_total);
   return(rates_total);
}

I keep having the same problems for some symbols as with my original code with the timer. However removing the timer makes the indicator blink between being painted and not, as the problem is that iBarShift suddenly works for a moment, and the next call to OnCalculate it returns and error; next time it works again and next it doesn't, just as I explained in my previous answer. 

This doesn't make any sense, since once the history in the other TF is loaded, iBarShift should not return an error anymore. Isn't that right, or am I missing something?

2020.05.19 18:36:33.753 OnCalculate_Fail (WTI_M0,D1)    On Calculate ==> Rates total = 134; Prev_calculated = 0
2020.05.19 18:36:33.753 OnCalculate_Fail (WTI_M0,D1)    start = 1
2020.05.19 18:36:33.753 OnCalculate_Fail (WTI_M0,D1)    Loop end, returning rates_total = 134
2020.05.19 18:36:33.780 OnCalculate_Fail (WTI_M0,D1)    On Calculate ==> Rates total = 134; Prev_calculated = 0
2020.05.19 18:36:33.780 OnCalculate_Fail (WTI_M0,D1)    start = 1
2020.05.19 18:36:33.780 OnCalculate_Fail (WTI_M0,D1)    iBarShift error #4401
2020.05.19 18:36:33.834 OnCalculate_Fail (WTI_M0,D1)    On Calculate ==> Rates total = 134; Prev_calculated = 0
2020.05.19 18:36:33.834 OnCalculate_Fail (WTI_M0,D1)    start = 1
2020.05.19 18:36:33.834 OnCalculate_Fail (WTI_M0,D1)    Loop end, returning rates_total = 134
2020.05.19 18:36:33.860 OnCalculate_Fail (WTI_M0,D1)    On Calculate ==> Rates total = 134; Prev_calculated = 0
2020.05.19 18:36:33.860 OnCalculate_Fail (WTI_M0,D1)    start = 1
2020.05.19 18:36:33.860 OnCalculate_Fail (WTI_M0,D1)    iBarShift error #4401
2020.05.19 18:36:33.918 OnCalculate_Fail (WTI_M0,D1)    On Calculate ==> Rates total = 134; Prev_calculated = 0
2020.05.19 18:36:33.918 OnCalculate_Fail (WTI_M0,D1)    start = 1
2020.05.19 18:36:33.918 OnCalculate_Fail (WTI_M0,D1)    Loop end, returning rates_total = 134
2020.05.19 18:36:33.947 OnCalculate_Fail (WTI_M0,D1)    On Calculate ==> Rates total = 134; Prev_calculated = 0
2020.05.19 18:36:33.947 OnCalculate_Fail (WTI_M0,D1)    start = 1
2020.05.19 18:36:33.947 OnCalculate_Fail (WTI_M0,D1)    iBarShift error #4401
2020.05.19 18:36:34.002 OnCalculate_Fail (WTI_M0,D1)    On Calculate ==> Rates total = 134; Prev_calculated = 0
2020.05.19 18:36:34.002 OnCalculate_Fail (WTI_M0,D1)    start = 1
2020.05.19 18:36:34.002 OnCalculate_Fail (WTI_M0,D1)    Loop end, returning rates_total = 134
2020.05.19 18:36:34.031 OnCalculate_Fail (WTI_M0,D1)    On Calculate ==> Rates total = 134; Prev_calculated = 0
2020.05.19 18:36:34.031 OnCalculate_Fail (WTI_M0,D1)    start = 1
2020.05.19 18:36:34.031 OnCalculate_Fail (WTI_M0,D1)    iBarShift error #4401

As you can see it goes forever like that. 

 
carlicus:

Alain, this is my code as you suggested to Favio, without timer:

I keep having the same problems for some symbols as with my original code with the timer. However removing the timer makes the indicator blink between being painted and not, as the problem is that iBarShift suddenly works for a moment, and the next call to OnCalculate it returns and error; next time it works again and next it doesn't, just as I explained in my previous answer. 

This doesn't make any sense, since once the history in the other TF is loaded, iBarShift should not return an error anymore. Isn't that right, or am I missing something?

As you can see it goes forever like that. 

As you can see the ChartSetSymbolPeriod() send 2 "reset" commands, why ? No idea.

As you can see all is going well on the first, and you get the iBarShift error on the second, why ? No idea.

A bug or a feature, we don't know but I am sure, by experience, it will be hard to convince the developers it's a weird behaviour.

Here is a solution, without a timer.

bool AllDone=false;

int OnInit()
 {
  AllDone=false;
  Print("Entering onInit");
//--- indicator buffers mapping
  SetIndexBuffer(0, otherTF_Buffer, INDICATOR_DATA);
  IndicatorSetInteger(INDICATOR_DIGITS,_Digits);
  PlotIndexSetDouble(0, PLOT_EMPTY_VALUE, EMPTY_VALUE);

  return(INIT_SUCCEEDED);
 }

int OnCalculate(const int rates_total,
                const int prev_calculated,
                const datetime &time[],
                const double &open[],
                const double &high[],
                const double &low[],
                const double &close[],
                const long &tick_volume[],
                const long &volume[],
                const int &spread[])
 {
  int    start;
  int    shift;
  double c;     // close value in the other timeframe
  double tf1, tf2;
  int offset=0;

  Print("On Calculate ==> Rates total = ", rates_total, "; Prev_calculated = ", prev_calculated);
//--- Detect start position
  if(prev_calculated == 0)
   {
    start = 1;
    ArrayInitialize(otherTF_Buffer,EMPTY_VALUE);

    tf1 = PeriodSeconds(_Period);
    tf2 = PeriodSeconds(other_TimeFrame);
    offset = (int)ceil(tf2/tf1);
   }
  else
   {
    start = prev_calculated-offset;
   }

//--- Main loop
  Print("start = ", start);
  for(int i = start; i < rates_total && !IsStopped(); i++)
   {
    shift = iBarShift(NULL, other_TimeFrame, time[i], false);
    if(shift < 0)
     {
      Print("iBarShift error #", GetLastError(), " ",time[i]);
      if(!AllDone) ChartSetSymbolPeriod(0,NULL,0);
      return(0);
     }

    c = iClose(NULL, other_TimeFrame, shift);
    if(c < 0)
     {
      Print("iClose error #", GetLastError());
      if(!AllDone) ChartSetSymbolPeriod(0,NULL,0);
      return(0);
     }

    // Save the close value from the other timeframe
    otherTF_Buffer[i] = c;
   }

  AllDone=true;
//--- return value of prev_calculated for next call
  Print("Loop end, returning rates_total = ", rates_total);
  return(rates_total);
 }

Please note that most of the time I provide a clue towards the solution, it's not my responsibility to provide a solution that takes into account all possibilities.

 
Alain Verleyen:

As you can see the ChartSetSymbolPeriod() send 2 "reset" commands, why ? No idea.

As you can see all is going well on the first, and you get the iBarShift error on the second, why ? No idea.

A bug or a feature, we don't know but I am sure, by experience, it will be hard to convince the developers it's a weird behaviour.

Here is a solution, without a timer.

Please note that most of the time I provide a clue towards the solution, it's not my responsibility to provide a solution that takes into account all possibilities.

Thank you for your time and help Alain.

Everything is a bit strange, since the problem I described yesterday has not happened to me anymore. I have been testing dozens of american stocks with markets closed  and my original code didn't fail anymore. Anyway thanks again for your idea, it helped me too. 

 
carlicus:

Thank you for your time and help Alain.

Everything is a bit strange, since the problem I described yesterday has not happened to me anymore. I have been testing dozens of american stocks with markets closed  and my original code didn't fail anymore. Anyway thanks again for your idea, it helped me too. 

With the last code you posted (no timer), it's easy to reproduce.

I asked to a Metaquotes developer his opinion, we will see if he answers me.

 
Alain Verleyen:

With the last code you posted (no timer), it's easy to reproduce.

I asked to a Metaquotes developer his opinion, we will see if he answers me.

Thank you again, Alain, very nice of you. I hope he answers you, since the problem it's quite random. Yesterday it didn't happen to me with any symbol, but today it's happening again.
However, given your extensive experience, I am aware that the developers will most likely not take this into account :)

New comment