Getting date of a bar from history (not from indicator data that is not-yet loaded) Alternative to iTime() (which requires data loaded)

 

----- will CopyTime() do this?  Looks like it also requires data loaded in the indicator.

----- so I'm looking for the equivalent of this but for a specific bar and return a date instead of a bar number.

SeriesInfoInteger(_Symbol,_Period,SERIES_FIRSTDATE,first_bar)

// or

iBarShift(_Symbol,TFtoDisplay,earliestRequested,false);

Hello,

In an indicator, I'm having great luck with some code I found for querying the state of history data when it is not yet loaded in the indicator.

My problem is, if I could find one more datapoint from history instead of waiting until it is loaded in the indicator data, I could call the entire data-prep from the OnInit() rather than waiting until OnCalculate().

I have been unable to find a solution to replace this one line of code.....

int getRequiredHigherTFDate(ENUM_TIMEFRAMES higherTF, datetime earliestRequested=0)
   {   
   //////// find out the date of the earliest possible bar on the current period chart without data actually being loaded in the indicator yet.
   long first_bar;
   bool result;
   double maxBarsOnChart;
   
   int earliestBarOnCurrPeriod;
   datetime earliestBarDateNeeded;

   // get the earliest bar date in history, but that could be way, way earlier than the maximum bars on a chart (rates_total)
   result=SeriesInfoInteger(_Symbol,_Period,SERIES_FIRSTDATE,first_bar);
   
   // get the maximum allowed bars on a chart (but realize, as bars accumulate, the current chart could have more than that).
   maxBarsOnChart=TerminalInfoInteger(TERMINAL_MAXBARS);
   
   //Print("The earliest available date for ",_Symbol,"-",ChartPeriod()," is ",first_bar," with Terminal Max Bars of ",maxBarsOnChart);
   
   // if the chart has more bars on it than the terminal setting maximum, get that instead
   maxBarsOnChart=MathMax(maxBarsOnChart,iBars(_Symbol,_Period));

   // if the history has fewer bars than the terminal maximum get that bar instead
   earliestBarOnCurrPeriod=MathMin(first_bar,maxBarsOnChart)-1;

// -> THIS IS THE LINE OF CODE I WANT TO REPLACE because it requires data to be loaded in the indicator.
// I need the open time of the bar identified above.  Everything else does not require data loaded in the indicator
// Need something other than iTime()
   earliestBarDateNeeded=iTime(_Symbol,_Period,earliestBarOnCurrPeriod);
//   
   
   if(earliestBarDateNeeded<earliestRequested)
      {
      int earliestBarNowNeeded=iBarShift(_Symbol,TFtoDisplay,earliestRequested,false);

// likewise, need same alternative to iTime
      earliestBarDateNeeded=iTime(_Symbol,_Period,earliestBarNowNeeded);
      }
   
   //Print("Max is set to Bar ",earliestBarOnCurrPeriod," - ",earliestBarDateNeeded);
   
   // Note that iBars tested on an M1 timeframe returned 61 bars more than the above logic, i.e. returned a number greater than the max bars allowed on chart, and Bars() returned the same as iBars().
   //Print("...and iBars reports: ",iBars(_Symbol,_Period)," while Bars reports: ",Bars(_Symbol,_Period));
   ////////
   
   return(earliestBarDateNeeded);
   
  }

After acquiring this date, I can call my history loading function without requesting more than I need.

If anyone can help direct me it would be greatly appreciated.

Also of note:  I've noticed that even though the documentation says that iBars is limited to the "max bars in chart", as a chart accumulates additional bars when live, it can actually have iBars return more periods than the terminal maximum bars setting.


Thanks in advance,

Documentation on MQL5: Constants, Enumerations and Structures / Environment State / Client Terminal Properties
Documentation on MQL5: Constants, Enumerations and Structures / Environment State / Client Terminal Properties
  • www.mql5.com
Client Terminal Properties - Environment State - Constants, Enumerations and Structures - MQL5 Reference - Reference on algorithmic/automated trading language for MetaTrader 5
 
Christian Berrigan:

----- will CopyTime() do this?  Looks like it also requires data loaded in the indicator.

----- so I'm looking for the equivalent of this but for a specific bar and return a date instead of a bar number.

In an indicator, I'm having great luck with some code I found for querying the state of history data when it is not yet loaded in the indicator. My problem is, if I could find one more datapoint from history instead of waiting until it is loaded in the indicator data, I could call the entire data-prep from the OnInit() rather than waiting until OnCalculate(). I have been unable to find a solution to replace this one line of code..... After acquiring this date, I can call my history loading function without requesting more than I need. If anyone can help direct me it would be greatly appreciated.

  1. No! You will always need to have the historical data downloaded before you can access any individual information about the bars, and that includes their open times.
  2. OnInit() is for handling the initialisation process, so don't block it or use it for something that it is not meant to do.
  3. Use the OnCalculate() instead. If you don't need plots or buffers, then don't declare them, but do the processing in the OnCalculate event handler, even if you just need to do it once.

    Forum on trading, automated trading systems and testing trading strategies

    Download history in MQL4 EA

    Fernando Carreiro, 2022.05.23 23:14

    Maybe something like this. You will need to adjust it based on your exact requirements. Code is untested and only an example.

    // Define indicator properties
       #property indicator_chart_window
       #property indicator_buffers 0
       #property indicator_plots   0
       
    // Initialisation event handler
       int OnInit()
       {
          // Do something ...
          return INIT_SUCCEEDED;  // Successful initialisation of indicator
       };
    
    // OnCalculate event handler
       int OnCalculate( const int rates_total,
                        const int prev_calculated,
                        const datetime &time[],
                        const double &open[],
                        const double &high[],
                        const double &low[],
                        const double &close[],
                        const long &tick_volume[],
                        const long &volume[],
                        const int &spread[] )
       {
          static bool bOnce = true;
          if( bOnce && rates_total > 0 ) // Check for flag and available data
          {
             // Do some operation just once
             bOnce = false;
          };
    
          return rates_total;  // Return value for prev_calculated of next call
       };
 
Christian Berrigan:

----- will CopyTime() do this?  Looks like it also requires data loaded in the indicator.

----- so I'm looking for the equivalent of this but for a specific bar and return a date instead of a bar number.

Hello,

In an indicator, I'm having great luck with some code I found for querying the state of history data when it is not yet loaded in the indicator.

My problem is, if I could find one more datapoint from history instead of waiting until it is loaded in the indicator data, I could call the entire data-prep from the OnInit() rather than waiting until OnCalculate().

I have been unable to find a solution to replace this one line of code.....

After acquiring this date, I can call my history loading function without requesting more than I need.

If anyone can help direct me it would be greatly appreciated.

Also of note:  I've noticed that even though the documentation says that iBars is limited to the "max bars in chart", as a chart accumulates additional bars when live, it can actually have iBars return more periods than the terminal maximum bars setting.


Thanks in advance,

Thank you for the reference.  I have looked through all the TerminalInfo**** stuff and the SeriesInfoInteger() (which seems more appropriate for my problem), but I have found nothing to address the issue of getting a date from a specific bar in HISTORY (not from the data in the indicator).  With these above functions I can get info about the first bar in history, the last bar in history, and the number of bars in history, and the bar number that matches a date (iBarShift), but I cannot get a date for a specific bar. 

CopyRates(), CopyTime() MqlRates structure etc, all require data loaded into the indicator and are not getting the data from history.


I suppose I can try to scan through iBarShift() one date at a time until I get a hit for the bar number I'm looking for, but that would be pretty laborious if a function already existed that would return the datetime from a known bar number.


The problem is SeriesInfoInteger(_Symbol,_Period,SERIES_FIRSTDATE,first_bar); will return 1.6million records when I know I never need more than max bars on chart (10,000) in my case.  If the function doesn't exist I can iterate through some portion of 10,000 dates until I hit the right bar.  Seems ridiculous though....  I suppose I can cut the queried bars in half each time (the 1/2 before or after each iBarShift(), then in 1/2 again etc until I get a hit until I get within a reasonable number of bars and then for() iterate from there.  Uggghhh.

Any other suggestions?

 
Christian Berrigan #: Thank you for the reference.  I have looked through all the TerminalInfo**** stuff and the SeriesInfoInteger() (which seems more appropriate for my problem), but I have found nothing to address the issue of getting a date from a specific bar in HISTORY (not from the data in the indicator).  With these above functions I can get info about the first bar in history, the last bar in history, and the number of bars in history, and the bar number that matches a date (iBarShift), but I cannot get a date for a specific bar. CopyRates(), CopyTime() MqlRates structure etc, all require data loaded into the indicator and are not getting the data from history. I suppose I can try to scan through iBarShift() one date at a time until I get a hit for the bar number I'm looking for, but that would be pretty laborious if a function already existed that would return the datetime from a known bar number. The problem is SeriesInfoInteger(_Symbol,_Period,SERIES_FIRSTDATE,first_bar); will return 1.6million records when I know I never need more than max bars on chart (10,000) in my case.  If the function doesn't exist I can iterate through some portion of 10,000 dates until I hit the right bar.  Seems ridiculous though....  I suppose I can cut the queried bars in half each time (the 1/2 before or after each iBarShift(), then in 1/2 again etc until I get a hit until I get within a reasonable number of bars and then for() iterate from there.  Uggghhh.

Any other suggestions?

Something is wrong with your explanation and you don't seem to make much sense in some of your declarations. You seem to be overcomplicating things.

If you know the date or time range you want, then simply use CopyRates() with a start and stop datetime parameters.

If you want the time for a specific bar index, then just use iTime().

If the terminal needs to download data for that, then let it. What is the complication?

 
Fernando Carreiro #:

Something is wrong with your explanation and you don't seem to make much sense in some of your declarations. You seem to be overcomplicating things.

If you know the date or time range you want, then simply use CopyRates() with a start and stop datetime parameters.

If you want the time for a specific bar index, then just use iTime().

If the terminal needs to download data for that, then let it. What is the complication?

Thank you for your reply, you were definitely right that I was overcomplicating things, however, CopyRates() does not work because it relies on data already being loaded into the indicator, and my goal was to get history loaded via a call in the indicator's OnInit() rather than waiting for OnCalculate() and the timing of rates maybe being loaded or maybe not.

However, I did figure it out based on the premise of overcomplicating things, furthermore, bar numbers are not reliable in this case because if bars are missing from history, the whole thing breaks (based on me calculating the bar number I wanted a date for).

So the solution, since ultimately all I needed was the date of a bar, was to simply calculate the date I needed using PeriodSeconds(TFtoDisplay).  It works freakin' awesome and allows me to call the history data loading program reliably from the OnInit() and not worry about the other timeframe data being available when I get to the OnCalculate().

Here is the solution which I will always use in any indicator requiring data from a different timeframe.

//+------------------------------------------------------------------+
//| get the earliest date of data needed from a higher timeframe     |
//| based on the earliest date on the current timeframe chart,       |
//| or the earliest date in the terminal                             |
//| or for a specifically requested date                             |
//+------------------------------------------------------------------+
int getRequiredHigherTFDate(ENUM_TIMEFRAMES higherTF, datetime earliestRequested=0)
   {   
   //////// find out the date of the earliest possible bar on the current period chart
   bool result;
   
   datetime earliestBarDateNeeded;
   
   // the earliest date in series, (the highest bar number)
   long firstDate_long;
   result=SeriesInfoInteger(_Symbol,_Period,SERIES_FIRSTDATE,firstDate_long);
   
   // the most recent date in series (bar zero)
   long lastDate_long;
   SeriesInfoInteger(_Symbol,higherTF,SERIES_LASTBAR_DATE,lastDate_long);

   double maxBarOnChart;
   maxBarOnChart=TerminalInfoInteger(TERMINAL_MAXBARS)-1;
   
   // the number of bars on the chart, possible more than the TERMINAL_MAXBARS because of subsequent bars posted on open chart
   int chartBars=iBars(_Symbol,higherTF);
   
   datetime lastDate=lastDate_long;
   datetime firstDate=firstDate_long;

   //Print("The earliest available date for ",_Symbol,"-",ChartPeriod()," is ",firstDate_long," with Terminal Max Bars of ",maxBarsOnChart);
   // most recent bar is bar zero.
   
   #define EARLIEST_MAXBAR 0
   #define EARLIEST_ONCHART 1
   #define EARLIEST_REQUESTED 2
   datetime earliestDates[3];
   
   // calculate the dates without using bar numbers
   earliestDates[EARLIEST_MAXBAR]=lastDate-(maxBarOnChart*PeriodSeconds(higherTF));
   earliestDates[EARLIEST_ONCHART]=lastDate-(chartBars*PeriodSeconds(higherTF));
   earliestDates[EARLIEST_REQUESTED]=earliestRequested;
   
   earliestBarDateNeeded=earliestDates[ArrayMaximum(earliestDates,0,WHOLE_ARRAY)];
   
   return(earliestBarDateNeeded);
   
  }
//+------------------------------------------------------------------+
 

Then after having acquired the earliest date in history that I need, I use the code I got from a lifesaver at this most wonderful article on the topic of getting history from other timeframes:

https://www.mql5.com/en/docs/series/timeseries_access

Hope this helps someone as much as it did me!

//+------------------------------------------------------------------+
//|                                              TestLoadHistory.mq5 |
//|                        Copyright 2009, MetaQuotes Software Corp. |
//|                                              https://www.mql5.com |
//+------------------------------------------------------------------+
#property copyright "2009, MetaQuotes Software Corp."
#property link      "https://www.mql5.com"
#property version   "1.02"
#property script_show_inputs
//--- input parameters
input string          InpLoadedSymbol="NZDUSD";   // Symbol to be load
input ENUM_TIMEFRAMES InpLoadedPeriod=PERIOD_H1;  // Period to be loaded
input datetime        InpStartDate=D'2006.01.01'; // Start date
//+------------------------------------------------------------------+
//| Script program start function                                    |
//+------------------------------------------------------------------+
void OnStart()
  {
   Print("Start load",InpLoadedSymbol+","+GetPeriodName(InpLoadedPeriod),"from",InpStartDate);
//---
   int res=CheckLoadHistory(InpLoadedSymbol,InpLoadedPeriod,InpStartDate);
   switch(res)
     {
      case -1 : Print("Unknown symbol ",InpLoadedSymbol);             break;
      case -2 : Print("Requested bars more than max bars in chart"); break;
      case -3 : Print("Program was stopped");                        break;
      case -4 : Print("Indicator shouldn't load its own data");      break;
      case -5 : Print("Load failed");                                break;
      case  0 : Print("Loaded OK");                                  break;
      case  1 : Print("Loaded previously");                          break;
      case  2 : Print("Loaded previously and built");                break;
      default : Print("Unknown result");
     }
//---
   datetime first_date;
   SeriesInfoInteger(InpLoadedSymbol,InpLoadedPeriod,SERIES_FIRSTDATE,first_date);
   int bars=Bars(InpLoadedSymbol,InpLoadedPeriod);
   Print("First date ",first_date," - ",bars," bars");
//---
  }
//+------------------------------------------------------------------+
//|                                                                  |
//+------------------------------------------------------------------+
int CheckLoadHistory(string symbol,ENUM_TIMEFRAMES period,datetime start_date)
  {
   datetime first_date=0;
   datetime times[100];
//--- check symbol & period
   if(symbol==NULL || symbol=="") symbol=Symbol();
   if(period==PERIOD_CURRENT)     period=Period();
//--- check if symbol is selected in the Market Watch
   if(!SymbolInfoInteger(symbol,SYMBOL_SELECT))
     {
      if(GetLastError()==ERR_MARKET_UNKNOWN_SYMBOL) return(-1);
      SymbolSelect(symbol,true);
     }
//--- check if data is present
   SeriesInfoInteger(symbol,period,SERIES_FIRSTDATE,first_date);
   if(first_date>0 && first_date<=start_date) return(1);
//--- don't ask for load of its own data if it is an indicator
   if(MQL5InfoInteger(MQL5_PROGRAM_TYPE)==PROGRAM_INDICATOR && Period()==period && Symbol()==symbol)
      return(-4);
//--- second attempt
   if(SeriesInfoInteger(symbol,PERIOD_M1,SERIES_TERMINAL_FIRSTDATE,first_date))
     {
      //--- there is loaded data to build timeseries
      if(first_date>0)
        {
         //--- force timeseries build
         CopyTime(symbol,period,first_date+PeriodSeconds(period),1,times);
         //--- check date
         if(SeriesInfoInteger(symbol,period,SERIES_FIRSTDATE,first_date))
            if(first_date>0 && first_date<=start_date) return(2);
        }
     }
//--- max bars in chart from terminal options
   int max_bars=TerminalInfoInteger(TERMINAL_MAXBARS);
//--- load symbol history info
   datetime first_server_date=0;
   while(!SeriesInfoInteger(symbol,PERIOD_M1,SERIES_SERVER_FIRSTDATE,first_server_date) && !IsStopped())
      Sleep(5);
//--- fix start date for loading
   if(first_server_date>start_date) start_date=first_server_date;
   if(first_date>0 && first_date<first_server_date)
      Print("Warning: first server date ",first_server_date," for ",symbol,
            " does not match to first series date ",first_date);
//--- load data step by step
   int fail_cnt=0;
   while(!IsStopped())
     {
      //--- wait for timeseries build
      while(!SeriesInfoInteger(symbol,period,SERIES_SYNCHRONIZED) && !IsStopped())
         Sleep(5);
      //--- ask for built bars
      int bars=Bars(symbol,period);
      if(bars>0)
        {
         if(bars>=max_bars) return(-2);
         //--- ask for first date
         if(SeriesInfoInteger(symbol,period,SERIES_FIRSTDATE,first_date))
            if(first_date>0 && first_date<=start_date) return(0);
        }
      //--- copying of next part forces data loading
      int copied=CopyTime(symbol,period,bars,100,times);
      if(copied>0)
        {
         //--- check for data
         if(times[0]<=start_date)  return(0);
         if(bars+copied>=max_bars) return(-2);
         fail_cnt=0;
        }
      else
        {
         //--- no more than 100 failed attempts
         fail_cnt++;
         if(fail_cnt>=100) return(-5);
         Sleep(10);
        }
     }
//--- stopped
   return(-3);
  }
//+------------------------------------------------------------------+
//| Returns string value of the period                               |
//+------------------------------------------------------------------+
string GetPeriodName(ENUM_TIMEFRAMES period)
  {
   if(period==PERIOD_CURRENT) period=Period();
//---
   switch(period)
     {
      case PERIOD_M1:  return("M1");
      case PERIOD_M2:  return("M2");
      case PERIOD_M3:  return("M3");
      case PERIOD_M4:  return("M4");
      case PERIOD_M5:  return("M5");
      case PERIOD_M6:  return("M6");
      case PERIOD_M10: return("M10");
      case PERIOD_M12: return("M12");
      case PERIOD_M15: return("M15");
      case PERIOD_M20: return("M20");
      case PERIOD_M30: return("M30");
      case PERIOD_H1:  return("H1");
      case PERIOD_H2:  return("H2");
      case PERIOD_H3:  return("H3");
      case PERIOD_H4:  return("H4");
      case PERIOD_H6:  return("H6");
      case PERIOD_H8:  return("H8");
      case PERIOD_H12: return("H12");
      case PERIOD_D1:  return("Daily");
      case PERIOD_W1:  return("Weekly");
      case PERIOD_MN1: return("Monthly");
     }
//---
   return("unknown period");
  }
 
Fernando Carreiro #:
  1. No! You will always need to have the historical data downloaded before you can access any individual information about the bars, and that includes their open times.
  2. OnInit() is for handling the initialisation process, so don't block it or use it for something that it is not meant to do.
  3. Use the OnCalculate() instead. If you don't need plots or buffers, then don't declare them, but do the processing in the OnCalculate event handler, even if you just need to do it once.

Thanks so much for your time!  I really appreciate it.  See my final solution I posted.  Thanks again!!!

 
Christian Berrigan #:

Thank you for your reply, you were definitely right that I was overcomplicating things, however, CopyRates() does not work because it relies on data already being loaded into the indicator, and my goal was to get history loaded via a call in the indicator's OnInit() rather than waiting for OnCalculate() and the timing of rates maybe being loaded or maybe not.

However, I did figure it out based on the premise of overcomplicating things, furthermore, bar numbers are not reliable in this case because if bars are missing from history, the whole thing breaks (based on me calculating the bar number I wanted a date for).

So the solution, since ultimately all I needed was the date of a bar, was to simply calculate the date I needed using PeriodSeconds(TFtoDisplay).  It works freakin' awesome and allows me to call the history data loading program reliably from the OnInit() and not worry about the other timeframe data being available when I get to the OnCalculate().

Here is the solution which I will always use in any indicator requiring data from a different timeframe.

Then after having acquired the earliest date in history that I need, I use the code I got from a lifesaver at this most wonderful article on the topic of getting history from other timeframes:

https://www.mql5.com/en/docs/series/timeseries_access

Hope this helps someone as much as it did me!

What are you talking about?

CopyRates copies from rates history data, not from the Indicator. The indicator has it's own rates buffers from the OnCalculate parameters. CopyRates has nothing to do with that, whether there is an indicator or not. CopyRates works on Scripts, EAs, Services or Indicators irrespectively. Don't confuse it with CopyBuffer which is something completely different.

Again, you are over complicating things by basing it on assumptions which are incorrect.

CopyRates

Gets history data of MqlRates structure of a specified symbol-period in specified quantity into the rates_array array. The elements ordering of the copied data is from present to the past, i.e., starting position of 0 means the current bar.

 
Fernando Carreiro #:

What are you talking about?

CopyRates copies from rates history data, not from the Indicator. The indicator has it's own rates buffers from the OnCalculate parameters. CopyRates has nothing to do with that, whether there is an indicator or not. CopyRates works on Scripts, EAs, Services or Indicators irrespectively. Don't confuse it with CopyBuffer which is something completely different.

Again, you are over complicating things by basing it on assumptions which are incorrect.

Well in my testing CopyRates() failed when called from OnInit() in an indicator, and only succeeded after the OnCalculate() had been called at least once.

I read the documentation for CopyRates() and under the "Notes" section it says:

CopyRates

"When requesting data from the indicator, if requested timeseries are not yet built or they need to be downloaded from the server, the function will immediately return -1, but the process of downloading/building will be initiated."

That was a problem as it was always returning -1 from the OnInit() of an indicator.

Really appreciate your time.  In this case, I specifically tested in the debugger ad-nauseum and it never worked for me.

 
Christian Berrigan #:

Well in my testing CopyRates() failed when called from OnInit() in an indicator, and only succeeded after the OnCalculate() had been called at least once.

I read the documentation for CopyRates() and under the "Notes" section it says:

CopyRates

"When requesting data from the indicator, if requested timeseries are not yet built or they need to be downloaded from the server, the function will immediately return -1, but the process of downloading/building will be initiated."

It has already been said, that you should not be calling CopyRates from the OnInit(). In this case it may fail, not because of the indicator, but because history data may still be loading, the same data that will then be used for the Indicator call in OnCalculate.

The very sentence you have underlined is explaining that very fact. The timeseries is the historical data. You have just assumed that sentence is referring to the indicator, but it's not.

Christian Berrigan #: That was a problem as it was always returning -1 from the OnInit() of an indicator. Really appreciate your time. In this case, I specifically tested in the debugger ad-nauseum and it never worked for me.
Because you are making the wrong assumptions, and not really addressing the true issue.
 

Also, read the section in the documentation dedicated to this issue and how to synchronise data access: Documentation on MQL5: Timeseries and Indicators Access / Organizing Data Access

Documentation on MQL5: Timeseries and Indicators Access / Organizing Data Access
Documentation on MQL5: Timeseries and Indicators Access / Organizing Data Access
  • www.mql5.com
Organizing Data Access - Timeseries and Indicators Access - MQL5 Reference - Reference on algorithmic/automated trading language for MetaTrader 5
 
Fernando Carreiro #:

It has already been said, that you should not be calling CopyRates from the OnInit(). In this case it may fail, not because of the indicator, but because history data may still be loading, the same data that will then be used for the Indicator call in OnCalculate.

The very sentence you have underlined is explaining that very fact. The timeseries is the historical data. You have just assumed that sentence is referring to the indicator, but it's not.

Yes I was aware that you already said that, and the premise of my post was trying to get other timeframe history from OnInit() instead of OnCalculate(), which was accomplished.  It works really well.  I was so grateful to find the code in in the article I linked to above.  That guy's code worked phenomenally well, and now on any indicator where I need data from other timeframes I can do it reliably and without headaches.  He also gave ample warning in the article about not attempting to get data for the same timeframe as the indicator in this way, and his code checks for that circumstance as well.  Calculating the needed minimal date instead of calculating bars and trying to find the date from a bar solved the whole problem.

Reason: