MQL5: How do I initialize data for CopyRates, Bars and iBars?

 

I'm Looking for help on how to initialize terminal data before OnCalculate is called.

Attached is a sample indicator and associated log file showing the errors posted - I thought that after calling CopyRates for an 'old' bar, eventually the data would download and things would start working - but after days I am just getting error 4401, Requested history not found :

From OnInit:

SetBarCounts The function "Bars" returned 0 for EURUSD and Timeframe PERIOD_M1. Last Error is: 4401, Requested history not found
SetiBarCounts The function "iBars" returned 0 for EURUSD and Timeframe PERIOD_M1. Last Error is: 4401, Requested history not found
SetRatesForBar Caller: OnInit CopyRates Call index 0 for bar: 0 of Time Frame: PERIOD_M1 posted error: 4401, Requested history not found

  • Bars and IBars do work for M30 (the chart time frame), but I'd like to get values for all timeframes (but CopyRates for old bars logged the 4401 error on M30):

SetBarCounts Called by OnInit "Bars" succeeded, Bar count for EURUSD in time frame PERIOD_M30 is 100240
SetiBarCounts Called by OnInit "iBars" succeeded, Bar count for EURUSD in time frame PERIOD_M30 is 100240
SetRatesForBar Caller: OnInit CopyRates Call index 10 for bar: 100239 of Time Frame: PERIOD_M30 posted error: 4401, Requested history not found

SetBarCounts The function "Bars" returned 0 for EURUSD and Timeframe PERIOD_MN1. Last Error is: 4401, Requested history not found
SetiBarCounts The function "iBars" returned 0 for EURUSD and Timeframe PERIOD_MN1. Last Error is: 4401, Requested history not found
SetRatesForBar Caller: OnInit CopyRates Call index 20 for bar: 0 of Time Frame: PERIOD_MN1 posted error: 4401, Requested history not found

Followed by OnCalculate:

SetRatesForBar Caller: OnCalculate CopyRates Call index 0 for bar: 0 of Time Frame: PERIOD_M1 posted error: 4401, Requested history not found
SetRatesForBar Caller: OnCalculate CopyRates Call index 10 for bar: 100239 of Time Frame: PERIOD_M30 posted error: 4401, Requested history not found
SetRatesForBar Caller: OnCalculate CopyRates Call index 20 for bar: 0 of Time Frame: PERIOD_MN1 posted error: 4401, Requested history not found
Terminal Caluculations were NOT complete for OnCalculate, call count: 1

How do I get the data initialized / get the history / during OnInit so  CopyRates (and Bars or iBars) will return valid values once OnCalcualate starts?

 (updated with cleaned up indicator and log file 8/03/2019}

 

Still no solution.  Here's another version that can produce a good initialization for a reduced number of Bars, but comment out line 114 (// AdjustOldestBar(tfBarsCount_ary, rates_total-1);) and all the lower timeframes (with lots of bars) still fail after 3,875 (and continuing) events have been received by OnCalculate.

Terminal Caluculations were NOT complete for OnCalculate, at the start of call count: 3875
SetTerminalData CopyBuffer for caller OnCalculate on caller count 3875, for AUDUSD in time frame PERIOD_M1 failed with error 4806, Requested data not found
Thru
SetTerminalData CopyBuffer for caller OnCalculate on caller count 3875, for AUDUSD in time frame PERIOD_M30 failed with error 4806, Requested data not found
- the higher timeframes have gotten data.

The bar counts seem to start working on the 2nd OnCalculate call

TestMQLRatesAndBars_2 (AUDUSD,M30)    SetBarCounts Called by OnCalculate "Bars" succeeded on caller count 1. Bar count for AUDUSD in time frame PERIOD_M2 is 100000
thrue
TestMQLRatesAndBars_2 (AUDUSD,M30)    SetBarCounts Called by OnCalculate "Bars" succeeded on caller count 1. Bar count for AUDUSD in time frame PERIOD_MN1 is 198
TestMQLRatesAndBars_2 (AUDUSD,M30)    Terminal Bar counts READY for OnCalculate, call count: 2

This version use calls to Custom Events in the OnCalculate handler to trigger data initialization - I used the events handler as I don't think it a good idea to delay OnCalculate while doing things to trigger data downloads.

- I'd like to be able to directly ask the terminal how many bars have data for each time frame, but haven't identified a way to do that.

Is it possible that Bars and iBars do not trigger data downloads, but getting a handle and asking the terminal for data (iATR) does?  Seems like that would be a bug... To that end, this version includes getting a handle for iATR and asking the terminal for the data.  However, creating an indicator buffer and populating data in it is overhead not needed in many indicators.

Any experts offering ideas?  Anyone with better ways to start up that will work better....


Files:
 

It's a bit hard to understand what you are trying to do.

When you request data on a symbol/timeframe different from current chart, it's ALWAYS possible to have a data error, so you can't "initialize data". You just need to wait the next tick and try again. If you don't want to wait the next tick the trouble start.

 

Ah, a clearer statement of the Goal.

  • Get the number of bars available for analysis in each timeframe for any symbol.
  • Be able to get the 'rates' information for any (all) of the bars in each timeframe.
  • If no guarantee of being able to get rates information for all bars, get the oldest bar with available rates information (assuming its always complete, newest to oldest).

I've been having some trouble converting some MQ4 stuff to MQ5 because of the inability to get that info.

Please See:

https://www.mql5.com/en/code/26316, Bars and Rates Information Utility

At this point, I'm left guessing that Bar 999,999 will always be the oldest bar that CopyRates will work for.  But, CopyRates is not always available for that bar immediately, it can take some time before the terminal is prepared to deliver the data after its first request.

  • Is there a direct way to ask the terminal what the oldest bar CoypRates will work for is, for a given Symbol and timeframe?

If I could get an affirmative on that question (with the method), all would be well.

I believe iHigh, iLow, IOpen, IClose all have the same data availablity as CopyRates.

Bars and Rates Information Utility, displays how many bars there are in all timeframes for any user selected symbol, and shows the 'CopyRates' information availablity for any bar
Bars and Rates Information Utility, displays how many bars there are in all timeframes for any user selected symbol, and shows the 'CopyRates' information availablity for any bar
  • www.mql5.com
This utility is not a trading information utility.  It will only have value for those getting bar information using the Bars (or iBars, as they are near equivalent) and the CopyRates functions over multiple timeframes.  Testing was mostly done using the "Basket Viewer" utility and clicking on each of the symbols in it's list to switch the chart...
 
LukeB:

Ah, a clearer statement of the Goal.

  • Get the number of bars available for analysis in each timeframe for any symbol.
  • Be able to get the 'rates' information for any (all) of the bars in each timeframe.
  • If no guarantee of being able to get rates information for all bars, get the oldest bar with available rates information (assuming its always complete, newest to oldest).

I've been having some trouble converting some MQ4 stuff to MQ5 because of the inability to get that info.

Please See:

https://www.mql5.com/en/code/26316, Bars and Rates Information Utility

At this point, I'm left guessing that Bar 999,999 will always be the oldest bar that CopyRates will work for.  But, CopyRates is not always available for that bar immediately, it can take some time before the terminal is prepared to deliver the data after its first request.

  • Is there a direct way to ask the terminal what the oldest bar CoypRates will work for is, for a given Symbol and timeframe?

If I could get an affirmative on that question (with the method), all would be well.

I believe iHigh, iLow, IOpen, IClose all have the same data availablity as CopyRates.

SeriesInfoInteger should help you.
 
Alain Verleyen:
SeriesInfoInteger should help you.

SeriesInfoInteger seems to be a better way to get the valid bars information, but it still suffers from returning bars as valid that CopyRates cannot get information for.

I made, and published, a twin indicator to the one referenced above that uses SeriesInfoInteger instead of Bars to get the bar counts.

https://www.mql5.com/en/code/26330\

SeriesIntegerInfo and Rates Bar Information Utility, displays how many bars there are in all timeframes for any user selected symbol, and shows the 'CopyRates' information availablity for any bar
SeriesIntegerInfo and Rates Bar Information Utility, displays how many bars there are in all timeframes for any user selected symbol, and shows the 'CopyRates' information availablity for any bar
  • www.mql5.com
This utility is not a trading information utility.  It will only have value for those getting information on the Number of Bars that have terminal data.  This utility uses the functions SeriesIntegerInfo and CopyRates to obtain and display information about the availability of bar information from the terminal. The utility allows the user to...
 
Alain Verleyen #:
You just need to wait the next tick and try again

Apart from the fact that waiting for the second click is an unacceptable anomaly of the architecture which defines itself as "professional", and that when it's the weekend no second click arrives, I can also accept the compromise but the problem is that "prev_calculated" variable is already gone, to go and restore the cycle on the data you have to keep it if the reading is not successful.

The architecture is so flawed that not even explicitly calling CopyRates, cycled with a sleep, solves the problem. Necessarily wants a tick, ChartRedraw is completely useles. The ideal solution would be to invoke the "data refresh" key (right click, click on "update"), but "obviously" there is no such function helping about it.

I'm wondering does anyone know the equivalent key combination?, with windows GUI programming would it always be possible to write a DLL that resembles specifically clicking the "update" button.

Has anyone tried?

 
Sabino Martiradonna #:

Apart from the fact that waiting for the second click is an unacceptable anomaly of the architecture which defines itself as "professional", and that when it's the weekend no second click arrives, I can also accept the compromise but the problem is that "prev_calculated" variable is already gone, to go and restore the cycle on the data you have to keep it if the reading is not successful.

The architecture is so flawed that not even explicitly calling CopyRates, cycled with a sleep, solves the problem. Necessarily wants a tick, ChartRedraw is completely useles. The ideal solution would be to invoke the "data refresh" key (right click, click on "update"), but "obviously" there is no such function helping about it.

I'm wondering does anyone know the equivalent key combination?, with windows GUI programming would it always be possible to write a DLL that resembles specifically clicking the "update" button.

Has anyone tried?

Becoming emotional leads nowhere.

The call of ChartSetSymbolPeriod with the same symbol and timeframe can be used to update the chart (similar to the terminal's Refresh command). In its turn, the chart update triggers re-calculation of the indicators attached to it. Thus, you are able to calculate an indicator on the chart even if there are no ticks (e.g., on weekends).

Documentation on MQL5: Chart Operations / ChartSetSymbolPeriod
Documentation on MQL5: Chart Operations / ChartSetSymbolPeriod
  • www.mql5.com
ChartSetSymbolPeriod - Chart Operations - MQL5 Reference - Reference on algorithmic/automated trading language for MetaTrader 5
Reason: