MQL5 Economic Calendar / CalendarValueHistory in backtest

To add comments, please log in or register
Eric Emmrich
3829
Eric Emmrich  

Hi all,

is there any way to access the built-in economic calendar during backtest?

I tried CalendarValueHistory and while it does give proper results when executing the code on a live chart, it returns no results during backtest.

Thank you in advance.

Documentation on MQL5: Economic Calendar / CalendarValueHistory
Documentation on MQL5: Economic Calendar / CalendarValueHistory
  • www.mql5.com
//| Script program start function                                    |         [id] [event_id]           [time]               [] [revision]   [actual_value] [prev_value] [revised_prev_value] [forecast_value] [impact_type] [reserved]...
Chris70
644
Chris70  
Eric Emmrich:

Hi all,

is there any way to access the built-in economic calendar during backtest?

Yes, there is. We've had this topic on a few occasions in the german part of the forum and according to your name a guess you also speak german(?).

Use the files from this link: https://www.mql5.com/de/forum/326263#comment_13885444,

some more explanations here: https://www.mql5.com/de/forum/319172

Generally speaking, the trick is to read all relevant information from a file that has previously been created (and stored in the common files folder) if the EA/sript/indicator has at least once been started in live mode (or alternatively debugging with live data). Then you have those data for eternity, as long as the backtest period precedes that date that the code hast last been executed in live mode.

Round about 90.000 past news events are accessible through this method.

Another benefit of using the suggested method (not only with backtests / historical data): It seems like an unnecessary complication that MQL distinguishes between the event information that is available via the CalendarEventHistory method and data accessible by event ID, instead of treating all data related to an event the same way in one big struct[].xxx kind of variable. And here comes the advantage with my method:

You can access all infos the same way, e.g. news.event[index].actual_value, news.event[index].country_id, news.event[index].time etc...... and it's fairly easy: all you need to do in order to access these values (regardless if in live mode or backtests) is add the files in the link  as "include" and call the news.update() function [edit: =assuming we've created the class object "CNews news;"].

Alain Verleyen
38725
Alain Verleyen  
Chris70:

Yes, there is. We've had this topic on a few occasions in the german part of the forum and according to your name a guess you also speak german(?).

Use the files from this link: https://www.mql5.com/de/forum/326263#comment_13885444,

some more explanations here: https://www.mql5.com/de/forum/319172

Generally speaking, the trick is to read all relevant information from a file that has previously been created (and stored in the common files folder) if the EA/sript/indicator has at least once been started in live mode (or alternatively debugging with live data). Then you have those data for eternity, as long as the backtest period precedes that date that the code hast last been executed in live mode.

Round about 90.000 past news events are accessible through this method.

Another benefit of using the suggested method (not only with backtests / historical data): It seems like an unnecessary complication that MQL distinguishes between the event information that is available via the CalendarEventHistory method and data accessible by event ID, instead of treating all data related to an event the same way in one big struct[].xxx kind of variable. And here comes the advantage with my method:

You can access all infos the same way, e.g. news.event[index].actual_value, news.event[index].country_id, news.event[index].time etc...... and it's fairly easy: all you need to do in order to access these values (regardless if in live mode or backtests) is add the files in the link  as "include" and call the news.update() function.

Do you know what are the oldest news date available this way ?
Chris70
644
Chris70  
Alain Verleyen:
Do you know what are the oldest news date available this way ?

generally it starts from jan 2007 but it should be mentioned that the more recent events are much more complete --> many of the very old events are not listed

[edit: I just looked at events of 2010 and I can't see much of a difference in the amount of data compared to now; don't know when exactly it get's worse, so somewhere before 2010; in other words: we have a good 10+ years of events to work with --> should be good for most backtest scenarios]

historical news example: 2010

Alain Verleyen
38725
Alain Verleyen  
Chris70:

generally it starts from jan 2007 but it should be mentioned that the more recent events are much more complete --> many of the very old events are not listed

[edit: I just looked at events of 2010 and I can't see much of a difference in the amount of data compared to now; don't know when exactly it get's worse, so somewhere before 2010; in other words: we have a good 10+ years of events to work with --> should be good for most backtest scenarios]


Thanks :-)
Eric Emmrich
3829
Eric Emmrich  
Chris70:

Yes, there is. We've had this topic on a few occasions in the german part of the forum and according to your name a guess you also speak german(?).

Use the files from this link: https://www.mql5.com/de/forum/326263#comment_13885444,

some more explanations here: https://www.mql5.com/de/forum/319172

Generally speaking, the trick is to read all relevant information from a file that has previously been created (and stored in the common files folder) if the EA/sript/indicator has at least once been started in live mode (or alternatively debugging with live data). Then you have those data for eternity, as long as the backtest period precedes that date that the code hast last been executed in live mode.

Round about 90.000 past news events are accessible through this method.

Another benefit of using the suggested method (not only with backtests / historical data): It seems like an unnecessary complication that MQL distinguishes between the event information that is available via the CalendarEventHistory method and data accessible by event ID, instead of treating all data related to an event the same way in one big struct[].xxx kind of variable. And here comes the advantage with my method:

You can access all infos the same way, e.g. news.event[index].actual_value, news.event[index].country_id, news.event[index].time etc...... and it's fairly easy: all you need to do in order to access these values (regardless if in live mode or backtests) is add the files in the link  as "include" and call the news.update() function [edit: =assuming we've created the class object "CNews news;"].

Thank you very much for this detailed explanation and the effort you put into the workaround for the missing news in the backtest! I've tested it already and I am pretty pleased with your news class :)

Culture Capital Ltd.
166
Mark Lewis  

Firstly, thank you! this is amazing. everything works great.

However, I only seem to be able to get data back to November 2017. Have i done something wrong? is the amount of data held broker dependant?

here are the first and last data points i can access at index values 0 and total_events:


history available


regards

Mark

Chris70
644
Chris70  

I'm glad it is useful. I assume you're using the latest version (https://www.mql5.com/en/forum/312462#comment_14491488 or file below).

It is a little strange to see those very large numbers for some values. I didn't observe that myself. Of course something like a speech has e.g. no 'actual value' or 'forecast value', but it's strange that they are not assigned with zeros then. This shouldn't be an initialization error, because whatever is initialized is actively overwritten with data derived from the MQL5 calendar functions.

The last event in the list should be the one with index "total_events-1".

About the dates, as mentioned in an earlier post, I'm getting access to about 90.000 historical events (plus events without a specific time of day, like national holidays), starting from jan 2007. It surprises me to see that you can only access data since 2017. As I assume that these data are supplied by Metaquotes and not the broker, we should see the same results.

If you want you can send me the .mq5 file here or via private message and I'll be happy to check if I see different results or whatever the issue may be.


You can also use this as a test code to see which range of data you get (for me it starts from 06.jan2007):

#include <News.mqh>
CNews news;

int OnInit()
  {
   
   return(INIT_SUCCEEDED);
  }
  
void OnTick()
  {
   int events_total=news.update();
   for (int n=0;n<10;n++){printlog_show_event(n);}
   for (int n=events_total-10;n<events_total;n++){printlog_show_event(n);}
   ExpertRemove();
  }
  
void printlog_show_event(int index)
  {
   Print("event ",index,": ",news.eventname[index],", ",TimeToString(news.event[index].time));
   if (news.event[index].unit!=CALENDAR_UNIT_NONE)
     {
      Print("--- actual: ",DoubleToString(news.event[index].actual_value,2),
            ", previous: ",DoubleToString(news.event[index].prev_value,2),
            ", forecast: ",DoubleToString(news.event[index].forecast_value,2));
     }
  }
Files:
News.mqh 24 kb
Culture Capital Ltd.
166
Mark Lewis  

Thanks so much Chris!

looks like the same outcome to me using your test code.

the initialisation thing where it displays the smallest possible long value rather than 0 is no biggie for me (https://www.mql5.com/en/docs/basis/types/integer/integertypes), I can easily work around that one. 

You also mentioned that the newshistory.bin file does not rewrite over the old data with every refresh, merely updates with the new information, so does this mean all I need is to replace my newshistory.bin with someone else's who has all the data AND the same broker time AND the same language?  MY broker (IC Markets) is NY close broker time so might be a simple fix if so. Or I could open demos with Pepperstone or whoever just to get the full history  .bin file for example

regards

Mark


news test code

Documentation on MQL5: Language Basics / Data Types / Integer Types / Char, Short, Int and Long Types
Documentation on MQL5: Language Basics / Data Types / Integer Types / Char, Short, Int and Long Types
  • www.mql5.com
//| Script program start function                                    | //| return the session's working hours as a string                   | //
Chris70
644
Chris70  

The date of the last update is saved with the newshistory.bin file, it is checked against and there is a shortcut included to prevent overwriting if the current date is in the future relative to the time stamp that the update() function is called upon.

I did this in part for performance reasons. This also means that - if you run an expert on historical data in the tester (="past" time stamps) or via genetic optimization - once saved data are kept.

Only the date(!) of the most recent saving matters. The language or the broker that you are logged in with is irrelevant. I'm astonished that you don't get data from back to jan. 2007, too, cause a would have thought that they're supplied by Metaquotes and that the broker doesn't change anything, but I might be wrong. If you have several brokers (demo or live), you can of cause check if it makes a difference, maybe even with Metaquotes' demo accounts. You are right that the newshistory.bin file won't be overwritten then and that files from different sources should be compatible (meaning that it doesn't need to be the same broker, language or time zone; just make sure that if your actual broker is in the NY time zone you use the correct settings for the GMT offset for winter time and daylight saving time). I sticked to only my main broker because events since 2007 are more than enough for backtesting. Please keep me updated if different broker give you more data.

If you have any other problems or ideas how to improve the functionality of that file, feel free to post it here and I'll be happy to put those improvements into code.

Honestly, it's far from any code that I'm proud of, I did very little in the field of news trading over the past months and in retrospect I would do it in a better organized way. I share it anyway cause here in the forum it just seems that people again and again stumble upon problems using the build-in calendar functions.

Culture Capital Ltd.
166
Mark Lewis  

fixed it!

i tried just deleting the newshistory.bin file and making it create a new one. When it did that, it downloaded the full 106,371  data points going back to 2007


fixed


i think your implementation is great, the only thing i added was this (see below) so that it is easier to differentiate between European and German data points for example. 

//| country id to country                                           |
//+------------------------------------------------------------------+
string CNews::CountryIdToCountry(ENUM_COUNTRY_ID c)
  {
   switch(c)
     {
      case 999:      return "EU";     		// EU
      case 840:      return "USA";     		// USA
      case 36:       return "Australia";     	// Australia
      case 554:      return "NewZealand";     	// NewZealand
      case 156:      return "China";     	// China
      case 826:      return "UK";     		// UK
      case 756:      return "Switzerland";     	// Switzerland
      case 276:      return "Germany";     	// Germany
      case 250:      return "France";     	// France
      case 380:      return "Italy";     	// Italy
      case 724:      return "Spain";     	// Spain
      case 76:       return "Brazil";     	// Brazil
      case 410:      return "South Korea";     	// South Korea
      default:       return "";
     }
  }  

The only other thing i noticed, the common folder is shared by all installs of MT5 right? Does that mean that if I install two copies of MT5 on the same machine - a NY and a London broker time install, that it might mix up the broker times for the downloaded news?

Pretty simple to redirect that to the  MQL5/Files  folder if that is the case though


regards

Mark

12
To add comments, please log in or register