MQL5 Economic Calendar / CalendarValueHistory in backtest

[Eric Emmrich]

Hi all,

is there any way to access the built-in economic calendar during backtest?

I tried CalendarValueHistory and while it does give proper results when executing the code on a live chart, it returns no results during backtest.

Thank you in advance.

[Chris70]

Eric Emmrich:

Hi all,

is there any way to access the built-in economic calendar during backtest?

Yes, there is. We've had this topic on a few occasions in the german part of the forum and according to your name a guess you also speak german(?).

Use the files from this link: https://www.mql5.com/de/forum/326263#comment_13885444,

some more explanations here: https://www.mql5.com/de/forum/319172

Generally speaking, the trick is to read all relevant information from a file that has previously been created (and stored in the common files folder) if the EA/sript/indicator has at least once been started in live mode (or alternatively debugging with live data). Then you have those data for eternity, as long as the backtest period precedes that date that the code hast last been executed in live mode.

Round about 90.000 past news events are accessible through this method.

Another benefit of using the suggested method (not only with backtests / historical data): It seems like an unnecessary complication that MQL distinguishes between the event information that is available via the CalendarEventHistory method and data accessible by event ID, instead of treating all data related to an event the same way in one big struct[].xxx kind of variable. And here comes the advantage with my method:

You can access all infos the same way, e.g. news.event[index].actual_value, news.event[index].country_id, news.event[index].time etc...... and it's fairly easy: all you need to do in order to access these values (regardless if in live mode or backtests) is add the files in the link  as "include" and call the news.update() function [edit: =assuming we've created the class object "CNews news;"].

[Alain Verleyen]

Chris70:

Yes, there is. We've had this topic on a few occasions in the german part of the forum and according to your name a guess you also speak german(?).

Use the files from this link: https://www.mql5.com/de/forum/326263#comment_13885444,

some more explanations here: https://www.mql5.com/de/forum/319172

Generally speaking, the trick is to read all relevant information from a file that has previously been created (and stored in the common files folder) if the EA/sript/indicator has at least once been started in live mode (or alternatively debugging with live data). Then you have those data for eternity, as long as the backtest period precedes that date that the code hast last been executed in live mode.

Round about 90.000 past news events are accessible through this method.

Another benefit of using the suggested method (not only with backtests / historical data): It seems like an unnecessary complication that MQL distinguishes between the event information that is available via the CalendarEventHistory method and data accessible by event ID, instead of treating all data related to an event the same way in one big struct[].xxx kind of variable. And here comes the advantage with my method:

You can access all infos the same way, e.g. news.event[index].actual_value, news.event[index].country_id, news.event[index].time etc...... and it's fairly easy: all you need to do in order to access these values (regardless if in live mode or backtests) is add the files in the link  as "include" and call the news.update() function.

Do you know what are the oldest news date available this way ?

[Chris70]

Alain Verleyen:
Do you know what are the oldest news date available this way ?

generally it starts from jan 2007 but it should be mentioned that the more recent events are much more complete --> many of the very old events are not listed

[edit: I just looked at events of 2010 and I can't see much of a difference in the amount of data compared to now; don't know when exactly it get's worse, so somewhere before 2010; in other words: we have a good 10+ years of events to work with --> should be good for most backtest scenarios]

[Alain Verleyen]

Chris70:

generally it starts from jan 2007 but it should be mentioned that the more recent events are much more complete --> many of the very old events are not listed

[edit: I just looked at events of 2010 and I can't see much of a difference in the amount of data compared to now; don't know when exactly it get's worse, so somewhere before 2010; in other words: we have a good 10+ years of events to work with --> should be good for most backtest scenarios]

Thanks :-)

[Eric Emmrich]

Chris70:

Yes, there is. We've had this topic on a few occasions in the german part of the forum and according to your name a guess you also speak german(?).

Use the files from this link: https://www.mql5.com/de/forum/326263#comment_13885444,

some more explanations here: https://www.mql5.com/de/forum/319172

Generally speaking, the trick is to read all relevant information from a file that has previously been created (and stored in the common files folder) if the EA/sript/indicator has at least once been started in live mode (or alternatively debugging with live data). Then you have those data for eternity, as long as the backtest period precedes that date that the code hast last been executed in live mode.

Round about 90.000 past news events are accessible through this method.

Another benefit of using the suggested method (not only with backtests / historical data): It seems like an unnecessary complication that MQL distinguishes between the event information that is available via the CalendarEventHistory method and data accessible by event ID, instead of treating all data related to an event the same way in one big struct[].xxx kind of variable. And here comes the advantage with my method:

You can access all infos the same way, e.g. news.event[index].actual_value, news.event[index].country_id, news.event[index].time etc...... and it's fairly easy: all you need to do in order to access these values (regardless if in live mode or backtests) is add the files in the link  as "include" and call the news.update() function [edit: =assuming we've created the class object "CNews news;"].

Thank you very much for this detailed explanation and the effort you put into the workaround for the missing news in the backtest! I've tested it already and I am pretty pleased with your news class :)

[Mark Lewis]

Firstly, thank you! this is amazing. everything works great.

However, I only seem to be able to get data back to November 2017. Have i done something wrong? is the amount of data held broker dependant?

here are the first and last data points i can access at index values 0 and total_events:

regards

Mark

[Chris70]

I'm glad it is useful. I assume you're using the latest version (https://www.mql5.com/en/forum/312462#comment_14491488 or file below).

It is a little strange to see those very large numbers for some values. I didn't observe that myself. Of course something like a speech has e.g. no 'actual value' or 'forecast value', but it's strange that they are not assigned with zeros then. This shouldn't be an initialization error, because whatever is initialized is actively overwritten with data derived from the MQL5 calendar functions.

The last event in the list should be the one with index "total_events-1".

About the dates, as mentioned in an earlier post, I'm getting access to about 90.000 historical events (plus events without a specific time of day, like national holidays), starting from jan 2007. It surprises me to see that you can only access data since 2017. As I assume that these data are supplied by Metaquotes and not the broker, we should see the same results.

If you want you can send me the .mq5 file here or via private message and I'll be happy to check if I see different results or whatever the issue may be.

You can also use this as a test code to see which range of data you get (for me it starts from 06.jan2007):

#include <News.mqh>
CNews news;

int OnInit()
  {
   
   return(INIT_SUCCEEDED);
  }
  
void OnTick()
  {
   int events_total=news.update();
   for (int n=0;n<10;n++){printlog_show_event(n);}
   for (int n=events_total-10;n<events_total;n++){printlog_show_event(n);}
   ExpertRemove();
  }
  
void printlog_show_event(int index)
  {
   Print("event ",index,": ",news.eventname[index],", ",TimeToString(news.event[index].time));
   if (news.event[index].unit!=CALENDAR_UNIT_NONE)
     {
      Print("--- actual: ",DoubleToString(news.event[index].actual_value,2),
            ", previous: ",DoubleToString(news.event[index].prev_value,2),
            ", forecast: ",DoubleToString(news.event[index].forecast_value,2));
     }
  }

[Mark Lewis]

Thanks so much Chris!

looks like the same outcome to me using your test code.

the initialisation thing where it displays the smallest possible long value rather than 0 is no biggie for me (https://www.mql5.com/en/docs/basis/types/integer/integertypes), I can easily work around that one. 

You also mentioned that the newshistory.bin file does not rewrite over the old data with every refresh, merely updates with the new information, so does this mean all I need is to replace my newshistory.bin with someone else's who has all the data AND the same broker time AND the same language?  MY broker (IC Markets) is NY close broker time so might be a simple fix if so. Or I could open demos with Pepperstone or whoever just to get the full history  .bin file for example

regards

Mark

[Chris70]

The date of the last update is saved with the newshistory.bin file, it is checked against and there is a shortcut included to prevent overwriting if the current date is in the future relative to the time stamp that the update() function is called upon.

I did this in part for performance reasons. This also means that - if you run an expert on historical data in the tester (="past" time stamps) or via genetic optimization - once saved data are kept.

Only the date(!) of the most recent saving matters. The language or the broker that you are logged in with is irrelevant. I'm astonished that you don't get data from back to jan. 2007, too, cause a would have thought that they're supplied by Metaquotes and that the broker doesn't change anything, but I might be wrong. If you have several brokers (demo or live), you can of cause check if it makes a difference, maybe even with Metaquotes' demo accounts. You are right that the newshistory.bin file won't be overwritten then and that files from different sources should be compatible (meaning that it doesn't need to be the same broker, language or time zone; just make sure that if your actual broker is in the NY time zone you use the correct settings for the GMT offset for winter time and daylight saving time). I sticked to only my main broker because events since 2007 are more than enough for backtesting. Please keep me updated if different broker give you more data.

If you have any other problems or ideas how to improve the functionality of that file, feel free to post it here and I'll be happy to put those improvements into code.

Honestly, it's far from any code that I'm proud of, I did very little in the field of news trading over the past months and in retrospect I would do it in a better organized way. I share it anyway cause here in the forum it just seems that people again and again stumble upon problems using the build-in calendar functions.

[Mark Lewis]

fixed it!

i tried just deleting the newshistory.bin file and making it create a new one. When it did that, it downloaded the full 106,371  data points going back to 2007

i think your implementation is great, the only thing i added was this (see below) so that it is easier to differentiate between European and German data points for example. 

//| country id to country                                           |
//+------------------------------------------------------------------+
string CNews::CountryIdToCountry(ENUM_COUNTRY_ID c)
  {
   switch(c)
     {
      case 999:      return "EU";     		// EU
      case 840:      return "USA";     		// USA
      case 36:       return "Australia";     	// Australia
      case 554:      return "NewZealand";     	// NewZealand
      case 156:      return "China";     	// China
      case 826:      return "UK";     		// UK
      case 756:      return "Switzerland";     	// Switzerland
      case 276:      return "Germany";     	// Germany
      case 250:      return "France";     	// France
      case 380:      return "Italy";     	// Italy
      case 724:      return "Spain";     	// Spain
      case 76:       return "Brazil";     	// Brazil
      case 410:      return "South Korea";     	// South Korea
      default:       return "";
     }
  }  

The only other thing i noticed, the common folder is shared by all installs of MT5 right? Does that mean that if I install two copies of MT5 on the same machine - a NY and a London broker time install, that it might mix up the broker times for the downloaded news?

Pretty simple to redirect that to the  MQL5/Files  folder if that is the case though

regards

Mark