MT5 - Multi-Currency Indicator in Tester

New comment
 

I have an indicator that pulls from eight different currency pairs.

Into this indicator I have code that ensures historical data for all pairs is available. If the code is not available, it will pull the data from the broker.

So far, nothing out of the ordinary.

The oddity is that this "ensuring" and "pulling" code does not work in the Tester, and I am not exactly sure what it is doing. At present, I'm in an infinite loop looking for data (using BarsCalculated()) and trying to pull data (Using CopyTime()).

I think there is some sort of thread contention where the Tester is running in one thread and has the indicator work in another thread.



The obvious fix is to have the data already downloaded. The indicator works fine then, in the Tester.

But I cannot assume the data is downloaded when I use the automated validator, i.e. when publishing an indicator.

Any thoughts on what the Tester is doing and/or how to handle this situation?


 
You don't need to do that by code. The Strategy Tester itself will download the needed data, you just have to use to use your CopyTime() call on each symbol at the start.
 
Alain Verleyen:
You don't need to do that by code. The Strategy Tester itself will download the needed data, you just have to use to use your CopyTime() call on each symbol at the start.

Thank you for your response.

Do you mean that I need only call CopyTime() only once per instrument, like in OnInit(), when using the Tester?

 
Anthony Garot:

Thank you for your response.

Do you mean that I need only call CopyTime() only once per instrument, like in OnInit(), when using the Tester?

Yes. The first time you call it, it will download the data for the given symbol.

int OnInit()
  {
//--- initialize pairs
   string symbols[]={ "EURUSD","GBPUSD","USDCAD" };
   int size=ArraySize(symbols);
//---
   datetime times[];
   for(int i=0;i<size;i++)
     {
      int countTime=CopyTime(symbols[i],_Period,TimeCurrent(),1000,times);
     }

//---
   return(INIT_SUCCEEDED);
  }

2018.10.30 09:39:33.849    Tester    EURUSD: preliminary downloading of M1 history started
2018.10.30 09:39:35.766    Tester    EURUSD: 20% history downloaded
2018.10.30 09:39:36.568    Tester    EURUSD: 30% history downloaded
2018.10.30 09:39:37.593    Tester    EURUSD: 40% history downloaded
2018.10.30 09:39:38.595    Tester    EURUSD: 51% history downloaded
2018.10.30 09:39:39.598    Tester    EURUSD: 60% history downloaded
2018.10.30 09:39:41.002    Tester    EURUSD: 72% history downloaded
2018.10.30 09:39:42.104    Tester    EURUSD: 80% history downloaded
2018.10.30 09:39:43.408    Tester    EURUSD: 90% history downloaded
2018.10.30 09:39:44.866    Tester    EURUSD: preliminary downloading of M1 history completed in 0:11.031
2018.10.30 09:39:44.866    Tester    EURUSD: history data begins from 2014.11.24 00:00
2018.10.30 09:39:44.945    Core 1    agent process started
2018.10.30 09:39:44.945    Core 1    connecting to 127.0.0.1:3000
2018.10.30 09:39:50.807    Core 1    connected
2018.10.30 09:39:50.829    Core 1    authorized (agent build 1932)
2018.10.30 09:39:50.830    Tester    EURUSD,H1 (MetaQuotes-Beta): visual testing of Indicators\Forum\TesterAutoDownload.ex5 from 2016.09.01 00:00 to 2018.09.01 00:00
2018.10.30 09:39:50.965    Core 1    common synchronization completed
2018.10.30 09:39:53.296    Tester    quality of analyzed history is 98%
2018.10.30 09:39:53.506    Core 1    GBPUSD: history synchronization started
2018.10.30 09:40:18.339    Core 1    GBPUSD: history downloading completed
2018.10.30 09:40:21.455    Core 1    USDCAD: history synchronization started
2018.10.30 09:40:48.069    Core 1    USDCAD: history downloading completed

By the way, it is downloading ALL history for each symbol, whatever the dates you are using.
 
Alain Verleyen:

Yes. The first time you call it, it will download the data for the given symbol.

Well, that certainly makes things simpler. Thank you!

 

By the way, for anyone reading this thread, I found the specific problem of why my code wasn't working in the Tester.

BarsCalculated() sometimes returns -1 even though bars are there. I know the data is there, because I pre-loaded them.

So, when BarsCalculated() returned -1, my code ran CopyTime() to request more data, but I didn't check the result of CopyTime(), and an infinite loop occurred.

The fix:

CopyTime() actually returns a result when the data is there!

So, in pseudo code:

// Loop over all symbols
for ( int j=0; j<NUM_SYMBOLS;j++ )
{
    if ( BarsCalculated < rates_total )
    {
        if ( CopyTime doesn't return a value )
        {
            isSynchronized = false;
            break;
        }
    }
}

To sum up: there is now a double check of data existence. When BarsCalculated() returns -1 but CopyTime() returns a value, the data is there.

If anyone wants a copy of the code, contact me. Not posting here because it's still under development.

 
Just as a side note (not knowing your exact code, but because you use isSynchronized variable) - probably you should look at SymbolIsSynchronized as well? 
 
Stanislav Korotky:
Just as a side note (not knowing your exact code, but because you use isSynchronized variable) - probably you should look at SymbolIsSynchronized as well? 

I had SymbolIsSynchronized() integrated in my code, but then I removed it because it always returned true.

My understanding of this function is: "If it return false, then we know that there's new/updated price at server, and/or the price that EA or CI is currently working is old/out-of-date."

Since my indicator only uses fully formed bars, I assumed that this is why the function never returned false.

If I am operating under a mistaken understanding, please let me know.

 
Anthony Garot:

I have an indicator that pulls from eight different currency pairs.

Into this indicator I have code that ensures historical data for all pairs is available. If the code is not available, it will pull the data from the broker.

So far, nothing out of the ordinary.

The oddity is that this "ensuring" and "pulling" code does not work in the Tester, and I am not exactly sure what it is doing. At present, I'm in an infinite loop looking for data (using BarsCalculated()) and trying to pull data (Using CopyTime()).

I think there is some sort of thread contention where the Tester is running in one thread and has the indicator work in another thread.



The obvious fix is to have the data already downloaded. The indicator works fine then, in the Tester.

But I cannot assume the data is downloaded when I use the automated validator, i.e. when publishing an indicator.

Any thoughts on what the Tester is doing and/or how to handle this situation?


Hi Anthony, Alain et al,

It seems like you may be analyzing higher timeframes, but I was wondering if you have any thoughts on handling lower timeframes such as M1/M5/M15, to deal with holes in the historical data (e.g. due to no ticks for a particular bar/symbol, or different daily/weekly opening times for the different pairs by a particular broker). By simply indexing back through the time series, even if all the historical data has been properly loaded, the Nth index for say EURUSD may represent a totally different time than say AUDCHF or any other pair for that matter.

For example when you are using 8 different currency pairs, and then present this on a chart for one of those or possibly some other symbol, there can be holes in any of 8 pairs as well as holes in the chart symbol's data that have to be accounted for. I have been looking at a similar problem, and using iTime to see if a particular bar represents the same time for all currencies, and if not then trying to figure out how to compensate for the missing data. And similarly, when presenting the data on a chart, the specific bar for the chart needs to select the data from the 8 currencies that corresponds to this bar. It is quite tedious and I was really hoping that you might have some thoughts on how to effectively deal with this? For example is there a way to have MT4/Mt5 to fill in with 0-tick bars so that there are no holes/mismatches in the indexed arrays?

Any pointers or help on this will be much appreciated.

Thanks,

Al

 

albry:

By simply indexing back through the time series, even if all the historical data has been properly loaded, the Nth index for say EURUSD may represent a totally different time than say AUDCHF or any other pair for that matter.

I have not run into the scenario, so I have nothing I can suggest.

 
Anthony Garot:

I have not run into the scenario, so I have nothing I can suggest.

Hi Anthony,

Thanks for getting back on this. Much appreciated !

I originally also didn't think that I had any problems until I started running this little bit of code periodically (after all history has been loaded) on each timeframe used (M1 to D1), to check whether all the pairs had the same index limits and hence (likely) indexed the same times or NOT... Generally had issues on every timeframe but M1 and M5 clearly the most problematic... 

   

   // #define  maxSymbols 28
   // string   currentSymNamesTable[maxSymbols];         // stores full symbol name with prefix/suffix
   // int      currentSymbolsTotal;                      // stores total number of symbols in currentSymNamesTable

   string str=""; datetime dt=0; int limit = 1000; 

   for (i = 0; i < currentSymbolsTotal; i++) {
      datetime cdt = iTime(currentSymNamesTable[i], 0, limit); 
      if (dt == 0) {dt = cdt;}
      if (cdt != dt) {
         if (str == "") {str += "Time Diffs ("+string(_Period)+") - "+currentSymNamesTable[0]+": "+TimeToStr(dt,TIME_DATE|TIME_MINUTES);}
         str += ", "+currentSymNamesTable[i]+": "+TimeToStr(cdt,TIME_DATE|TIME_MINUTES);
      }     
   }
   if (str != "") Print(str);
12
New comment
Reason: