Historical spread during backtesting in the strategy tester is different from the historical spread stored in the Mql rates array. What are they? Why they are different?
I have found accidently that the historical spread (Ask-bid) used to open and close orders during backtesting in the strategy tseter is different from those stored in the Mql rates.
For example, you can get historical spread using the following code (Lets say your EA is attached on M1 chart).
MqlRates rates[];
ArraySetAsSeries(rates,true);
int copied=CopyRates(Symbol(),0,0,100,rates);
int yourSpread = rates[i].spread;
However the spread obtained using this methods seems different from the spread used to open and close the orders during backtesting using 1 minutes OHLC tick.
I have also found that the spread during backtesting using 1 minutes OHLC price is different from the spread during backtesting using open price only too.
So why they are different? Where the Metatrader 5 getting those historical spread information during backtesting? Are they simulating the spread rather than using historical spread during backtesting?
Can you provide all data (ST settings, logs, screenshot...and eventually code) that demonstrate your issue ?
AFAIK, spread used by the Strategy Tester comes from M1 historical data.
I have found accidently that the historical spread (Ask-bid) used to open and close orders during backtesting in the strategy tseter is different from those stored in the Mql rates.
For example, you can get historical spread using the following code (Lets say your EA is attached on M1 chart).
MqlRates rates[];
ArraySetAsSeries(rates,true);
int copied=CopyRates(Symbol(),0,0,100,rates);
int yourSpread = rates[i].spread;
However the spread obtained using this methods seems different from the spread used to open and close the orders during backtesting using 1 minutes OHLC tick.
I have also found that the spread during backtesting using 1 minutes OHLC price is different from the spread during backtesting using open price only too.
So why they are different? Where the Metatrader 5 getting those historical spread information during backtesting? Are they simulating the spread rather than using historical spread during backtesting?
What I aimed was to get the historical spread when the H1 bar is just opened. So I can analyze further.
To demonstrate my problem, I will attach three data files.
HistoricalSpreadFromBackTesting.csv containes the historical spread recorded from backtesting when each H1 bar is opened. I use OnTick function to do the task. The code is written below.
void OnTick()
{
if(IsNewBarForAllSymbol())
{
datetime curTime = TimeCurrent();
string str = TimeToString(curTime);
for(int i = 0; i < symbolDim; i++)
{
string symbolToTrade = SymbolArray[i];
double ask=SymbolInfoDouble(symbolToTrade,SYMBOL_ASK);
double bid=SymbolInfoDouble(symbolToTrade,SYMBOL_BID);
double spread=ask-bid;
int spreadPoint=(int)MathRound(spread/SymbolInfoDouble(symbolToTrade,SYMBOL_POINT));
str = str + ", " + IntegerToString(spreadPoint);
}
FileWrite(fileHandle, str);
}
}
bool IsNewBar(int i)
{
string symbolToTrade = SymbolArray[i];
datetime Tnew[1];
CopyTime(symbolToTrade,PERIOD_CURRENT,0,1,Tnew);
if(Tnew[0]!=Told[i])
{
Told[i]=Tnew[0];
return(true);
}
return(false);
}
//+-----------------------------------+
bool IsNewBarForAllSymbol()
{
for(int i =0; i < symbolDim; i++)
{
if(IsNewBar(i) == true)
{
newBarSymbolCnt = newBarSymbolCnt + 1;
}
}
if(newBarSymbolCnt >= symbolDim)
{
newBarSymbolCnt = 0;
return(true);
}
else
{
return(false);
}
}
HistoricalSpreadFromMQLRateMethod.csv containes the historical spread from MQL rate methods and the some part of the code is written below.
string SymbolCode1 = "EURUSD";
string SymbolCode2 = "GBPUSD";
string SymbolCode3 = "EURGBP";
string SymbolCode4 = "NZDUSD";
string SymbolCode5 = "AUDUSD";
string SymbolCode6 = "USDCAD";
string SymbolCode7 = "USDCHF";
string SymbolCode8 = "USDJPY";
string SymbolCode9 = "EURCAD";
string SymbolCode10 = "EURAUD";
string SymbolCode11 = "EURJPY";
string SymbolCode12 = "EURCHF";
int copied;
MqlRates rates_H1[], rates_M1[];
ArraySetAsSeries(rates_H1,false);
ArraySetAsSeries(rates_M1,false);
copied=CopyRates(SymbolCode1,0,0,length,rates_H1);
if(copied>0)
{
for(int i=0;i<length;i++)
{
CopyRates(SymbolCode1,PERIOD_M1,rates_H1[i].time,1,rates_M1);
spreadMatrix[i][0] = rates_M1[0].spread;
}
}
else Print("Failed to get history data for the symbol ",SymbolCode1);
copied=CopyRates(SymbolCode2,0,0,length,rates_H1);
if(copied>0)
{
for(int i=0;i<length;i++)
{
CopyRates(SymbolCode2,PERIOD_M1,rates_H1[i].time,1,rates_M1);
spreadMatrix[i][1] = rates_M1[0].spread;
}
}
else Print("Failed to get history data for the symbol ",SymbolCode2);
copied=CopyRates(SymbolCode3,0,0,length,rates_H1);
if(copied>0)
{
for(int i=0;i<length;i++)
{
CopyRates(SymbolCode3,PERIOD_M1, rates_H1[i].time, 1, rates_M1);
spreadMatrix[i][2] = rates_M1[0].spread;
}
}
else Print("Failed to get history data for the symbol ",SymbolCode3);
Between these two file I calculated the discrepency in HistoricalSpreadDiscrepencyCheck.xlsx file.
From 2013.07.23.04:00 to 2014.01.16 23:00 for 3002 H1 bar, the discrepency between two spreads are shown in the table below (Sorry for bad tables!!!). For some pairs like NZDUSD, the dicrepancy is quite serious. They are about 3573 points difference between the two spreads.
I don't understand where these discrepency comes from? How can we make things more accurately?
| Pairs | EURUSD Spread | GBPUSD Spread | EURGBP Spread | NZDUSD Spread | AUDUSD Spread | USDCAD Spread | USDCHF Spread | USDJPY Spread | EURCAD Spread | EURAUD Spread | EURJPY Spread | EURCHF Spread |
| Sum of difference | 568 | 1053 | 342 | -3573 | 889 | -338 | 1098 | 402 | 630 | 2809 | 620 | 1290 |
Checking the Historical discrepancy (Excel file) I did a EUR/USD hour by hour difference compare.
The main conclusion I got is that the discrepancy occurs mainly in day shift (at least for EUR/USD), with practically zero or a minimal noise in the remaining hours.
For instance, at 2013.07.29 00:04 spread difference is 64, and so on ...
Coincidence or any conclusion about that?
Checking the Historical discrepancy (Excel file) I did a EUR/USD hour by hour difference compare.
The main conclusion I got is that the discrepancy occurs mainly in day shift (at least for EUR/USD), with practically zero or a minimal noise in the remaining hours.
For instance, at 2013.07.29 00:04 spread difference is 64, and so on ...
Coincidence or any conclusion about that?
Thanks for your interesting observation. I agree that there is certainly discrepancy observed when the day shift happens. So this is one of the problem. However I found also another problem too. That is the historical spread request on real time chart does not allow me to go back as before as 9 October 2013 on M1 chart from Alpari UK broker. Therefore, if I try to access the historical spread before 9 October 2013, then I get strange number filled on my array to hold the historical spread. So that is another problem on this.
For example, I get lots of error message using this code below as the CopyRates method can not access to the M1 data before 9 October 2013.
copied = CopyRates(SymbolCode1,PERIOD_M1, rates_H1[i+1].time, 1, rates_M1);
if(copied > 0)
{
spreadMatrix[i][0] = rates_M1[0].spread;
}
else
{
printf("copy failed at index %d at time %s\n", i, TimeToString(rates_H1[i+1].time));
}
At the same time, when I use back testing, the historical data goes back before 9 October 2013.
So the historical spread I obtained from the real time chart is different from the historical spread I got from Back testing with huge discrepancy.
This is what my investigate tells me so far. So the question is why we can go back to before 9 October 2013 during back testing but why can we not access the historical spread before 9 October 2013 on real time chart?
Is there any way around this? I need to get some good amount of historical spread data from real time chart when the H1 bar is opening.
Kind regards,
Thanks for your interesting observation. I agree that there is certainly discrepancy observed when the day shift happens. So this is one of the problem. However I found also another problem too. That is the historical spread request on real time chart does not allow me to go back as before as 9 October 2013 on M1 chart from Alpari UK broker. Therefore, if I try to access the historical spread before 9 October 2013, then I get strange number filled on my array to hold the historical spread. So that is another problem on this.
FinanceEngineer, in my opinion this is a different issue, that for sure is relevant too, but probably not the same reason (this is my vision).
So, my suggestion is we return to the analysis of the daily problem encountered before start to analyse this new one.
Maybe they are related, but I don't belive so and have a theory to that coincidence I got.
In this sense, I may be wrong, but in my opinion the discrepancy is a bug related to a lag to calc spread after IsNewBarForAllSymbol().
As the code just execute spread=ask-bid after the routine IsNewBarForAllSymbol(), and this routine must wait a new bar for all symbols, the spread is considering the sum of several OHLC minutes until all symbols detect the new bar, and maybe generating the main discrepancy.
FinanceEngineer, in my opinion this is a different issue, that for sure is relevant too, but probably not the same reason (this is my vision).
So, my suggestion is we return to the analysis of the daily problem encountered before start to analyse this new one.
Maybe they are related, but I don't belive so and have a theory to that coincidence I got.
In this sense, I may be wrong, but in my opinion the discrepancy is a bug related to a lag to calc spread after IsNewBarForAllSymbol().
As the code just execute spread=ask-bid after the routine IsNewBarForAllSymbol(), and this routine must wait a new bar for all symbols, the spread is considering the sum of several OHLC minutes until all symbols detect the new bar, and maybe generating the main discrepancy.
Yes, you are absolutely right on this. I was actually able to see your logic. Sure I may be better to change my code to get the spread at the opening of the H1 bar stored in the Mql rate method instead of waiting for all the H1 bar is formed for the 12 pairs and calculating spread from ask -bid.
However my main concern is that when I tried to get the historical spread from real time chart, they are really different from the historical spread during back testing.
For example, executing this code on the real time chart does not provide me any historical spread near the value of the spread used during back testing when I compare each spread using hour by hour comparison. I agree that this can be separate issue from above one. However this is my main concern as I need to get some good amount of historical spread data on real time chart.
string SymbolCode1 = "EURUSD";
string SymbolCode2 = "GBPUSD";
string SymbolCode3 = "EURGBP";
string SymbolCode4 = "NZDUSD";
string SymbolCode5 = "AUDUSD";
string SymbolCode6 = "USDCAD";
string SymbolCode7 = "USDCHF";
string SymbolCode8 = "USDJPY";
string SymbolCode9 = "EURCAD";
string SymbolCode10 = "EURAUD";
string SymbolCode11 = "EURJPY";
string SymbolCode12 = "EURCHF";
int copied;
MqlRates rates_H1[], rates_M1[];
ArraySetAsSeries(rates_H1,false);
ArraySetAsSeries(rates_M1,false);
copied=CopyRates(SymbolCode1,0,0,length,rates_H1);
if(copied>0)
{
for(int i=0;i<length;i++)
{
CopyRates(SymbolCode1,PERIOD_M1,rates_H1[i].time,1,rates_M1);
spreadMatrix[i][0] = rates_M1[0].spread;
}
}
else Print("Failed to get history data for the symbol ",SymbolCode1);
copied=CopyRates(SymbolCode2,0,0,length,rates_H1);
if(copied>0)
{
for(int i=0;i<length;i++)
{
CopyRates(SymbolCode2,PERIOD_M1,rates_H1[i].time,1,rates_M1);
spreadMatrix[i][1] = rates_M1[0].spread;
}
}
else Print("Failed to get history data for the symbol ",SymbolCode2);
copied=CopyRates(SymbolCode3,0,0,length,rates_H1);
if(copied>0)
{
for(int i=0;i<length;i++)
{
CopyRates(SymbolCode3,PERIOD_M1, rates_H1[i].time, 1, rates_M1);
spreadMatrix[i][2] = rates_M1[0].spread;
}
}
else Print("Failed to get history data for the symbol ",SymbolCode3);
Beside the technical or coding challenge on this problem, our finding might suggest that broker's historical data on real time chart and the historical data used during back testing can be quite different.
I am not pointing out the availability of historical data but I am saying the two data are filled with different values with quite notable discrepancy. So this provoke another question like "can we rely on back testing results when the data used on back testing is different from the data on real time chart?"
I am quite concerned to find the truth on this matter.
Yes, you are absolutely right on this. I was actually able to see your logic. Sure I may be better to change my code to get the spread at the opening of the H1 bar stored in the Mql rate method instead of waiting for all the H1 bar is formed for the 12 pairs and calculating spread from ask -bid.
After that, test again using the same methodology.
In this case, maybe the better solution is we use the KISS principle (an acronym for "Keep it simple, stupid", but I prefer "Keep it stupid simple"). So, my suggestion is change the code to a "stupid simple" version, using just one pair (maybe an input parameter to define it), and, in the same sense, try to reduce all kind of possible algorithm that could generate some conceptual error like the one we are talking about.
After that, test again using the same methodology.
- Free trading apps
- Over 8,000 signals for copying
- Economic news for exploring financial markets
You agree to website policy and terms of use
I have found accidently that the historical spread (Ask-bid) used to open and close orders during backtesting in the strategy tseter is different from those stored in the Mql rates.
For example, you can get historical spread using the following code (Lets say your EA is attached on M1 chart).
MqlRates rates[];
ArraySetAsSeries(rates,true);
int copied=CopyRates(Symbol(),0,0,100,rates);
int yourSpread = rates[i].spread;
However the spread obtained using this methods seems different from the spread used to open and close the orders during backtesting using 1 minutes OHLC tick.
I have also found that the spread during backtesting using 1 minutes OHLC price is different from the spread during backtesting using open price only too.
So why they are different? Where the Metatrader 5 getting those historical spread information during backtesting? Are they simulating the spread rather than using historical spread during backtesting?