Misleading "unmatched data error" message or a bug?

 

During backesting, I'm seeing the following error message:


2009.03.30 23:27:26 TestGenerator: unmatched data error (low value 1.3987 at 2008.12.30 09:05 and price 1.3986 mismatched)


However (as depicted on the attached screenshot) the low value is 1.3986, not 1.3887 as stated above.


There are numerous errors like this, and although I didn't verify each of these, it makes my wonder if the tester is reliable. What is the reason for the above error? I spent a great amount of time in search for a good quality Forex historical data but now I'm confused as there is no way to tell if the data is good.


First, I tried Alpari data and when saw these messages thought that their data is not reliable. Then I obtained historical data from another (supposedly one of the best) free data provider and got the same errors again. This prompted me to manually verify (cross check) the error with the actual data as shown in the history center. My conclusion is that the MT4 tester is broken and cannot be trusted.


If someone can prove me wrong I'd be very grateful.



Misleading "unmatched data error"


Thanks,

Lukasz

 
lukasz74nj:

There are numerous errors like this, and although I didn't verify each of these, it makes my wonder if the tester is reliable. What is the reason for the above error?

This is one of those questions which gets asked over and over again (despite being widely answered both in forum and the documentation). See, for example, 'Mismatched chart error'. Basically, the data for a longer time period such as H1 doesn't fit with the data for a smaller time period - e.g. the high or low implied by the 60 individual M1 bars for an hour doesn't match the high/low for the equivalent H1 bar.


I spent a great amount of time in search for a good quality Forex historical data but now I'm confused as there is no way to tell if the data is good.

You might need to rein in your expectations a little. Forex is an OTC market, and historic data is notoriously broker-dependent. It's also notoriously weak, unless you spend a lot of money.



 

Basically, the data for a longer time period such as H1 doesn't fit with the data for a smaller time period - e.g. the high or low implied by the 60 individual M1 bars for an hour doesn't match the high/low for the equivalent H1 bar.

Ahhhh! But - you have imho stated in a succinct statement what many other posts do not. Thanks.

 
jjc:

This is one of those questions which gets asked over and over again (despite being widely answered both in forum and the documentation). See, for example, 'Mismatched chart error'. Basically, the data for a longer time period such as H1 doesn't fit with the data for a smaller time period - e.g. the high or low implied by the 60 individual M1 bars for an hour doesn't match the high/low for the equivalent H1 bar.


Wait a sec... I'm referring to a M1 data which I imported and is shown above in the history window. Shouldn't be all higher timeframe periods, i.e. M5, M15, etc. calculated properly based on M1 data? How come you can get any discrepancy? I would understand an opposite but not in this case.


I have a valid M1 data, which is not calculated in any way, yet the error improperly states that the low value for is 1.3887, which is really 1.3886. On what basis?

 
lukasz74nj:

I have a valid M1 data, which is not calculated in any way, yet the error improperly states that the low value for is 1.3887, which is really 1.3886. On what basis?

I'd guess that the M1 data doesn't match the tick data. Same principle.

 
jjc:

I'd guess that the M1 data doesn't match the tick data. Same principle.


Well, if your guess is correct, then you just confirmed that the tester is broken.


The tick data is simulated, generated by the tester from M1 data. So if the resulting tick data is incorrect, something must be wrong with the algorithm that produces the tick data.


There is nothing wrong with this particular M1 data entry:


2008.12.30,09:05,1.4002,1.4002,1.3986,1.3989,4


M1 data was imported using the history center, and all higher time frames should be generated automatically. Again, if the problem is with these higher time frames being not in sync with M1 (data of the least time frame don't correspond with data of tested time frame), then there must be a problem with the history center import function. M1 data is just data, all higher time frames should be produced correctly.


All of this results in 15593 mismatched data errors, and "n/a" modeling quality. This really worries me and cast a serious doubt on usefulness of the tester end results. This can potentially turn a winning trade into a loser.

 
lukasz74nj:

Well, if your guess is correct, then you just confirmed that the tester is broken.

Quite possibly. Though it's less clear to me than it seems to be to you what level of reliability and accuracy the Metaquotes developers were aiming for, given the fundamental constraints they're working within. They may have taken the view that fairly marginal inaccuracies in tick simulation were trivial given other considerations such as having to approximate the historic spread - as far as I'm aware MetaTrader only holds the historic bid, and the historic ask must therefore be some sort of extrapolation, probably based on the current spread. As a general rule, I'd expect that any strategy which can't be meaningfully backtested if there are small tick discrepancies (though a couple of yours do look to be quite large) also can't be meaningfully backtested except in an environment which has true historic bid and ask.

 
jjc:

Quite possibly. Though it's less clear to me than it seems to be to you what level of reliability and accuracy the Metaquotes developers were aiming for, given the fundamental constraints they're working within. They may have taken the view that fairly marginal inaccuracies in tick simulation were trivial given other considerations such as having to approximate the historic spread - as far as I'm aware MetaTrader only holds the historic bid, and the historic ask must therefore be some sort of extrapolation, probably based on the current spread. As a general rule, I'd expect that any strategy which can't be meaningfully backtested if there are small tick discrepancies (though a couple of yours do look to be quite large) also can't be meaningfully backtested except in an environment which has true historic bid and ask.


I don't expect 100% accuracy, I would be satisfied with 90% modeling quality. However, such large number of errors is not acceptable to me if it impacts the end result or causes modeling quality to drop to "n/a".


The problem is even more serious than described in this thread. Please check out my other post:


'The same test with the same settings (EA, symbol. etc.) run twice - why is modeling quality different?'