MT5 : ticks data questions, issues and bugs. All about ticks.
Build 1940/64 bits. Broker A, real account. EURUSD.
On this first point, I am not quite sure if it's a broker issue, an MT5 issue, or maybe even me missing something. Or a mix of all.
I will take last close month as reference. As you can see we have 2,448,455 ticks in October.
Confirmed on the chart :
However if you check ticks data, if we get "All ticks" :
We have 6,621,441 ticks !? Ok maybe we need to choose "Bid/Ask ticks" ?
2,920,555 ticks, closer but not the same.
The difference between "All ticks" and "Bid/Ask ticks" is certainly
due to this broker offering a (limited) Depth of Market, so you get a
tick on each DOM change. That's to confirm, I will do it in a future
post.
I always thought that the "OHLC tick volume" is the real number of ticks, not the number of ticks effectively received by the terminal. But seeing these numbers I started to doubt. If it's real why is there more historical ticks (500,000 is not a small thing). Why this difference ?
For completeness, as expected, there is no "Last/Volume ticks" as it's an OTC market.
I also checked by code and got the same results.
I am not sure, but could it be that the monthly or OHLC ticks is a "merged" count, that ignore (or remove) duplicate ticks where the bid and ask prices are the same in consecutive ticks?
In other words, Historical ticks contain all the duplicates, while OHLC does not count those. In order to confirm, you would have to run some EA code to count the ticks, detecting duplicates, and seeing if the numbers confirm this.
The reason why I considered this as a possibility is because on MT4, when I generate tick data for back-tests with 3rd party tools, I often choose the option "Ignore Duplicate Ticks" to save on space as well as processing time. So maybe, MT5 or the broker may be doing something similar and not counting the duplicates.
I am not sure, but could it be that the monthly or OHLC ticks is a "merged" count, that ignore (or remove) duplicate ticks where the bid and ask prices are the same in consecutive ticks?
In other words, Historical ticks contain all the duplicates, while OHLC does not count those. In order to confirm, you would have to run some EA code to count the ticks, detecting duplicates, and seeing if the numbers confirm this.
The reason why I considered this as a possibility is because on MT4, when I generate tick data for back-tests with 3rd party tools, I often choose the option "Ignore Duplicate Ticks" to save on space as well as processing time. So maybe, MT5 or the broker may be doing something similar and not counting the duplicates.
Maybe, good idea, I will check that.
Done. There was only 10,750 duplicate ticks.
TICKS: PERIOD_MN1 : from 2018.10.01 00:00 to 2018.11.01 00:00, TICKS_BIDASK = 2920555 (OHLC Tick volume = 2448455) Duplicate(Bid/Ask) = 10750
So it's not the explanation.
A bit of a story (maybe saves some time) :
Couple of years ago, I was messing with some stuff that included tick volume. To my surprise, the stuff started to repaint. In the process to check what the hell is going on I also have turned on the display of the tick volumes on the chart. To my surprise, and quite accidentally, since it was one long ago closed displayed tick bar that changed a lot and it caught my eye, I have noticed that the past tick bars were changed - not always and usually not much, but more frequently then expected. Even wrote a code that was checking the already closed bars for tick volume change, and, it happened quite frequently. On demos (multiple brokers tested). And live account(s). It happened most frequently after a very volatile period. Since then I was not messing with the tick volume, but I can imagine that some things remained the same
To cut the long story short : I would not count on accuracy of the tick volume since it seems that the tick volume is managed as the broker decides and how it decides
TICKS: PERIOD_MN1 : from 2018.10.01 00:00 to 2018.11.01 00:00, TICKS_BIDASK = 2920555 (OHLC Tick volume = 2448455) Duplicate(Bid/Ask) = 10750
Judging from the code quality of MetaTrader and the numerous bugs it has, that MetaQuotes never gets around to fix, adding more of them with every new release, then I suppose that we can also conclude that on the broker's side, MetaQuotes' server software, also has many bugs that they don't fix or care about, because they don't consider them important.
So, I tried with WDO$ (mini USD dollar futures Brazilian BMF Bovespa). Guess what? Different brokers, different database. But no difference from demo and real accounts.
There's also a difference when all ticks are requested (broker M: 11,502,790 vs 13,171,340; broker R: 11,747,636 vs 11,720,021):
Broker R does not store Bid/Ask ticks, but broker M does:
So, my best guess is that some brokers store some information, while others filters data. I am used to operational historian (basically a database server) doing optimization to trash useless information or even condensate them that are not unique. For exemple, if same price arrives with different volumes, it can add the volumes, and use last timestamp. It's called server side compression :P
A bit of a story (maybe saves some time) :
Couple of years ago, I was messing with some stuff that included tick volume. To my surprise, the stuff started to repaint. In the process to check what the hell is going on I also have turned on the display of the tick volumes on the chart. To my surprise, and quite accidentally, since it was one long ago closed displayed tick bar that changed a lot and it caught my eye, I have noticed that the past tick bars were changed - not always and usually not much, but more frequently then expected. Even wrote a code that was checking the already closed bars for tick volume change, and, it happened quite frequently. On demos (multiple brokers tested). And live account(s). It happened most frequently after a very volatile period. Since then I was not messing with the tick volume, but I can imagine that some things remained the same
To cut the long story short : I would not count on accuracy of the tick volume since it seems that the tick volume is managed as the broker decides and how it decides
Thanks for your input. It was on MT4 I suppose ?
Anyway, it could be the reason but my concern is not about OHLC tick volume, but about real ticks. I want to know if I get all the real historical ticks or not. If the OHLC tick volume was changed I want to know why, as taken into account the above data, I have around 500,000 additional ticks and when you are working on strategy relying on ticks it matters. How can we be sure it's not an MT5 bug ?
Judging from the code quality of MetaTrader and the numerous bugs it has, that MetaQuotes never gets around to fix, adding more of them with every new release, then I suppose that we can also conclude that on the broker's side, MetaQuotes' server software, also has many bugs that they don't fix or care about, because they don't consider them important.
Yes it's also a possibility. But I am not the one which will accept bugs silently.
Anyway, first thing to do is to determine if it's a bug.
Yes it's also a possibility. But I am not the one which will accept bugs silently.
Anyway, first thing to do is to determine if it's a bug.
So, I tried with WDO$ (mini USD dollar futures Brazilian BMF Bovespa). Guess what? Different brokers, different database. But no difference from demo and real accounts.
There's also a difference when all ticks are requested (broker M: 11,502,790 vs 13,171,340; broker R: 11,747,636 vs 11,720,021):
Broker R does not store Bid/Ask ticks, but broker M does:
So, my best guess is that some brokers store some information, while others filters data. I am used to operational historian (basically a database server) doing optimization to trash useless information or even condensate them that are not unique. For exemple, if same price arrives with different volumes, it can add the volumes, and use last timestamp. It's called server side compression :P
Thanks for you interesting comparison. It results in even more questions :-D
I am also working on exchange markets data, in details, for example I already reported an issue about ticks with same timestamp (millisecond precision), being mixed up, with some consequence.
What you said about server side compression is useful, and it should be checked, however all brokers used MT5 server, so it would be surprising they behave differently, but who knows, maybe MQ provided an option to activate it. But anyway, here I have MORE ticks than what is reported by OHLC tick volume, so it's rather a data decompression ;-) Seriously, it's a Forex symbol, without real volume, so I don't think it's relevant in this case.
- Free trading apps
- Over 8,000 signals for copying
- Economic news for exploring financial markets
You agree to website policy and terms of use
Build 1940/64 bits. Broker A, real account. EURUSD.
On this first point, I am not quite sure if it's a broker issue, an MT5 issue, or maybe even me missing something. Or a mix of all.
I will take last close month as reference. As you can see we have 2,448,455 ticks in October.
Confirmed on the chart :
However if you check ticks data, if we get "All ticks" :
We have 6,621,441 ticks !? Ok maybe we need to choose "Bid/Ask ticks" ?
2,920,555 ticks, closer but not the same.
The difference between "All ticks" and "Bid/Ask ticks" is certainly due to this broker offering a (limited) Depth of Market, so you get a tick on each DOM change. That's to confirm, I will do it in a future post.
I always thought that the "OHLC tick volume" is the real number of ticks, not the number of ticks effectively received by the terminal. But seeing these numbers I started to doubt. If it's real why is there more historical ticks (500,000 is not a small thing). Why this difference ?
For completeness, as expected, there is no "Last/Volume ticks" as it's an OTC market.
I also checked by code and got the same results.