MT5 : ticks data questions, issues and bugs. All about ticks.

To add comments, please log in or register
Alain Verleyen
37172
Alain Verleyen  

Build 1940/64 bits. Broker A, real account. EURUSD.

On this first point, I am not quite sure if it's a broker issue, an MT5 issue, or maybe even me missing something. Or a mix of all.

Tick volumes 2018

I will take last close month as reference. As you can see we have 2,448,455 ticks in October.

Confirmed on the chart :

EURUSD Monthly

However if you check ticks data, if we get "All ticks" :

We have 6,621,441 ticks !? Ok maybe we need to choose "Bid/Ask ticks" ?

Bid Ask ticks

2,920,555 ticks, closer but not the same.

The difference between "All ticks" and "Bid/Ask ticks" is certainly due to this broker offering a (limited) Depth of Market, so you get a tick on each DOM change. That's to confirm, I will do it in a future post.

I always thought that the "OHLC tick volume" is the real number of ticks, not the number of ticks effectively received by the terminal. But seeing these numbers I started to doubt. If it's real why is there more historical ticks (500,000 is not a small thing). Why this difference ?

For completeness, as expected, there is no "Last/Volume ticks" as it's an OTC market.

I also checked by code and got the same results.

Fernando Carreiro
4948
Fernando Carreiro  
Alain Verleyen:

Build 1940/64 bits. Broker A, real account. EURUSD.

On this first point, I am not quite sure if it's a broker issue, an MT5 issue, or maybe even me missing something. Or a mix of all.

I will take last close month as reference. As you can see we have 2,448,455 ticks in October.

Confirmed on the chart :

However if you check ticks data, if we get "All ticks" :

We have 6,621,441 ticks !? Ok maybe we need to choose "Bid/Ask ticks" ?

2,920,555 ticks, closer but not the same.

The difference between "All ticks" and "Bid/Ask ticks" is certainly due to this broker offering a (limited) Depth of Market, so you get a tick on each DOM change. That's to confirm, I will do it in a future post.

I always thought that the "OHLC tick volume" is the real number of ticks, not the number of ticks effectively received by the terminal. But seeing these numbers I started to doubt. If it's real why is there more historical ticks (500,000 is not a small thing). Why this difference ?

For completeness, as expected, there is no "Last/Volume ticks" as it's an OTC market.

I also checked by code and got the same results.

I am not sure, but could it be that the monthly or OHLC ticks is a "merged" count, that ignore (or remove) duplicate ticks where the bid and ask prices are the same in consecutive ticks?

In other words, Historical ticks contain all the duplicates, while OHLC does not count those. In order to confirm, you would have to run some EA code to count the ticks, detecting duplicates, and seeing if the numbers confirm this.

The reason why I considered this as a possibility is because on MT4, when I generate tick data for back-tests with 3rd party tools, I often choose the option "Ignore Duplicate Ticks" to save on space as well as processing time. So maybe, MT5 or the broker may be doing something similar and not counting the duplicates.

Alain Verleyen
37172
Alain Verleyen  
Fernando Carreiro:

I am not sure, but could it be that the monthly or OHLC ticks is a "merged" count, that ignore (or remove) duplicate ticks where the bid and ask prices are the same in consecutive ticks?

In other words, Historical ticks contain all the duplicates, while OHLC does not count those. In order to confirm, you would have to run some EA code to count the ticks, detecting duplicates, and seeing if the numbers confirm this.

The reason why I considered this as a possibility is because on MT4, when I generate tick data for back-tests with 3rd party tools, I often choose the option "Ignore Duplicate Ticks" to save on space as well as processing time. So maybe, MT5 or the broker may be doing something similar and not counting the duplicates.

Maybe, good idea, I will check that.
Alain Verleyen
37172
Alain Verleyen  
Alain Verleyen:
Maybe, good idea, I will check that.

Done. There was only 10,750 duplicate ticks.

TICKS: PERIOD_MN1 : from 2018.10.01 00:00 to 2018.11.01 00:00, TICKS_BIDASK = 2920555 (OHLC Tick volume = 2448455) Duplicate(Bid/Ask) = 10750

So it's not the explanation.

Mladen Rakic
114135
Mladen Rakic  

A bit of a story (maybe saves some time) :

Couple of years ago, I was messing with some stuff that included tick volume. To my surprise, the stuff started to repaint. In the process to check what the hell is going on I also  have turned on the display of the tick volumes on the chart. To my surprise, and quite accidentally, since it was one long ago closed displayed tick bar that changed a lot and it caught my eye, I have noticed that the past tick bars were changed - not always and usually not much, but more frequently then expected. Even wrote a code that was checking  the already closed bars for tick volume change, and, it happened quite frequently. On demos (multiple brokers tested). And live account(s). It happened most frequently after a very volatile period. Since then I was not messing with the tick volume, but I can imagine that some things remained the same

To cut the long story short : I would not count on accuracy of the tick volume since it seems that the tick volume is managed as the broker decides and how it decides

Fernando Carreiro
4948
Fernando Carreiro  
Alain Verleyen: Done. There was only 10,750 duplicate ticks. So it's not the explanation.

TICKS: PERIOD_MN1 : from 2018.10.01 00:00 to 2018.11.01 00:00, TICKS_BIDASK = 2920555 (OHLC Tick volume = 2448455) Duplicate(Bid/Ask) = 10750

Judging from the code quality of MetaTrader and the numerous bugs it has, that MetaQuotes never gets around to fix, adding more of them with every new release, then I suppose that we can also conclude that on the broker's side, MetaQuotes' server software, also has many bugs that they don't fix or care about, because they don't consider them important.

Arthur Albano
538
Arthur Albano  

So, I tried with WDO$ (mini USD dollar futures  Brazilian BMF Bovespa). Guess what? Different brokers, different database. But no difference from demo and real accounts.

There's also a difference when all ticks are requested (broker M: 11,502,790 vs 13,171,340; broker R: 11,747,636 vs 11,720,021):

Broker R does not store Bid/Ask ticks, but broker M does:

So, my best guess is that some brokers store some information, while others filters data. I am used to operational historian (basically a database server) doing optimization to trash useless information or even condensate them that are not unique. For exemple, if same price arrives with different volumes, it can add the volumes, and use last timestamp. It's called server side compression :P

Alain Verleyen
37172
Alain Verleyen  
Mladen Rakic:

A bit of a story (maybe saves some time) :

Couple of years ago, I was messing with some stuff that included tick volume. To my surprise, the stuff started to repaint. In the process to check what the hell is going on I also  have turned on the display of the tick volumes on the chart. To my surprise, and quite accidentally, since it was one long ago closed displayed tick bar that changed a lot and it caught my eye, I have noticed that the past tick bars were changed - not always and usually not much, but more frequently then expected. Even wrote a code that was checking  the already closed bars for tick volume change, and, it happened quite frequently. On demos (multiple brokers tested). And live account(s). It happened most frequently after a very volatile period. Since then I was not messing with the tick volume, but I can imagine that some things remained the same

To cut the long story short : I would not count on accuracy of the tick volume since it seems that the tick volume is managed as the broker decides and how it decides

Thanks for your input. It was on MT4 I suppose ?

Anyway, it could be the reason but my concern is not about OHLC tick volume, but about real ticks. I want to know if I get all the real historical ticks or not. If the OHLC tick volume was changed I want to know why, as taken into account the above data, I have around 500,000 additional ticks and when you are working on strategy relying on ticks it matters. How can we be sure it's not an MT5 bug ?

Alain Verleyen
37172
Alain Verleyen  
Fernando Carreiro:

Judging from the code quality of MetaTrader and the numerous bugs it has, that MetaQuotes never gets around to fix, adding more of them with every new release, then I suppose that we can also conclude that on the broker's side, MetaQuotes' server software, also has many bugs that they don't fix or care about, because they don't consider them important.

Yes it's also a possibility.  But I am not the one which will accept bugs silently.

Anyway, first thing to do is to determine if it's a bug.

Arthur Albano
538
Arthur Albano  
Alain Verleyen:

Yes it's also a possibility.  But I am not the one which will accept bugs silently.

Anyway, first thing to do is to determine if it's a bug.

I believe there are two methods for calculating ticks/volume: a register and a SQL COUNT(). If data gets veryfied after being in DB, results will differ. Also, a mid-tier server storing information for INSERT INTO will also cause data delays, such as InnoDB, MyISAM and Memory, for MySQL. So SELECT and INSERT are asynchronous. Only the DB manager knows what is going on, I guess...
Alain Verleyen
37172
Alain Verleyen  
Arthur Albano:

So, I tried with WDO$ (mini USD dollar futures  Brazilian BMF Bovespa). Guess what? Different brokers, different database. But no difference from demo and real accounts.

There's also a difference when all ticks are requested (broker M: 11,502,790 vs 13,171,340; broker R: 11,747,636 vs 11,720,021):

Broker R does not store Bid/Ask ticks, but broker M does:

So, my best guess is that some brokers store some information, while others filters data. I am used to operational historian (basically a database server) doing optimization to trash useless information or even condensate them that are not unique. For exemple, if same price arrives with different volumes, it can add the volumes, and use last timestamp. It's called server side compression :P

Thanks for you interesting comparison. It results in even more questions :-D

I am also working on exchange markets data, in details, for example I already reported an issue about ticks with same timestamp (millisecond precision), being mixed up, with some consequence.

What you said about server side compression is useful, and it should be checked, however all brokers used MT5 server, so it would be surprising they behave differently, but who knows, maybe MQ provided an option to activate it. But anyway, here I have MORE ticks than what is reported by OHLC tick volume, so it's rather a data decompression ;-) Seriously, it's a Forex symbol, without real volume, so I don't think it's relevant in this case.

123
To add comments, please log in or register