Why is column "Flags" no longer present in the CSV tick data file exported from MT5 ?

 

Hello all !

I have two machines with different MT5 builds installed.

The oldest one is build 1643, downloaded from my brokerage account at ActivTrades.

The newest one is buid 1730, downloaded directly from MetaQuotes.

When trying to export the tick data file from the newest MT5 instance, I noticed that the last column (Flags) is no longer available.

This is a problem for me because I use machine learning techniques for scalping that require tons of historical tick data for training the model, and the information in the "Flags" column is necessary to know the aggressor side (buyer or seller) of each trade, in order to compute the direction and the intensity of the order flow, which are two important predictor variables that compose the model.   

Does anyone know why the "Flags" column was discarded in the newer version? Am I doing something wrong? 

I attached some images below. Notice the "Flags" column present in the old version export, and absent in the new version. The images were captured on the same day, when exporting the same tick data period, from the same active, on the same brokerage account. The only difference is the MT5 version.


The "About" screen from the old MT5 version:

Data exported from the old MT5 version: Tick data exported from MT Version 5.00 Build 1643

The "About" screen from the new MT5 version:     

Data exported from the new MT5 version: 


The same problem (lack of the last column "Flags") also occurs in the exported CSV file (not only in the viewer window).

Thank you in advance!

Heraldo.

 
Because these flags are generated by the terminal itself.
 
fxsaber: Because these flags are generated by the terminal itself.

That makes absolutely no sense!

Where is the official MetaQuotes explanation or documentation on the matter in order to inform users of how such functionality is implemented?

 
Fernando Carreiro:

That makes absolutely no sense!

Where is the official MetaQuotes explanation or documentation on the matter in order to inform users of how such functionality is implemented?

There are practices, but there are theorists. Less read the documentation, more programming.

 
fxsaber: There are practices, but there are theorists. Less read the documentation, more programming.

Is that some kind of spiritual mantra for coders? Again, your remarks offers no real answer to the OP's query! Traders and coders deal and facts and figures - not philosophical ideals!

Even if its true that the flags are "generated" by the software and even if after analysing that data we can maybe figure out how it is to be represented - the fact is still that it has not been explained or documented, or at least no one has come forth and provided such a link.

This is especially a need for the Traders that don't necessarily code. Obviously coders can take some of the knowledge described in the MqlTick structure and indirectly "reverse engineer" how it is possibly "generating" the flags, but a non-coder will not be able to do that, and MetaQuotes should at least provide some documentation for it.

So, since no one else informed on this, is actually providing an real answer for the OP, I will attempt to do so based on deduction and intuition:

@HeraldoAlmeida, please consider the following answer only as my attempt at trying to decipher the results and not actually an official answer.

From your screenshot, of the newer version, it seems that only a few fields get filled in and the rest remain blank. So, I suspect the following:

  • When only the "Bid" field gets filled in, then the equivalent flag would be "Bid"
  • When only the "Ask" field gets filled in, then the equivalent flag would be "Ask"
  • When only the "Last" and "Volume" fields get filled in, then it will be the respective "Buy" or "Sell" volume, depending on whether the most recent price quote was either a "Bid" or an "Ask".

There is probably more nuances to this, and since you use this functionality you will probably be able to detect them better than I can. You can also have a look at the MqlTick structure to see more details that might help you decipher it further.

My apologies for not being able to offer you more assistance, but unfortunately I have no experience with it. Those that do have more experience have opted to be more cryptic than to actually answer you properly.

I hope that has helped you with the problem.

 

Only one way to settle this! OP needs to backup his ActvTrades terminal folder, login to a MQ demo from said terminal, upgrade the terminal, and report back whether or not the tick flag was removed by the terminal or whether the field is broker specific. 

 
nicholishen: Only one way to settle this! OP needs to backup his ActvTrades terminal folder, login to a MQ demo from said terminal, upgrade the terminal, and report back whether or not the tick flag was removed by the terminal or whether the field is broker specific. 

There is no need to do that! It is like this since Build 1700. You just need to go to the Symbols and confirm it yourself! In fact, it is even in the screenshot given by the MetaQuotes threads:


New MetaTrader 5 Build 1730: Projects in MetaEditor and Synthetic financial instruments
New MetaTrader 5 Build 1730: Projects in MetaEditor and Synthetic financial instruments
  • 2017.12.20
  • www.mql5.com
New MetaTrader 5 Build 1730: Projects in MetaEditor and Synthetic financial instruments The MetaTrader 5 platform update will be released on Decemb...
 
fxsaber:

There are practices, but there are theorists. Less read the documentation, more programming.

You are ready to be hired by Metaquotes. 

 
nicholishen:

Only one way to settle this! OP needs to backup his ActvTrades terminal folder, login to a MQ demo from said terminal, upgrade the terminal, and report back whether or not the tick flag was removed by the terminal or whether the field is broker specific. 

As Fernando said, and the OP, the flag is removed, that's a fact.

I totally agree with Fernando. No documentation, no communication, and fxsaber answer doesn't make sense. 

These flags come from the trading server, there is no way to reproduce them all on the terminal side. 

 
fxsaber:
Because these flags are generated by the terminal itself.

I believe these flags are generated at the MT5 Server, by means of some kind of heuristics like the one @FernandoCarreiro described. I've observed that, at least in the Brazilian stock exchange (BM&F-BOVESPA), the contents of the column "Flags" are different depending on the account/broker I'm connected to (I'm too lazy to repeat the tests and upload images to prove it, you are free to believe me or not).

If the flags were generated at the terminal, as well as if they came from the stock exchange, they should not vary from broker to broker, since in this case both the stock exchange and the terminal are the same, only the broker's MT5 Server is different. 

Anyway, my question does not depend on where the flags come from.

I just wanted to know whether the removal the flags is a bug or a "decision" from the developers, and how I should do to report the bug, or to complain about the decision.


 
Fernando Carreiro:

@HeraldoAlmeida, please consider the following answer only as my attempt at trying to decipher the results and not actually an official answer.

From your screenshot, of the newer version, it seems that only a few fields get filled in and the rest remain blank. So, I suspect the following:

  • When only the "Bid" field gets filled in, then the equivalent flag would be "Bid"
  • When only the "Ask" field gets filled in, then the equivalent flag would be "Ask"
  • When only the "Last" and "Volume" fields get filled in, then it will be the respective "Buy" or "Sell" volume, depending on whether the most recent price quote was either a "Bid" or an "Ask".

There is probably more nuances to this, and since you use this functionality you will probably be able to detect them better than I can. You can also have a look at the MqlTick structure to see more details that might help you decipher it further.

My apologies for not being able to offer you more assistance, but unfortunately I have no experience with it. Those that do have more experience have opted to be more cryptic than to actually answer you properly.

I hope that has helped you with the problem.


Thank you for your answer. The 3 rules you enumerated fits perfectly well all the tick data I have. Unfortunately it does not help me much, because in the way I analyze the flags I don't care about bid/ask ticks, I'm interested only on trade (buy and sell) ticks, and the information that really matters for me is whether the aggressor side (the one who sent the market order) was the buyer or the seller, in order to estimate the direction and intensity of the flow at every moment.

There are some well-known heuristic rules to infer which side was the aggressor:

1) For ticks where last == ask, the aggressor was "certainly" the buyer

2) For ticks where last == bid, the aggressor was "certainly" the seller

3) For ticks where bid < last < ask, then it is impossible to know the aggressor for sure, but it is (statistically) more likely to be the buyer or the seller depending on what price changes first in the sequence (bid or ask) and to which direction.    

ps: "certainly" here means "as long there are no missing ticks, the ticks are processed in the right order, etc."  

I believe that (before the disappearance of column Flags) the MT5 server employed some kind of heuristics like that to infer the buy/sell flags. 

Reason: