CopyTicks vs DOM : datas consistency problem

[Giovanni Catti]

I saw some differences between DOM timesandsales and CopyTicks in indicator.
I attach a simple Indicator as example and an image to explain it.

//+------------------------------------------------------------------+
//|                                       OnTicksExampleForForum.mq5 |
//|                        Copyright 2017, MetaQuotes Software Corp. |
//|                                             https://www.mql5.com |
//+------------------------------------------------------------------+
#property copyright ""
#property link      ""
#property version   "1.00"
#property indicator_chart_window
#include <Files\File.mqh>
#include <Files\FileTxt.mqh>
#property indicator_plots 0
CFileTxt file2;
//+------------------------------------------------------------------+
//| Custom indicator initialization function                         |
//+------------------------------------------------------------------+
int OnInit()
  {
//--- indicator buffers mapping
string date = TimeToString(TimeCurrent(),TIME_DATE |TIME_MINUTES);
StringReplace(date,":","");
file2.Open(Symbol()+"TicksArrived_"+date+".txt",FILE_WRITE );   
//---
   return(INIT_SUCCEEDED);
  }
  
  
void OnDeinit(const int reason){
   file2.Close();
}
//+------------------------------------------------------------------+
//| Custom indicator iteration function                              |
//+------------------------------------------------------------------+
int OnCalculate(const int rates_total,
                const int prev_calculated,
                const int begin,
                const double &price[])
  {
//---
  MqlTick tick_array[];
   ResetLastError();
   int copied=CopyTicks(Symbol()
   ,tick_array,COPY_TICKS_TRADE,0,1); 

   
   if(copied<0){
            PrintFormat("%s: Ticks are not synchronized yet, %d ticks received. Error=%d", 
            _Symbol,copied,_LastError); 
        } 
   else {    
   file2.WriteString(GetTickDescription(tick_array[0])+"\n");
   }
//--- return value of prev_calculated for next call
   return(rates_total);
  }
//+------------------------------------------------------------------+

  
string GetTickDescription(MqlTick &tick) 
  { 
   string time=StringFormat("%s.%03d", 
                            TimeToString(tick.time,TIME_DATE|TIME_MINUTES|TIME_SECONDS),tick.time_msc%1000); 
//--- Checking flags 
   bool buy_tick=((tick.flags&TICK_FLAG_BUY)==TICK_FLAG_BUY); 
   bool sell_tick=((tick.flags&TICK_FLAG_SELL)==TICK_FLAG_SELL); 
   bool ask_tick=((tick.flags&TICK_FLAG_ASK)==TICK_FLAG_ASK); 
   bool bid_tick=((tick.flags&TICK_FLAG_BID)==TICK_FLAG_BID); 
   bool last_tick=((tick.flags&TICK_FLAG_LAST)==TICK_FLAG_LAST); 
   bool volume_tick=((tick.flags&TICK_FLAG_VOLUME)==TICK_FLAG_VOLUME); 

   string typeTrade="";
   string postFix=StringFormat(",%G,%G,%G,%d,",+tick.bid,tick.ask,tick.last,tick.volume);


   if(buy_tick || sell_tick) 
     { 
     
      if(buy_tick && sell_tick)
                     {
                      typeTrade = "Buy/Sell"; 
                     }

      else if(sell_tick)
                     {
                        typeTrade = "Sell"; 
                     }
      else  if(buy_tick)
                     {
                     typeTrade = "Buy"; 
                     }
      else
                     {
                      typeTrade = "????"; 
                     }     
     } 
   else 
     { 
     
      if(ask_tick && bid_tick)
                     {
                      typeTrade = "Ask/Bid"; //impossible?!
                     }

      else if(ask_tick)
                     {
                        typeTrade = "Ask"; 
                     }
      else  if(bid_tick)
                     {
                     typeTrade = "Bid"; 
                     }
      else  if(last_tick)
                     {
                     typeTrade = "Last"; 
                     }
      else  if(volume_tick)
                     {
                     typeTrade = "Vol"; 
                     }                                          
      else
                     {
                      typeTrade = "????"; 
                     }       
     } 
//--- Returning tick description 
   return (time+postFix+typeTrade); 
  }   

In the image on the left side there are datas exported from DOM (csv), on the right side there are datas saved by attached indicator using CopyTicks in the OnCalculate event.

I tried to match the common lines in the two file (white color) and highliht the mismatching lines (yellow color).

How we can see in the figure sometimes the DOM seems to loose same datas sometimes, and vice versa sometimes the Indicator seem to loose some other datas.

It's curiose that if we export for example the last 1000 ticks from DOM and we write a script, that runs once, using "CopyTicks" function with the same number ticks, the result is the same as DOM.

Here this second kind of code (the script that save 1000 ticks):

#property copyright                 ""
#property link                      ""
#property version                   ""
#property description               "Times and Sales"
#property indicator_plots 0
#property indicator_chart_window
#include <Files\File.mqh>
#include <Files\FileTxt.mqh>

//save the last _ticks in a file 


input int  _ticks = 1000;          // Number of Requested Ticks
CFileTxt file2;

void OnStart()
  {
//--- requesting ticks
string date = TimeToString(TimeCurrent(),TIME_DATE |TIME_MINUTES);
StringReplace(date,":","");  
string nomefile= Symbol()+"TimeAndSales2"+date+".txt";
file2.Open(nomefile,FILE_WRITE );
file2.Save(file2.Handle());

   MqlTick tick_array[];
   int copied=CopyTicks(Symbol(),tick_array,COPY_TICKS_TRADE,0,_ticks);
   ArraySetAsSeries(tick_array,true);
   string typeTrade;
//---
   if(copied>0)
     {
         for(int i=_INPticks-1;i>=0;i--)
            {
               MqlTick tick   = tick_array[i];   
               bool buy_tick=((tick.flags&TICK_FLAG_BUY)==TICK_FLAG_BUY);           //a tick is a result of a buy deal
               bool sell_tick=((tick.flags&TICK_FLAG_SELL)==TICK_FLAG_SELL);        //a tick is a result of a sell deal
               bool ask_tick=((tick.flags&TICK_FLAG_ASK)==TICK_FLAG_ASK);           //a tick has changed an Ask price
               bool bid_tick=((tick.flags&TICK_FLAG_BID)==TICK_FLAG_BID);           //a tick has changed a Bid price
               bool last_tick=((tick.flags&TICK_FLAG_LAST)==TICK_FLAG_LAST);        //a tick has changed the last deal price
               bool volume_tick=((tick.flags&TICK_FLAG_VOLUME)==TICK_FLAG_VOLUME);  //a tick has changed a volume
               bool between = (buy_tick && sell_tick);          
               if(between)
                     {
                      typeTrade = ",Buy/Sell"; 
                     }
               else if(sell_tick)
                     {
                        typeTrade = ",Sell"; 
                     }
               else  if(buy_tick)
                     {
                     typeTrade = ",Buy"; 
                     }
               else
                     {
                      typeTrade = ",????"; 
                     }
              file2.WriteString(TimeToString(tick_array[i].time,TIME_DATE|TIME_MINUTES|TIME_SECONDS)+"."+StringFormat("%03I64u",tick_array[i].time_msc%1000)+","+StringFormat("%.5d",(int)tick_array[i].bid)+","+StringFormat("%.5d",(int)tick_array[i].ask)+","+StringFormat("%.5d",(int)tick_array[i].last)+","+(string)tick_array[i].volume+typeTrade+"\n");
            }           
 Alert("Created file: "+nomefile);  
            
            ZeroMemory(tick_array);
     }
//---    
   file2.Close();
  }
//+------------------------------------------------------------------+

It seems like the DOM show a statically snapshots of market, but in real market sometimes DOM looses some datas, and sometimes the indicator looses some datas.

My question are:

Why if OnCalculate is called after every arrival of each tick, if I use CopyTicks to get 1 ticks, I get different result vs export of DOM?

Why if in a script if is called Copyticks for multiple ticks the DOM and the script give the same results?

In mql5 documentation (https://www.mql5.com/en/docs/series/copyticks) I read this:
-) In indicators, the OnCalculate() function is called after the arrival of each tick.
-) OnTick() in Expert Advisor is not a handler of every tick, it only notifies an Expert Advisor about changes in the market. It can be a batch of changes: the terminal can simultaneously make a few ticks, but OnTick() will be called only once to notify the EA of the latest market state.

In documentation it's said thath OnTick may join more ticks. I'm afraid that the same thing is done with OnCalculate.
How can we reproduce the "real" tick by tick flow in an Indicator?

I even tried using SymbolInfoTick, but results don't seem get better!

every suggestion is accepted.

[Giovanni Catti]

I found the answer to all my questions here:

https://www.mql5.com/en/articles/3336#i3