Synch CopyBuffer in EA using iCustom

Stanislav Korotky 2025.11.17 21:45 #21

Dominik Egert # :

Well, cant say anything on the threads being used for the indicators. - But I can say, OnCalculate is being called for every tick received. - We had prooven that. - I dont know when calls will be skipped, might be, if the event queue is full, but as it seems, its a long way until that happens.

https://www.mql5.com/en/forum/454153

Well, here is my part of citations:

Forum on trading, automated trading systems, and trading strategy testing

Indicators are missing ticks on the Exchange

fxsaber , 2016.09.14 15:43

The indicator shows how many ticks the Calculate event skips compared to the History

 long GetTime( void )
{
   MqlTick Tick;

   return ( SymbolInfoTick ( _Symbol , Tick) ? Tick.time_msc : - 1 );
}

int GetAmountTicks( const ulong From )
{
   MqlTick Ticks[];

   const int Amount = CopyTicks ( _Symbol , Ticks, COPY_TICKS_ALL , From);

   int i = - 1 ;

   if ((Amount > 0 ) && (Ticks[ 0 ].time_msc == From) && (Ticks[Amount - 1 ].time_msc != From))
     for (i = 0 ; i < Amount; i++)
       if (Ticks[i].time_msc != From)
         break ;

   return (i);
}

#define INDICATOR // Переключение между индикатором и экспертом

#ifdef INDICATOR
   #define NAME "Indicator: "

   #property indicator_chart_window
   #property indicator_buffers 0
   #property indicator_plots    0

   int OnCalculate ( const int rates_total,
                   const int prev_calculated,
                   const datetime &time[],
                   const double &open[],
                   const double &high[],
                   const double &low[],
                   const double &close[],
                   const long &tick_volume[],
                   const long &volume[],
                   const int &spread[] )
#else
   #define NAME "Expert: "

   void OnTick ( void )
#endif
{
   static long PrevTime = 0 ;
   static int Amount = 0 ;

   const long NowTime = GetTime();

   if (NowTime > PrevTime)
  {
     if (PrevTime > 0 )
    {
       if (Amount > 0 )
      {
         const int HistoryAmount = GetAmountTicks(PrevTime);

         if ((HistoryAmount > 0 ) /* && (Amount != HistoryAmount)*/ )
           Print (NAME + "Ticks with time " + ( string )(( datetime )(PrevTime / 1000 )) + ". " + ( string )(PrevTime % 1000 ) +
                 ": missing " + ( string )(HistoryAmount - Amount) + " (= " + ( string )HistoryAmount + " - " + ( string )Amount + ")." );
      }

      Amount = 1 ;
    }

    PrevTime = NowTime;
  }
   else if (Amount > 0 )
    Amount++;

#ifdef INDICATOR
   return (rates_total);
#endif
}

Result

 2016.09 . 14 16 : 35 : 06.882 Test (Si- 9.16 ,M1)       Indicator: Ticks with time 2016.09 . 14 16 : 34 : 48 . 301 : missing 0 (= 1 - 1 ).
2016.09 . 14 16 : 35 : 06.702 Test (Si- 9.16 ,M1)       Indicator: Ticks with time 2016.09 . 14 16 : 34 : 48 . 83 : missing 0 (= 1 - 1 ).
2016.09 . 14 16 : 35 : 06.236 Test (Si- 9.16 ,M1)       Indicator: Ticks with time 2016.09 . 14 16 : 34 : 47 . 819 : missing 0 (= 1 - 1 ).
2016.09 . 14 16 : 35 : 06.206 Test (Si- 9.16 ,M1)       Indicator: Ticks with time 2016.09 . 14 16 : 34 : 47 . 598 : missing - 16 (= 1 - 17 ) .
2016.09 . 14 16 : 35 : 05.933 Test (Si- 9.16 ,M1)       Indicator: Ticks with time 2016.09 . 14 16 : 34 : 47 . 486 : missing 8 (= 9 - 1 ) .
2016.09 . 14 16 : 35 : 05.478 Test (Si- 9.16 ,M1)       Indicator: Ticks with time 2016.09 . 14 16 : 34 : 47 . 31 : missing 0 (= 1 - 1 ).
2016.09 . 14 16 : 35 : 05.443 Test (Si- 9.16 ,M1)       Indicator: Ticks with time 2016.09 . 14 16 : 34 : 46 . 996 : missing 0 (= 1 - 1 ).
2016.09 . 14 16 : 35 : 05.411 Test (Si- 9.16 ,M1)       Indicator: Ticks with time 2016.09 . 14 16 : 34 : 46 . 989 : missing 0 (= 1 - 1 ).
2016.09 . 14 16 : 35 : 05.381 Test (Si- 9.16 ,M1)       Indicator: Ticks with time 2016.09 . 14 16 : 34 : 46 . 552 : missing 0 (= 1 - 1 ).
2016.09 . 14 16 : 35 : 04.966 Test (Si- 9.16 ,M1)       Indicator: Ticks with time 2016.09 . 14 16 : 34 : 46 . 506 : missing 0 (= 1 - 1 ).
2016.09 . 14 16 : 35 : 04.594 Test (Si- 9.16 ,M1)       Indicator: Ticks with time 2016.09 . 14 16 : 34 : 46 . 165 : missing 0 (= 1 - 1 ).
2016.09 . 14 16 : 35 : 04.557 Test (Si- 9.16 ,M1)       Indicator: Ticks with time 2016.09 . 14 16 : 34 : 46 . 22 : missing 0 (= 1 - 1 ).
2016.09 . 14 16 : 35 : 04.444 Test (Si- 9.16 ,M1)       Indicator: Ticks with time 2016.09 . 14 16 : 34 : 45 . 900 : missing - 1 (= 1 - 2 ) .
2016.09 . 14 16 : 35 : 04.299 Test (Si- 9.16 ,M1)       Indicator: Ticks with time 2016.09 . 14 16 : 34 : 45 . 855 : missing 0 (= 1 - 1 ).
2016.09 . 14 16 : 35 : 04.254 Test (Si- 9.16 ,M1)       Indicator: Ticks with time 2016.09 . 14 16 : 34 : 45 . 553 : missing 0 (= 1 - 1 ).
2016.09 . 14 16 : 35 : 03.962 Test (Si- 9.16 ,M1)       Indicator: Ticks with time 2016.09 . 14 16 : 34 : 44 . 555 : missing 0 (= 1 - 1 ).
2016.09 . 14 16 : 35 : 02.974 Test (Si- 9.16 ,M1)       Indicator: Ticks with time 2016.09 . 14 16 : 34 : 43 . 904 : missing 0 (= 1 - 1 ).
2016.09 . 14 16 : 35 : 02.304 Test (Si- 9.16 ,M1)       Indicator: Ticks with time 2016.09 . 14 16 : 34 : 43 . 878 : missing 0 (= 1 - 1 ).
2016.09 . 14 16 : 35 : 02.212 Test (Si- 9.16 ,M1)       Indicator: Ticks with time 2016.09 . 14 16 : 34 : 43 . 439 : missing 0 (= 1 - 1 ).
2016.09 . 14 16 : 35 : 01.529 Test (Si- 9.16 ,M1)       Indicator: Ticks with time 2016.09 . 14 16 : 34 : 43 . 104 : missing 0 (= 1 - 1 ).
2016.09 . 14 16 : 35 : 01.399 Test (Si- 9.16 ,M1)       Indicator: Ticks with time 2016.09 . 14 16 : 34 : 42 . 960 : missing 0 (= 1 - 1 ).
2016.09 . 14 16 : 35 : 01.127 Test (Si- 9.16 ,M1)       Indicator: Ticks with time 2016.09 . 14 16 : 34 : 42 . 697 : missing 0 (= 1 - 1 ).
2016.09 . 14 16 : 35 : 00.879 Test (Si- 9.16 ,M1)       Indicator: Ticks with time 2016.09 . 14 16 : 34 : 42 . 455 : missing 0 (= 1 - 1 ).
2016.09 . 14 16 : 35 : 00.697 Test (Si- 9.16 ,M1)       Indicator: Ticks with time 2016.09 . 14 16 : 34 : 42 . 270 : missing 25 (= 26 - 1 ) .

The same thing, of course, happens with experts (the Tick event). While Forex indicators don't miss ticks (they didn't catch them), expert advisors do.

Please note that there are situations where the tick history (with an accuracy of 1ms) contains fewer entries with the same time than the indicator calls. In this case, the indicator was called 17 times with identical ticks, but there is only one such tick in the history.

The discovered bug, when CopyTicks and Event are not synchronized on the last tick , has been taken into account.

Forum on trading, automated trading systems, and trading strategy testing

Indicators are missing ticks on the Exchange

fxsaber , 2016.09.20 10:28

Script

 #property script_show_inputs

sinput int Count = 100000 ; // Количество тиков на анализ

int GetMaxQueue( const MqlTick &Ticks[], const int ExecutionTime )
{
   int MaxQueue = 0 ;
   long NextTime = 0 ;
   const int Amount = ArraySize (Ticks);

   for ( int i = 0 ; i < Amount; i++)
  {
     const long NowTime = Ticks[i].time_msc;
     const int Queue = ( int )(NextTime - NowTime);

     if (Queue > 0 )
    {
       if (Queue > MaxQueue)
        MaxQueue = Queue;
    }
     else
      NextTime = NowTime;

    NextTime += ExecutionTime;
  }

   return (MaxQueue);
}

void OnStart ( void )
{
   MqlTick Ticks[];

   CopyTicks ( _Symbol , Ticks, COPY_TICKS_ALL , 0 , Count);

   for ( int i = 0 ; i < 100 ; i++)
     Print ( "ExecutionTime = " + ( string )i + " ms., MaxQueue = " + ( string )GetMaxQueue(Ticks, i) + " ms." );
}

Result

 2016.09 . 20 12 : 09 : 19.714 Test (RTS- 12.16 ,M1)     ExecutionTime = 10 ms., MaxQueue = 9778 ms.
2016.09 . 20 12 : 09 : 19.714 Test (RTS- 12.16 ,M1)     ExecutionTime = 9 ms., MaxQueue = 8525 ms.
2016.09 . 20 12 : 09 : 19.714 Test (RTS- 12.16 ,M1)     ExecutionTime = 8 ms., MaxQueue = 7273 ms.
2016.09 . 20 12 : 09 : 19.714 Test (RTS- 12.16 ,M1)     ExecutionTime = 7 ms., MaxQueue = 6040 ms.
2016.09 . 20 12 : 09 : 19.711 Test (RTS- 12.16 ,M1)     ExecutionTime = 6 ms., MaxQueue = 4807 ms.
2016.09 . 20 12 : 09 : 19.711 Test (RTS- 12.16 ,M1)     ExecutionTime = 5 ms., MaxQueue = 3575 ms.
2016.09 . 20 12 : 09 : 19.711 Test (RTS- 12.16 ,M1)     ExecutionTime = 4 ms., MaxQueue = 2361 ms.
2016.09 . 20 12 : 09 : 19.711 Test (RTS- 12.16 ,M1)     ExecutionTime = 3 ms., MaxQueue = 1464 ms.
2016.09 . 20 12 : 09 : 19.711 Test (RTS- 12.16 ,M1)     ExecutionTime = 2 ms., MaxQueue = 820 ms.
2016.09 . 20 12 : 09 : 19.711 Test (RTS- 12.16 ,M1)     ExecutionTime = 1 ms., MaxQueue = 325 ms.
2016.09 . 20 12 : 09 : 19.711 Test (RTS- 12.16 ,M1)     ExecutionTime = 0 ms., MaxQueue = 0 ms.

More details at the beginning of the chart

If an indicator takes 1 ms to execute, it can lag by several hundred ms. Sometimes it can lag by 5 seconds even if OnCalculate takes 6 ms to execute. And if OnCalculate takes as long as 50 ms to calculate, the lag can exceed a minute! And the lag depends greatly on the instrument. The higher the tick frequency, the higher the indicator lag.

Clearly, there can be no talk of any execution of indicators on every tick (like, indicators should not skip ticks)!

Nevertheless, I think that even if ticks can be skipped by EAs and (most likely) indicators, the queueing principle should make it impossible for EA to get called on other tick than most recently re-calculated by indicator.

Testing indicators Real and Generated Ticks Trading Principles - Trade

Alain Verleyen 2025.11.17 22:20 #22

Guys, this topic is about CopyBuffer() and iCustom call, given the platform as IT IS.

If you want to talk about how it should be or other platform "mechanics" please open an other topic.

Dominik Egert 2025.11.18 08:14 #23

Alain Verleyen #:

Guys, this topic is about CopyBuffer() and iCustom call, given the platform as IT IS.

If you want to talk about how it should be or other platform "mechanics" please open an other topic.

Yeah, these parts somehow are linked to some degree.

OnCalc being called for every tick is not part of this discussion. But, look at the results of my code, it bleeds into this topic to some degree.

I guess, we have referenced enough about execution of OnCalc so far and should focus on the topic.

Is CopyBuffer consistent. As far as I can tell, it seems so. I wasn't able to capture an intermediate state of the buffers.

The mechanics for how and why this is the case could be rooted in another topic though.

Statements Using ready-made indicators from Interactive events on charts

Stanislav Korotky 2025.11.18 13:40 #24

Alain Verleyen #:

Guys, this topic is about CopyBuffer() and iCustom call, given the platform as IT IS.

If you want to talk about how it should be or other platform "mechanics" please open an other topic.

We are all just do not know how IT IS behind the scene, without clarifications from MQ. This is why our last resort is the collective investigation of the matter from different aspects.

And synchronization of different threads (from where API functions are called and events are fired) is apparently related to queueing technique used in the terminal. And queueing can be (currently) investigated by ticks/events flow only. So the posts are on topic, imho.

Turning timer on and The main function of Depth of Market

Alain Verleyen 2025.11.18 16:24 #25

@Dominik Egert @Stanislav Korotky

Ok.

The conclusion is already clear : CopyBuffer (called once or several times for a given handle) is thread safe : the EA thread and the indicator thread are synchronized "somehow".

What is unclear is the mechanism used. So we can investigate that together, but please stay on topic.

Later I will open a more general topic on the MT5 platform mechanics in general.

Architectural solutions for improving Interactive events on charts Task Manager - For

Alain Verleyen 2025.11.18 16:26 #26

Stanislav Korotky #:

We are all just do not know how IT IS behind the scene, without clarifications from MQ. This is why our last resort is the collective investigation of the matter from different aspects.

And synchronization of different threads (from where API functions are called and events are fired) is apparently related to queueing technique used in the terminal. And queueing can be (currently) investigated by ticks/events flow only. So the posts are on topic, imho.

Ok maybe but then please provide data, logs, code, etc...

I don't really see the pertinence of your links to fxsaber 2016 posts.

Stanislav Korotky 2025.11.19 16:18 #27

Alain Verleyen #:

The conclusion is already clear : CopyBuffer (called once or several times for a given handle) is thread safe : the EA thread and the indicator thread are synchronized "somehow".

What is unclear is the mechanism used. So we can investigate that together

I'm more or less convinced that the single call to CopyBuffer is synchronized with the indicator referenced by handle, but I doubt that the synchronization persists between numerous calls.

Alain Verleyen #:

Ok maybe but then please provide data, logs, code, etc...

If you mean some info about the queues, then as a user I don't know anything about internals - only the fact that the chart and every interactive MQL5-program have their queues of events (including ticks), and this is a part of the synchronization mechanism, because otherwise any thread would have to wait/freeze a lot. Possibility of skipping ticks are related to the queueing, and I don't know at the moment at which degree it's related or not related to our research.

My test codes are similar to what has been already published. I can't yet trap a moment when buffer reading would be inconsistent, but this in itself does not guarantee that such problem can not occur at all.

Time management in the Interactive events on charts Turning timer on and

antony23 2025.12.13 01:07 #28

@Alain Verleyen

Hi Alain, do you have any news about developers?

I also couldn't get an incosistent buffer reading at the moment, in the tests i did. But a confirm by MQ would be appreciated, if not due.

MQL5 Wizard: Development of Economic calendar Types of Execution -

Synch CopyBuffer in EA using iCustom - page 3