Services , new functionality in the MT5 architech, the funeral of MT4 is just around the corner. - page 8

 

from Eng.

data

feed

datafeed -- data source (although the word "source" is not quite right)


feeder -- feeder, feeder, feeder

 

О датафидах. Сколько встречал этот термин, сложилось ощущение, что этим словом обозначают источник данных (котировок). Буквально "заполнитель данных". Когда речь идет о собственных инструментах, понимаю так, что мы сможем вычислять котировки, например, никем не котируемого MXNRUB по известным курсам MXNUSD и USDRUB, выгруженным из терминала в .csv формат, и легализовать для терминала новые котировки, указав файл .csv как новый датафид. Возможно, будет реализовано и более изящное решение, без выгрузки в файлы, путем онлайновых операций */ над тиками (MXNRUB =  MXNUSD * USDRUB). И это будет новый датафид.

Have they already imported .csv quotes in ticks for MT5???
 
Andrey Khatimlianskii:

No link left to a discussion, bug report or anything like that? Or just a feeling that there was one, but haven't checked it now?

As far as I remember, the collected ticks in OnCalculate matched the ones requested via CopyTix.

Now purposely checked it... as I suspected, got some crap, but not the one I was expecting :)

2017.05.22 12:47:10.591 (RTS-6.17,M1)   OnCalculate: 2017.05.22 10:47:11.27 ms v = 1, bid = 108320, ask = 108330, last = 108330, flags = 4
2017.05.22 12:47:10.591 (RTS-6.17,M1)   OnBookEvent: 2017.05.22 10:47:11.27 ms v = 1, bid = 108320, ask = 108330, last = 108330, flags = 4
2017.05.22 12:47:10.599 (RTS-6.17,M1)   OnBookEvent: 2017.05.22 10:47:11.27 ms v = 1, bid = 108320, ask = 108330, last = 108330, flags = 4
2017.05.22 12:47:10.605 (RTS-6.17,M1)   OnBookEvent: 2017.05.22 10:47:11.27 ms v = 1, bid = 108320, ask = 108330, last = 108330, flags = 4
2017.05.22 12:47:10.617 (RTS-6.17,M1)   OnBookEvent: 2017.05.22 10:47:11.27 ms v = 1, bid = 108320, ask = 108330, last = 108330, flags = 4
2017.05.22 12:47:10.637 (RTS-6.17,M1)   OnBookEvent: 2017.05.22 10:47:11.27 ms v = 1, bid = 108320, ask = 108330, last = 108330, flags = 4
2017.05.22 12:47:10.649 (RTS-6.17,M1)   OnBookEvent: 2017.05.22 10:47:11.27 ms v = 1, bid = 108320, ask = 108330, last = 108330, flags = 4
2017.05.22 12:47:10.683 (RTS-6.17,M1)   OnBookEvent: 2017.05.22 10:47:11.27 ms v = 1, bid = 108320, ask = 108330, last = 108330, flags = 4
2017.05.22 12:47:10.765 (RTS-6.17,M1)   OnBookEvent: 2017.05.22 10:47:11.27 ms v = 1, bid = 108320, ask = 108330, last = 108330, flags = 4
2017.05.22 12:47:10.771 (RTS-6.17,M1)   OnCalculate: 2017.05.22 10:47:11.193 ms v = 1, bid = 108320, ask = 108330, last = 108320, flags = 88
2017.05.22 12:47:10.781 (RTS-6.17,M1)   OnBookEvent: 2017.05.22 10:47:11.193 ms v = 1, bid = 108320, ask = 108330, last = 108320, flags = 88
2017.05.22 12:47:10.843 (RTS-6.17,M1)   OnBookEvent: 2017.05.22 10:47:11.244 ms v = 1, bid = 108320, ask = 108330, last = 108320, flags = 0
2017.05.22 12:47:10.843 (RTS-6.17,M1)   OnBookEvent: Время последнего OnCalculate = 2017.05.22 10:47:11.193 ms (88), время последнего OnBookEvent = 2017.05.22 10:47:11.244 ms (0)
2017.05.22 12:47:10.859 (RTS-6.17,M1)   OnCalculate: 2017.05.22 10:47:11.282 ms v = 1, bid = 108320, ask = 108330, last = 108320, flags = 88
2017.05.22 12:47:10.863 (RTS-6.17,M1)   OnBookEvent: 2017.05.22 10:47:11.282 ms v = 1, bid = 108320, ask = 108330, last = 108320, flags = 88

OnBookEvent() catches ticks with flag 0...

 

And this is what I expected to see:

2017.05.22 13:24:38.242 (RTS-6.17,M1)   OnCalculate: 2017.05.22 11:24:38.689 ms v = 2, bid = 108540, ask = 108560, last = 108550, flags = 'TICK_FLAG_BUY TICK_FLAG_LAST TICK_FLAG_VOLUME '
2017.05.22 13:24:38.242 (RTS-6.17,M1)   OnCalculate: 2017.05.22 11:24:38.689 ms v = 2, bid = 108540, ask = 108560, last = 108550, flags = 'TICK_FLAG_BUY TICK_FLAG_LAST TICK_FLAG_VOLUME '
2017.05.22 13:24:38.242 (RTS-6.17,M1)   OnCalculate: 2017.05.22 11:24:38.689 ms v = 2, bid = 108540, ask = 108560, last = 108550, flags = 'TICK_FLAG_BUY TICK_FLAG_LAST TICK_FLAG_VOLUME '
2017.05.22 13:24:38.242 (RTS-6.17,M1)   OnCalculate: 2017.05.22 11:24:38.689 ms v = 2, bid = 108540, ask = 108560, last = 108550, flags = 'TICK_FLAG_BUY TICK_FLAG_LAST TICK_FLAG_VOLUME '
2017.05.22 13:24:38.242 (RTS-6.17,M1)   OnBookEvent: 2017.05.22 11:24:38.689 ms v = 2, bid = 108540, ask = 108560, last = 108550, flags = 'TICK_FLAG_BUY TICK_FLAG_LAST TICK_FLAG_VOLUME '
2017.05.22 13:24:38.253 (RTS-6.17,M1)   OnCalculate: 2017.05.22 11:24:38.704 ms v = 2, bid = 108550, ask = 108560, last = 108550, flags = 'TICK_FLAG_SELL TICK_FLAG_LAST TICK_FLAG_VOLUME '
2017.05.22 13:24:38.253 (RTS-6.17,M1)   OnCalculate: Время последнего STick = 2017.05.22 11:24:38.704 ms (TICK_FLAG_SELL TICK_FLAG_LAST TICK_FLAG_VOLUME ), время последнего CopyTicks = 2017.05.22 11:24:38.697 ms (TICK_FLAG_SELL TICK_FLAG_LAST TICK_FLAG_VOLUME )
2017.05.22 13:24:38.253 (RTS-6.17,M1)   OnCalculate: 2017.05.22 11:24:38.704 ms v = 2, bid = 108550, ask = 108560, last = 108550, flags = 'TICK_FLAG_SELL TICK_FLAG_LAST TICK_FLAG_VOLUME '
2017.05.22 13:24:38.253 (RTS-6.17,M1)   OnCalculate: Время последнего STick = 2017.05.22 11:24:38.704 ms (TICK_FLAG_SELL TICK_FLAG_LAST TICK_FLAG_VOLUME ), время последнего CopyTicks = 2017.05.22 11:24:38.697 ms (TICK_FLAG_SELL TICK_FLAG_LAST TICK_FLAG_VOLUME )
2017.05.22 13:24:38.254 (RTS-6.17,M1)   OnCalculate: 2017.05.22 11:24:38.704 ms v = 2, bid = 108550, ask = 108560, last = 108550, flags = 'TICK_FLAG_SELL TICK_FLAG_LAST TICK_FLAG_VOLUME '
2017.05.22 13:24:38.255 (RTS-6.17,M1)   OnCalculate: 2017.05.22 11:24:38.704 ms v = 2, bid = 108550, ask = 108560, last = 108550, flags = 'TICK_FLAG_SELL TICK_FLAG_LAST TICK_FLAG_VOLUME '
2017.05.22 13:24:38.255 (RTS-6.17,M1)   OnCalculate: 2017.05.22 11:24:38.704 ms v = 2, bid = 108550, ask = 108560, last = 108550, flags = 'TICK_FLAG_SELL TICK_FLAG_LAST TICK_FLAG_VOLUME '
2017.05.22 13:24:38.255 (RTS-6.17,M1)   OnCalculate: 2017.05.22 11:24:38.704 ms v = 2, bid = 108550, ask = 108560, last = 108550, flags = 'TICK_FLAG_SELL TICK_FLAG_LAST TICK_FLAG_VOLUME '
2017.05.22 13:24:38.261 (RTS-6.17,M1)   OnCalculate: 2017.05.22 11:24:38.710 ms v = 1, bid = 108540, ask = 108560, last = 108560, flags = 'TICK_FLAG_BID '
2017.05.22 13:24:38.261 (RTS-6.17,M1)   OnCalculate: 2017.05.22 11:24:38.710 ms v = 1, bid = 108540, ask = 108560, last = 108560, flags = 'TICK_FLAG_BID '
2017.05.22 13:24:38.261 (RTS-6.17,M1)   OnCalculate: 2017.05.22 11:24:38.710 ms v = 1, bid = 108540, ask = 108560, last = 108560, flags = 'TICK_FLAG_BID '
2017.05.22 13:24:38.262 (RTS-6.17,M1)   OnCalculate: 2017.05.22 11:24:38.710 ms v = 1, bid = 108540, ask = 108560, last = 108560, flags = 'TICK_FLAG_BID '
2017.05.22 13:24:38.262 (RTS-6.17,M1)   OnCalculate: 2017.05.22 11:24:38.710 ms v = 1, bid = 108540, ask = 108560, last = 108560, flags = 'TICK_FLAG_BID '
2017.05.22 13:24:38.270 (RTS-6.17,M1)   OnBookEvent: 2017.05.22 11:24:38.710 ms v = 1, bid = 108540, ask = 108560, last = 108560, flags = 'TICK_FLAG_BID '
2017.05.22 13:24:38.280 (RTS-6.17,M1)   OnCalculate: 2017.05.22 11:24:38.730 ms v = 2, bid = 108550, ask = 108560, last = 108560, flags = 'TICK_FLAG_BUY TICK_FLAG_LAST TICK_FLAG_VOLUME '
2017.05.22 13:24:38.280 (RTS-6.17,M1)   OnCalculate: Время последнего STick = 2017.05.22 11:24:38.730 ms (TICK_FLAG_BUY TICK_FLAG_LAST TICK_FLAG_VOLUME ), время последнего CopyTicks = 2017.05.22 11:24:38.719 ms (TICK_FLAG_BID )
2017.05.22 13:24:38.282 (RTS-6.17,M1)   OnBookEvent: 2017.05.22 11:24:38.730 ms v = 2, bid = 108550, ask = 108560, last = 108560, flags = 'TICK_FLAG_BUY TICK_FLAG_LAST TICK_FLAG_VOLUME '
2017.05.22 13:24:38.283 (RTS-6.17,M1)   OnCalculate: 2017.05.22 11:24:38.730 ms v = 2, bid = 108550, ask = 108560, last = 108560, flags = 'TICK_FLAG_BUY TICK_FLAG_LAST TICK_FLAG_VOLUME '
As you can see, when requesting the last tick on each call to OnCalculate() with SymbolInfoTick() - there is a skip in some of the ticks that CopyTicks() returns.
 

Code to check:

#property indicator_chart_window
#property indicator_plots 0
//+------------------------------------------------------------------+
//|                                                                  |
//+------------------------------------------------------------------+
MqlTick _tick;
//+------------------------------------------------------------------+
//| Custom indicator initialization function                         |
//+------------------------------------------------------------------+
int OnInit()
  {
//--- Подписываемся на стакан
   MarketBookAdd(_Symbol);
//---
   return(INIT_SUCCEEDED);
  }
//+------------------------------------------------------------------+
//| Custom indicator iteration function                              |
//+------------------------------------------------------------------+
int OnCalculate(const int rates_total,
                const int prev_calculated,
                const datetime &time[],
                const double &open[],
                const double &high[],
                const double &low[],
                const double &close[],
                const long &tick_volume[],
                const long &volume[],
                const int &spread[])
  {
//---
   ResetLastError();
//--- Проверяем получение параметров тика
   if(!SymbolInfoTick(_Symbol,_tick))
      Print(__FUNCTION__,": ОШИБКА #",GetLastError(),": параметры тика не получены!");
   else
      Print(__FUNCTION__,": "+GetMsToStringTime(_tick.time_msc)+" v = ",_tick.volume,", bid = "+DoubleToString(_tick.bid,_Digits)+
            ", ask = "+DoubleToString(_tick.ask,_Digits)+", last = "+DoubleToString(_tick.last,_Digits)+", flags = '"+GetStrFlags(_tick.flags)+"'");
//---
   ResetLastError();
//---
   MqlTick t[1];
   if(CopyTicks(_Symbol,t,COPY_TICKS_ALL,0,1)!=1)
      Print(__FUNCTION__,": ОШИБКА #",GetLastError(),": Последний тик не получен!");
   else
     {
      if(t[0].time_msc!=_tick.time_msc)
        {
         Print(__FUNCTION__,": Время последнего STick = "+GetMsToStringTime(_tick.time_msc)+" ("+GetStrFlags(_tick.flags)+")"+
               ", время последнего CopyTicks = "+GetMsToStringTime(t[0].time_msc)+" ("+GetStrFlags(t[0].flags)+")");
        }
     }
//---
   return(rates_total);
  }
//+------------------------------------------------------------------+
//|                                                                  |
//+------------------------------------------------------------------+
void OnBookEvent(const string &symbol)
  {
//---
   if(symbol!=_Symbol)
      return;
//--- Структура - приемник последнего тика
   MqlTick tick;
//---
   ResetLastError();
//--- Проверяем получение параметров тика
   if(!SymbolInfoTick(_Symbol,tick))
      Print(__FUNCTION__,": ОШИБКА #",GetLastError(),": параметры тика не получены!");
   else
      Print(__FUNCTION__,": "+GetMsToStringTime(tick.time_msc)+" v = ",tick.volume,", bid = "+DoubleToString(tick.bid,_Digits)+
            ", ask = "+DoubleToString(tick.ask,_Digits)+", last = "+DoubleToString(tick.last,_Digits)+", flags = '"+GetStrFlags(tick.flags)+"'");
//--- Сравниваем последний тик OnCalculate с последним тиком OnBookEvent
   if(tick.time_msc!=_tick.time_msc)
     {
      Print( __FUNCTION__,": Время последнего OnCalculate = "+GetMsToStringTime( _tick.time_msc )+" ("+GetStrFlags( _tick.flags )+")"+
            ", время последнего OnBookEvent = "+GetMsToStringTime( tick.time_msc )+" ("+GetStrFlags( tick.flags )+")" );
     }
  }
//+------------------------------------------------------------------+
//|                                                                  |
//+------------------------------------------------------------------+
void OnDeinit(const int reason)
  {
//--- Отписываемся от стакана
   MarketBookRelease(_Symbol);
  }
//+------------------------------------------------------------------+
//| Получаем строку времени из миллисекунд                                                                      |
//+------------------------------------------------------------------+
string GetMsToStringTime(const ulong ms)
  {
   const int MS_KOEF=1000;               // Коэффициент перевода секунд в миллисекунды
   return( TimeToString( ms/MS_KOEF, TIME_DATE|TIME_SECONDS )+"."+string( ms%MS_KOEF )+" ms" );
  }
//+------------------------------------------------------------------+
//|                                                                  |
//+------------------------------------------------------------------+
string GetStrFlags(const uint flags)
  {
   string s="";
//---
   if(( flags&TICK_FLAG_ASK)==TICK_FLAG_ASK)
      s+="TICK_FLAG_ASK ";
   if(( flags&TICK_FLAG_BID)==TICK_FLAG_BID)
      s+="TICK_FLAG_BID ";
   if(( flags&TICK_FLAG_BUY)==TICK_FLAG_BUY)
      s+="TICK_FLAG_BUY ";
   if(( flags&TICK_FLAG_SELL)==TICK_FLAG_SELL)
      s+="TICK_FLAG_SELL ";
   if(( flags&TICK_FLAG_LAST)==TICK_FLAG_LAST)
      s+="TICK_FLAG_LAST ";
   if(( flags&TICK_FLAG_VOLUME)==TICK_FLAG_VOLUME)
      s+="TICK_FLAG_VOLUME ";
//---
   return( s );
  }
//+------------------------------------------------------------------+
 
Alexey Kozitsyn:

Code to test:

I would leave only writing ticks to file when OnCalculate is called (to test OnBookEvent and CopyTicks, butOnCalculate, as it was intended).

And then by script would have generated the same file, getting the ticks via CopyTicks. The difference would be more obvious, and would indicate either missed ticks in OnCalculate or incorrect operation of CopyTicks.

 
Andrey Khatimlianskii:

To be sure (not to test OnBookEvent and CopyTicks, but OnCalculate, as intended), I would leave only ticks writing to the file when OnCalculate is called.

And then by script would have generated the same file, getting the ticks via CopyTicks. The difference would be clearer, and would indicate either skipping of ticks in OnCalculate or incorrect work of CopyTicks.

With such a simple calculation I do not see the point to bother. There is a specific comparison of two calls: SymbolInfoTick() and CopyTicks(). And all calls of SymbolInfoTick() are clearly shown. And it can be clearly seen that the CopyTicks() returned the tick, which was not received in SymbolInfoTick() call from OnCalculate(). And there can be two reasons for the error here: 1. Either OnCalculate() misses a tick, or 2. SymbolInfoTick() returns the wrong thing somewhere.

Yes, this is part of the log, which was collected for about 30 seconds... so imagine the number of missing ticks.

 
Alexey Kozitsyn:

Our dialogue with the service desk:

Tick skipping in OnCalculate
Open, Started: 2017.05.22 14:52, #1751442

Look to not get lost:https://www.mql5.com/ru/forum/190129/page8#comment_5081300 and the next 2 posts.

CopyTicks from the indicator gives a tick that the indicator does not get in OnCalculate (neither before nor after).

Support Team2017.05.23 09:44
Status:UnprocessedOpened
Your application has been accepted for review.
Support Team2017.05.23 09:46

OnCalculate is called on every tick.

Ticks are added to the database in a separate thread. That is, asynchronously. It's not sure that with CopyTicks you'll get the same tick, that called OnCalculate.

Andrey Khatimlianskii 2017.05.23 14:08

It's not about that.

With CopyTicks it's possible to get a tick, which wasn't in OnCalculate (we learn about it in OnCalculate on a tick, which time is longer than missed).

Take a look at the logs.

Support Team2017.05.23 14:13

This is exactly what we are talking about.

A batch of ticks arrived. Two threads start to process this pack - one thread in the loop executes OnCalculate on each tick in the pack; another thread puts the whole pack into a tick database. These two threads are not synchronized between them in any way, and it is unknown which thread will process a pack of ticks faster

Andrey Khatimlianskii 2017.05.23 14:26

But every tick from the pack should be processed by OnCalculate. And this is not happening.

There is no tick with that time. There is a tick with less time, and a tick with more time. And there is no OnCalculate call in between.

Support Team2017.05.23 14:33

komposter

But every tick in the pack has to be handled by OnCalculate. And this is not happening.

There is no tick with that time. There is a tick with less time, and a tick with more time. And there is no call of OnCalculate between them.

Each tick in the pack is processed at OnCalculate. There are no skips there. This is how the loop is written.

To find out which tick is processed in OnCalculate, useSymbolInfoTick- everything is clear here. The tick is applied to the symbol information and then all indicators of this symbol are called, all in one thread

Andrey Khatimlianskii 2017.05.23 14:36

Please have a look at the log at the link.

SymbolInfoTick from OnCalculate doesn't see the tick, which can be obtained from CopyTicks.

That's why I wrote

Support Team2017.05.23 14:40

We need to do exactly as you suggested inhttps://www.mql5.com/ru/forum/190129/page8#comment_5082755

I would, to be sure (to test OnCalculate instead of OnBookEvent and CopyTicks, as intended), leave only the writing of ticks to the file when OnCalculate is called.

And then by script would have generated the same file, getting the ticks via CopyTicks. The difference would be much clearer, and would indicate either skipping of ticks in OnCalculate or incorrect work of CopyTicks.

And we exactly this way at one time checked the correctness of CopyTicks. The test scripts and indicators are left, we will check it again.

 

Forum on trading, automated trading systems and trading strategy testing

Bugs, bugs, questions

fxsaber, 2017.02.07 13:41

SymbolInfoTick in indicators works quite differently than in EAs.

In an indicator it always returns the tick that was the initiator of the OnCalculate call. And these initiator ticks in the indicator should not be skipped - this is the ideology of the developers. The problem is in the forming of the queue of these ticks.

In the Expert Advisor, the SymbolInfoTick in OnTick does not return the tick that initiated the call of OnTick, but makes a complete request for the current state.

Forum on trading, automated trading systems and strategy testing

Features of mql5 language, subtleties and tricks

fxsaber, 2017.03.29 22:32

Calculate event is generated on EVERY tick. So there is a tick queue for indicators. If it reaches a certain number, there is a warning in the log that the indicator is too slow.

The bars themselves are indicators by nature and are generated by Calculate-events. Therefore, if TF M1 and it is 10:15:00.020 and a 50ms pack comes, the first tick in it still has time 10:14:59.970. And all indicators are called first on this Calculate event - first the timeseries indicator and then custom indicators with these timeseries. I.e. in this layout the 10:15-bar is not yet formed as the zero-bar. And the zero bar is 10:14.


Then, when the pack starts to unwind via Calculate events, the 10:15-bar will also appear. I think I've described it in detail.

Forum on trading, automated trading systems and strategy testing

Features of mql5 language, subtleties and tricks

fxsaber, 2017.03.29 22:41

As a consequence, it is quite normal for an EA to get a tick of a bar that has not yet formed. But this is such a subtle point that it takes effort to even reproduce on purpose. In general, it is more theoretical than practical.


If we are paranoid about speed, using indicators (and bars, of course) is not an option. Everything is in the EA.

 
Andrey Khatimlianskii:

Our dialogue with Service Desk:

Service Desk is, unfortunately, in their repertoire. They don't need to throw in a link when it comes to discussion. But to give a concrete example, on their fingers. That's not what they're talking about. Actually they say, SymbolInfoTick() returns the current tick to be checked, and since OnCalculate() handles each tick - it turns out, that output should be of each tick. And CopyTicks() tells us otherwise. Well... And they say, do not look at CopyTicks() :). About two threads they explained me too, when they were fixing CopyTicks() at the end of year. Only it is not the issue here (although, maybe in it too, I don't know).

Did not ask them about ticks with 0 flag when received from OnBookEvent()?

Reason: