MT5 and speed in action - page 57

 
Renat Fatkhullin:

Acceleration by a factor of ten in cases of massively parallel access.

For other cases only processor, memory and operating system upgrades.

In PM, comparative logs from the same machine.

 
fxsaber:

This is a one time call of CopyTicks. It's done in order to make a virtual backtest for these ticks in OnInit, and then to continue it in real-time, feeding only fresh ticks.

As a compromise, I propose to release the memory in the Terminal immediately after the CopyTicks called in OnInit. Then we don't have to introduce a forced cooling function for CopyTicks.

Right now Sleep-version of cooling is very crutchy. But I showed above how this crutch saves memory.


Now it turns out that 20 Expert Advisors run fast even on slow VPS. But starting them up is a serious problem.

Not so long ago, you yourself have been stress-testing and demanded instant issuing of deep ticks on each tick.

That is, both you and 100% of other developers have applied and will apply pending full cache request strategies. Each time more and more costly and head-on experts are being written.

That's why we won't abolish caches by any means, but recommend to put 16-32gb of memory and forget about problems and delays.

Behavior and economy on VPS (keep to 512 mb-1 gb) we don't care at all.

 
Renat Fatkhullin:

You yourself were stress-testing not so long ago, and in fact you demanded instant deep ticks on every tick.

That is, both you and 100% of other developers have applied and will apply pending full cache request strategies. Each time more and more costly and head-on experts are being written.

That's why we won't abolish caches by any means, but recommend to put 16-32gb of memory and forget about problems and delays.

Behavior and economy on VPS (to fit in 512 mb-1gb) we don't care at all.

Above supplemented my post with code. It's about OnInit, not something else. Hot CopyTicks is very much needed, but not in OnInit.

 
fxsaber:

As a compromise, I suggest that after CopyTicks called in OnInit, memory should be freed immediately in the Terminal.

Not in any case!
And what if I on the contrary raise cache in OnInit in order to work with it without delays?

It's better to introduce the function to unload the terminal cache.
It is necessary not only for ticks, but also for time series and indicators (try to write a 5K for 5 instruments and 5 TFs).

@Renat?

 
Andrey Khatimlianskii:

No way!
What if I raise the cache in OnInit so that I can then work with it without delay?

The tick cache only lives for 10 seconds.

 

The tick history script screener (the only software running in the Terminal) eats up to 6GB if you take five months' worth of quotations.

The faster the machine and internet, the greater the memory load. I.e. terminal's internal caches accumulate to a few characters in 10 seconds.

 
Fast machine, nine advisers.
2020.10.19 22:32:06.965         Bench_Stack = 3, 100 <= Time[NewTicks.mqh 33 in NEWTICKS::GetMarketWatchTick: ::SymbolInfoTick(_Symbol,Tick)] = 1229 mcs.
2020.10.19 22:33:23.727         Bench_Stack = 3, 100 <= Time[NewTicks.mqh 33 in NEWTICKS::GetMarketWatchTick: ::SymbolInfoTick(_Symbol,Tick)] = 1250 mcs.
2020.10.19 22:34:29.802         Bench_Stack = 3, 100 <= Time[NewTicks.mqh 33 in NEWTICKS::GetMarketWatchTick: ::SymbolInfoTick(_Symbol,Tick)] = 1281 mcs.
2020.10.19 22:35:58.747         Bench_Stack = 2, 100 <= Time[NewTicks.mqh 33 in NEWTICKS::GetMarketWatchTick: ::SymbolInfoTick(_Symbol,Tick)] = 1164 mcs.
2020.10.19 22:37:20.196         Bench_Stack = 3, 100 <= Time[NewTicks.mqh 33 in NEWTICKS::GetMarketWatchTick: ::SymbolInfoTick(_Symbol,Tick)] = 1045 mcs.
2020.10.19 22:38:24.920         Bench_Stack = 3, 100 <= Time[NewTicks.mqh 33 in NEWTICKS::GetMarketWatchTick: ::SymbolInfoTick(_Symbol,Tick)] = 1907 mcs.
2020.10.19 22:39:48.359         Bench_Stack = 3, 100 <= Time[NewTicks.mqh 33 in NEWTICKS::GetMarketWatchTick: ::SymbolInfoTick(_Symbol,Tick)] = 1387 mcs.
2020.10.19 22:40:03.623         Bench_Stack = 3, 100 <= Time[NewTicks.mqh 33 in NEWTICKS::GetMarketWatchTick: ::SymbolInfoTick(_Symbol,Tick)] = 1302 mcs.
2020.10.19 22:40:44.569         Bench_Stack = 3, 100 <= Time[NewTicks.mqh 33 in NEWTICKS::GetMarketWatchTick: ::SymbolInfoTick(_Symbol,Tick)] = 1271 mcs.
2020.10.19 22:41:09.393         Bench_Stack = 3, 100 <= Time[NewTicks.mqh 33 in NEWTICKS::GetMarketWatchTick: ::SymbolInfoTick(_Symbol,Tick)] = 1482 mcs.
2020.10.19 22:41:19.831         Bench_Stack = 3, 100 <= Time[NewTicks.mqh 33 in NEWTICKS::GetMarketWatchTick: ::SymbolInfoTick(_Symbol,Tick)] = 1024 mcs.
2020.10.19 22:41:19.975         Alert: Bench_Stack = 3, 100 <= Time[NewTicks.mqh 33 in NEWTICKS::GetMarketWatchTick: ::SymbolInfoTick(_Symbol,Tick)] = 8026 mcs.
2020.10.19 22:41:50.137         Bench_Stack = 2, 100 <= Time[NewTicks.mqh 33 in NEWTICKS::GetMarketWatchTick: ::SymbolInfoTick(_Symbol,Tick)] = 1040 mcs.
2020.10.19 22:42:05.876         Bench_Stack = 2, 100 <= Time[NewTicks.mqh 33 in NEWTICKS::GetMarketWatchTick: ::SymbolInfoTick(_Symbol,Tick)] = 1346 mcs.
2020.10.19 22:43:21.478         Bench_Stack = 3, 100 <= Time[NewTicks.mqh 33 in NEWTICKS::GetMarketWatchTick: ::SymbolInfoTick(_Symbol,Tick)] = 1523 mcs.
2020.10.19 22:43:21.557         Bench_Stack = 3, 100 <= Time[NewTicks.mqh 33 in NEWTICKS::GetMarketWatchTick: ::SymbolInfoTick(_Symbol,Tick)] = 1245 mcs.
2020.10.19 22:43:21.843         Alert: Bench_Stack = 3, 100 <= Time[NewTicks.mqh 33 in NEWTICKS::GetMarketWatchTick: ::SymbolInfoTick(_Symbol,Tick)] = 4675 mcs.
2020.10.19 22:43:21.854         Alert: Bench_Stack = 3, 100 <= Time[NewTicks.mqh 33 in NEWTICKS::GetMarketWatchTick: ::SymbolInfoTick(_Symbol,Tick)] = 7567 mcs.
2020.10.19 22:44:01.181         Bench_Stack = 3, 100 <= Time[NewTicks.mqh 33 in NEWTICKS::GetMarketWatchTick: ::SymbolInfoTick(_Symbol,Tick)] = 1052 mcs.
2020.10.19 22:44:33.124         Bench_Stack = 3, 100 <= Time[NewTicks.mqh 33 in NEWTICKS::GetMarketWatchTick: ::SymbolInfoTick(_Symbol,Tick)] = 2714 mcs.
2020.10.19 22:44:54.967         Alert: Bench_Stack = 3, 100 <= Time[NewTicks.mqh 33 in NEWTICKS::GetMarketWatchTick: ::SymbolInfoTick(_Symbol,Tick)] = 13626 mcs.
2020.10.19 22:45:07.561         Bench_Stack = 3, 100 <= Time[NewTicks.mqh 33 in NEWTICKS::GetMarketWatchTick: ::SymbolInfoTick(_Symbol,Tick)] = 2800 mcs.
I don't know how to overcome such lags. I can't think of any other way to know if there is a new tick while OnTick is running.
 
fxsaber:
I can't think of any other way to know that there is a new tick while OnTick is running.

Three options.

// Идентификация нового тика
bool IsNewTick_CopyRates()
{
  static MqlRates PrevRates = {0};
  MqlRates Rates[1];
    
  const bool Res = (::CopyRates(_Symbol, PERIOD_CURRENT, 0, 1, Rates) == 1) &&
//                   (_R(PrevRates) != Rates[0]); // TypeToBytes.mqh
                   ((Rates[0].time != PrevRates.time) ||
                    (Rates[0].tick_volume > PrevRates.tick_volume) ||
                    (Rates[0].close != PrevRates.close)); // Лишнее условие, но на всякий случай.
                   
  if (Res)
    PrevRates = Rates[0];
    
  return(Res);
}

// Идентификация нового тика
bool IsNewTick_SymbolInfoTick()
{
  static MqlTick PrevTick = {0};
  MqlTick Tick;
  
  const bool Res = ::SymbolInfoTick(_Symbol, Tick) &&
//                   (_R(PrevTick) != Tick); // TypeToBytes.mqh
                   ((PrevTick.time_msc != Tick.time_msc) ||
                    (PrevTick.bid != Tick.bid) ||
                    (PrevTick.ask != Tick.ask));
                   
  if (Res)
    PrevTick = Tick;
    
  return(Res);
}

// Идентификация нового тика
bool IsNewTick_CopyTicks()
{
  static MqlTick PrevTick = {0};
  MqlTick Tick[1];
  
  const bool Res = (::CopyTicks(_Symbol, Tick, COPY_TICKS_ALL, 0, 1) == 1) &&
//                   (_R(PrevTick) != Tick[0]); // TypeToBytes.mqh
                   ((PrevTick.time_msc != Tick[0].time_msc) ||
                    (PrevTick.bid != Tick[0].bid) ||
                    (PrevTick.ask != Tick[0].ask));
                   
  if (Res)
    PrevTick = Tick[0];
    
  return(Res);
}

#define  PRINT(A)            \
  if (!A)                   \
    Print(#A + " = false");


void OnTick()
{
  PRINT(IsNewTick_CopyTicks());      // true
  PRINT(IsNewTick_SymbolInfoTick()); // true
  PRINT(IsNewTick_CopyRates())       // false
}

It is better not to identify a new tick through the bars.

 
It became clear how long CopyTicks can lag behind SymbolInfoTick.
// Замер длительности синхронизации CopyTicks и SymbolInfoTick.

#include <TypeToBytes.mqh> // https://www.mql5.com/ru/code/16280

bool IsSynch()
{
  MqlTick Tick;
  MqlTick Ticks[1];
  
  return(SymbolInfoTick(_Symbol, Tick) && (CopyTicks(_Symbol, Ticks, COPY_TICKS_ALL, 0, 1) == 1) && (_R(Tick) == Ticks[0]));
}

ulong WaitSynch( int &Count )
{
  while (!IsStopped() && IsSynch())
    Sleep(0);
    
  const ulong StartTime = GetMicrosecondCount();
  Count = 0;
  
  while (!IsStopped() && !IsSynch())
    Count++;
    
  return(GetMicrosecondCount() - StartTime);
}

void OnTick()
{
  int Count;
  const uint StartTime = GetTickCount();

  while (GetTickCount() - StartTime < 10000)
    Print("Cинхронизация CopyTicks и SymbolInfoTick длилась " + (string)(WaitSynch(Count) / 1000)  + " ms., число проверок = " + (string)Count);
}


Result(inAmount = 15).

2020.10.20 00:32:46.316 Test9 (GBPAUD,H1)       Cинхронизация CopyTicks и SymbolInfoTick длилась 163 ms., число попыток = 391432
2020.10.20 00:32:46.894 Test9 (GBPAUD,H1)       Cинхронизация CopyTicks и SymbolInfoTick длилась 221 ms., число попыток = 526533
2020.10.20 00:32:50.839 Test9 (GBPAUD,H1)       Cинхронизация CopyTicks и SymbolInfoTick длилась 69 ms., число попыток = 112339
2020.10.20 00:32:50.839 Test9 (EURAUD,H1)       Cинхронизация CopyTicks и SymbolInfoTick длилась 385 ms., число попыток = 837204
2020.10.20 00:32:51.958 Test9 (EURAUD,H1)       Cинхронизация CopyTicks и SymbolInfoTick длилась 74 ms., число попыток = 166407
2020.10.20 00:32:52.044 Test9 (EURAUD,H1)       Cинхронизация CopyTicks и SymbolInfoTick длилась 80 ms., число попыток = 180256
2020.10.20 00:32:52.797 Test9 (EURAUD,H1)       Cинхронизация CopyTicks и SymbolInfoTick длилась 737 ms., число попыток = 1469883
2020.10.20 00:33:04.229 Test9 (EURAUD,H1)       Cинхронизация CopyTicks и SymbolInfoTick длилась 8550 ms., число попыток = 14608891
2020.10.20 00:33:04.319 Test9 (GBPAUD,H1)       Cинхронизация CopyTicks и SymbolInfoTick длилась 159 ms., число попыток = 150630
2020.10.20 00:33:04.324 Test9 (AUDCHF,H1)       Cинхронизация CopyTicks и SymbolInfoTick длилась 85 ms., число попыток = 78591
2020.10.20 00:33:07.340 Test9 (EURAUD,H1)       Cинхронизация CopyTicks и SymbolInfoTick длилась 2947 ms., число попыток = 4899320
2020.10.20 00:33:08.076 Test9 (EURAUD,H1)       Cинхронизация CopyTicks и SymbolInfoTick длилась 727 ms., число попыток = 1209371
2020.10.20 00:33:08.138 Test9 (EURAUD,H1)       Cинхронизация CopyTicks и SymbolInfoTick длилась 55 ms., число попыток = 73155
2020.10.20 00:33:14.233 Test9 (EURCAD,H1)       Cинхронизация CopyTicks и SymbolInfoTick длилась 107 ms., число попыток = 149697
2020.10.20 00:33:15.985 Test9 (EURCAD,H1)       Cинхронизация CopyTicks и SymbolInfoTick длилась 190 ms., число попыток = 230501
2020.10.20 00:33:16.114 Test9 (GBPJPY,H1)       Cинхронизация CopyTicks и SymbolInfoTick длилась 208 ms., число попыток = 275639
2020.10.20 00:33:41.328 Test9 (GBPJPY,H1)       Cинхронизация CopyTicks и SymbolInfoTick длилась 9457 ms., число попыток = 14426849
2020.10.20 00:33:46.000 Test9 (GBPCHF,H1)       Cинхронизация CopyTicks и SymbolInfoTick длилась 53622 ms., число попыток = 109127013
2020.10.20 00:33:47.936 Test9 (GBPJPY,H1)       Cинхронизация CopyTicks и SymbolInfoTick длилась 1302 ms., число попыток = 3104323
2020.10.20 00:33:55.715 Test9 (GBPAUD,H1)       Cинхронизация CopyTicks и SymbolInfoTick длилась 59 ms., число попыток = 87263
2020.10.20 00:33:55.776 Test9 (GBPJPY,H1)       Cинхронизация CopyTicks и SymbolInfoTick длилась 88 ms., число попыток = 125641
2020.10.20 00:33:55.869 Test9 (GBPJPY,H1)       Cинхронизация CopyTicks и SymbolInfoTick длилась 86 ms., число попыток = 140282
2020.10.20 00:33:55.974 Test9 (GBPJPY,H1)       Cинхронизация CopyTicks и SymbolInfoTick длилась 95 ms., число попыток = 91802
2020.10.20 00:33:56.004 Test9 (GBPAUD,H1)       Cинхронизация CopyTicks и SymbolInfoTick длилась 125 ms., число попыток = 127590
2020.10.20 00:33:56.202 Test9 (GBPJPY,H1)       Cинхронизация CopyTicks и SymbolInfoTick длилась 214 ms., число попыток = 311277
2020.10.20 00:34:40.522 Test9 (GBPAUD,H1)       Cинхронизация CopyTicks и SymbolInfoTick длилась 31 ms., число попыток = 48189
2020.10.20 00:34:40.672 Test9 (GBPAUD,H1)       Cинхронизация CopyTicks и SymbolInfoTick длилась 69 ms., число попыток = 99619
2020.10.20 00:34:41.094 Test9 (GBPAUD,H1)       Cинхронизация CopyTicks и SymbolInfoTick длилась 412 ms., число попыток = 613382
2020.10.20 00:34:41.864 Test9 (GBPAUD,H1)       Cинхронизация CopyTicks и SymbolInfoTick длилась 768 ms., число попыток = 1136798
2020.10.20 00:34:46.035 Test9 (GBPAUD,H1)       Cинхронизация CopyTicks и SymbolInfoTick длилась 2012 ms., число попыток = 2955715
2020.10.20 00:34:46.124 Test9 (GBPAUD,H1)       Cинхронизация CopyTicks и SymbolInfoTick длилась 79 ms., число попыток = 109796
2020.10.20 00:34:46.239 Test9 (GBPAUD,H1)       Cинхронизация CopyTicks и SymbolInfoTick длилась 98 ms., число попыток = 152433
2020.10.20 00:34:47.261 Test9 (GBPAUD,H1)       Cинхронизация CopyTicks и SymbolInfoTick длилась 1009 ms., число попыток = 1300923
2020.10.20 00:34:47.337 Test9 (GBPAUD,H1)       Cинхронизация CopyTicks и SymbolInfoTick длилась 73 ms., число попыток = 66414
2020.10.20 00:34:47.850 Test9 (AUDCHF,H1)       Cинхронизация CopyTicks и SymbolInfoTick длилась 885 ms., число попыток = 832072
2020.10.20 00:34:48.383 Test9 (GBPAUD,H1)       Cинхронизация CopyTicks и SymbolInfoTick длилась 1040 ms., число попыток = 1239140
2020.10.20 00:34:49.093 Test9 (EURCAD,H1)       Cинхронизация CopyTicks и SymbolInfoTick длилась 46 ms., число попыток = 43232
2020.10.20 00:34:49.839 Test9 (GBPAUD,H1)       Cинхронизация CopyTicks и SymbolInfoTick длилась 1445 ms., число попыток = 2065046
2020.10.20 00:34:51.474 Test9 (GBPCHF,H1)       Cинхронизация CopyTicks и SymbolInfoTick длилась 55801 ms., число попыток = 119517920
2020.10.20 00:34:51.755 Test9 (GBPAUD,H1)       Cинхронизация CopyTicks и SymbolInfoTick длилась 1498 ms., число попыток = 1625714
2020.10.20 00:34:52.269 Test9 (GBPJPY,H1)       Cинхронизация CopyTicks и SymbolInfoTick длилась 403 ms., число попыток = 693015
2020.10.20 00:34:59.175 Test9 (AUDCAD,H1)       Cинхронизация CopyTicks и SymbolInfoTick длилась 9413 ms., число попыток = 16211601
2020.10.20 00:35:00.398 Test9 (EURAUD,H1)       Cинхронизация CopyTicks и SymbolInfoTick длилась 6302 ms., число попыток = 11626079
2020.10.20 00:35:05.587 Test9 (AUDCHF,H1)       Cинхронизация CopyTicks и SymbolInfoTick длилась 3645 ms., число попыток = 8637511
2020.10.20 00:35:07.247 Test9 (EURAUD,H1)       Cинхронизация CopyTicks и SymbolInfoTick длилась 133 ms., число попыток = 205326
2020.10.20 00:35:09.033 Test9 (AUDCHF,H1)       Cинхронизация CopyTicks и SymbolInfoTick длилась 3427 ms., число попыток = 8060967

Desynchronisation can last up to a minute. There is a bug somewhere.

 

The reason for SymbolInfoTick's braking seems to be parallel handling.

// Демонстрация тормозов SymbolInfoTick
#include <fxsaber\Benchmark\Benchmark.mqh> // https://www.mql5.com/ru/code/31279

void OnTick()
{
  const uint StartTime = GetTickCount();

  while (!IsStopped() && (GetTickCount() - StartTime < 10000))
  {
    _B(SymbolInfoInteger(_Symbol, SYMBOL_TIME_MSC), 100000);
    
//    Sleep(0); // Специально убрал.
  }
}


Result (inAmount = 15).

2020.10.20 01:01:30.517 Test9 (EURAUD,H1)       Alert: Bench_Stack = 0, 100000 <= Time[Test9.mq5 97 in OnTick: SymbolInfoInteger(_Symbol,SYMBOL_TIME_MSC)] = 105941 mcs.
2020.10.20 01:01:37.596 Test9 (USDCAD,H1)       Alert: Bench_Stack = 0, 100000 <= Time[Test9.mq5 97 in OnTick: SymbolInfoInteger(_Symbol,SYMBOL_TIME_MSC)] = 103045 mcs.
2020.10.20 01:01:38.968 Test9 (EURCHF,H1)       Alert: Bench_Stack = 0, 100000 <= Time[Test9.mq5 97 in OnTick: SymbolInfoInteger(_Symbol,SYMBOL_TIME_MSC)] = 103305 mcs.
2020.10.20 01:01:41.307 Test9 (EURUSD,H1)       Alert: Bench_Stack = 0, 100000 <= Time[Test9.mq5 97 in OnTick: SymbolInfoInteger(_Symbol,SYMBOL_TIME_MSC)] = 106121 mcs.
2020.10.20 01:01:42.573 Test9 (USDCAD,H1)       Alert: Bench_Stack = 0, 100000 <= Time[Test9.mq5 97 in OnTick: SymbolInfoInteger(_Symbol,SYMBOL_TIME_MSC)] = 128241 mcs.
2020.10.20 01:01:45.175 Test9 (GBPCHF,H1)       Alert: Bench_Stack = 0, 100000 <= Time[Test9.mq5 97 in OnTick: SymbolInfoInteger(_Symbol,SYMBOL_TIME_MSC)] = 120367 mcs.
2020.10.20 01:01:46.394 Test9 (EURCHF,H1)       Alert: Bench_Stack = 0, 100000 <= Time[Test9.mq5 97 in OnTick: SymbolInfoInteger(_Symbol,SYMBOL_TIME_MSC)] = 104458 mcs.
2020.10.20 01:01:57.832 Test9 (GBPCHF,H1)       Alert: Bench_Stack = 0, 100000 <= Time[Test9.mq5 97 in OnTick: SymbolInfoInteger(_Symbol,SYMBOL_TIME_MSC)] = 125348 mcs.

> 100 ms for SymbolInfoTick execution. It's clear that the code is self-shot. But it shows the reason why slows down on usual EAs.

Reason: