How to improve your program's performance? - General

Renat Fatkhullin 2020.10.27 11:38 #611

Rorschach:
Would a realtime kernel help in any way?

Put in more cores and don't push the situation to 100% single core load.

Don't believe in fairy tales, processors and thread scheduler don't work the way you imagine.

Igor Makanu 2020.10.27 11:50 #612

Anton:

In the latest builds the tick stream reception has no effect even theoretically. Practically SymbolInfoTick already works with cache, but individual citizens keep looking for a black cat.

And it's not even 80% in the test. It has 6 agents running on 4 cores, i.e. 100% guaranteed.

The only question is how his system's task scheduler is handling the situation. At the same time, some claims are being made that it is the terminal implementation which is to blame.

That is, a situation is artificially created when a computer is overloaded, when literally everything on it slows down, and then some claims are made in the form of "Oh, look, why is the terminal lags sometimes".

Let's close our eyes to the fact that even in such conditions it is "about 0.01%" - to hell with the details! Suffice it to say that "no one cares about the average hospital temperature", "lags cause problems when trading" and "we want HFT".

Moreover, of course we want HFT in 20 experts on an old office desktop or a dead virtual machine.

PS PositionSelectByTicket() in its implementation certainly has access to a shared resource with access synchronisation. And if you don't select the position on every call, you're reading the old price. It was easier to do a "snapshot" via SymbolInfoDouble.

thanks

I got my question because six months ago, I was optimizing my code and testing the speed of system functions, and six months ago, SymbolInfoDouble was slower than SymbolInfoTick

Overall, I think you're right. Just googled few articles today about multi-core cache (I haven't got interested in this info for a long time),

here's a short articlehttps://i2hard.ru/publications/25666/

the point is that data gets to run in ALU only from L1-cache which is very small and if you load processor full speed, then really - the test will turn into a test of the operating system + test of the processor cache speed (loading, predicting L1+L3 data) but not in testing code (application) performance

Errors, bugs, questions AMD or Intel as Any questions from newcomers

Aleksey Vyazmikin 2020.10.27 12:59 #613

fxsaber, what if you set a low priority for agents in task manager and a high one for MT5?

I can't find a utility that would block all programs/OS threads from being allocated to a specific CPU thread, otherwise it would be possible for MT5 to reserve a thread and automatically block it from being used by other programs, which could in theory reduce lags.

4 core instead of Errors, bugs, questions Any questions from newcomers

fxsaber 2020.10.28 08:21 #614

Anton:

In the latest builds the tick stream reception has no effect even theoretically. Practically, SymbolInfoTick already works with cache, but certain citizens keep looking for a black cat.

A particular citizen has got the MQL-codeduplicated from his wide trousers which showed that the crutch works faster than the regular function under the same conditions.

But you argue that your function is good, it just has limitations on conditions of use.

And it's not even 80% in the test. There are 6 agents running on 4 cores, i.e. 100% guaranteed.

The only question is how his system's task scheduler is handling the situation. At the same time, the authors make claims that it is the implementation of the terminal that is to blame.

That is, a situation is artificially created when a computer is overloaded to its limits and everything slows down on it, and then some claims are made in the form of "Oh, look, why is the terminal lags sometimes".

6/8 - nothing is lagging. Browsers, compilation, debugging etc. run in parallel without any hint of lags.

But now I've purposely turned everything off, leaving only 4/8 Agent. The situation has not changed with your function braking.

What's more, of course, we want HFT in 20 experts on an old office desktop or a dead virtual machine.

A fast machine was used. And only 6 charts were already giving the brakes.

PS PositionSelectByTicket() in its implementation certainly has access to a shared resource with access synchronisation. And if you don't select the position on each call, you read the old price. It was easier to "snapshot" via SymbolInfoDouble.

I use this too.

// Снепшот SymbolInfoTick для текущего символа.
bool SymbolInfoTick( MqlTick &Tick )
{
  static MqlTick PrevTick = {0};
  static ulong PrevTime = 0;
  
  const ulong NewTime = GetMicrosecondCount();
  const bool Res = (NewTime - PrevTime < 1000) || (::SymbolInfoTick(_Symbol, PrevTick) && (bool)(PrevTime = NewTime));
  
  Tick = PrevTick;
  
  return(Res);
}

Euro Rat [ARCHIVE]Any rookie question, so [ARCHIVE!] Any rookie question,

Vladimir Pastushak 2020.10.28 09:52 #615

What kind of speed in combat performance can we talk about when there are problems in the terminal.

The Expert Advisor scans all financial instruments and searches for a given pattern.

It stumbles on a symbol and hangs tight!!!

Code from help, I put only steps, financial instrument with problem and timer:

//+------------------------------------------------------------------+ 
//|                                              TestLoadHistory.mq5 | 
//|                        Copyright 2009, MetaQuotes Software Corp. | 
//|                                              https://www.mql5.com | 
//+------------------------------------------------------------------+ 
#property copyright "2009, MetaQuotes Software Corp." 
#property link      "https://www.mql5.com" 
#property version   "1.02" 
#property script_show_inputs 
//--- input parameters 
input string          InpLoadedSymbol="RTSCHH1";   // Symbol to be load 
input ENUM_TIMEFRAMES InpLoadedPeriod=PERIOD_H1;  // Period to be load 
input datetime        InpStartDate=D'2006.01.01'; // Start date 
//+------------------------------------------------------------------+ 
//| Script program start function                                    | 
//+------------------------------------------------------------------+ 
void OnStart() 
  { 
   Print("Start load",InpLoadedSymbol+","+GetPeriodName(InpLoadedPeriod),"from",InpStartDate); 
//--- 
   int res=CheckLoadHistory(InpLoadedSymbol,InpLoadedPeriod,InpStartDate); 
   switch(res) 
     { 
      case -1 : Print("Unknown symbol ",InpLoadedSymbol);             break; 
      case -2 : Print("Requested bars more than max bars in chart "); break; 
      case -3 : Print("Program was stopped ");                        break; 
      case -4 : Print("Indicator shouldn't load its own data ");      break; 
      case -5 : Print("Load failed ");                                break; 
      case  0 : Print("Loaded OK ");                                  break; 
      case  1 : Print("Loaded previously ");                          break; 
      case  2 : Print("Loaded previously and built ");                break; 
      default : Print("Unknown result "); 
     } 
//--- 
   datetime first_date; 
   SeriesInfoInteger(InpLoadedSymbol,InpLoadedPeriod,SERIES_FIRSTDATE,first_date); 
   int bars=Bars(InpLoadedSymbol,InpLoadedPeriod); 
   Print("First date ",first_date," - ",bars," bars"); 
//--- 
  } 
//+------------------------------------------------------------------+ 
//|                                                                  | 
//+------------------------------------------------------------------+ 
int CheckLoadHistory(string symbol,ENUM_TIMEFRAMES period,datetime start_date) 
  { 
  Print(" === 1 === ");
   datetime first_date=0; 
   datetime times[100]; 
//--- check symbol & period 
   if(symbol==NULL || symbol=="") symbol=Symbol(); 
   if(period==PERIOD_CURRENT)     period=Period(); 
//--- check if symbol is selected in the MarketWatch 
   if(!SymbolInfoInteger(symbol,SYMBOL_SELECT)) 
     { 
      if(GetLastError()==ERR_MARKET_UNKNOWN_SYMBOL) return(-1); 
      SymbolSelect(symbol,true); 
     } 
     
     ulong time = GetMicrosecondCount();
     Print(" === 2 === ",first_date);
//--- check if data is present 
  bool date = SeriesInfoInteger(symbol,period,SERIES_FIRSTDATE,first_date); 
   
   Print(" === 2.1 === ",(time - GetMicrosecondCount()));
   
   if(first_date>0 && first_date<=start_date) return(1); 
//--- don't ask for load of its own data if it is an indicator 

Print(" === 2.2 === ");
   if(MQL5InfoInteger(MQL5_PROGRAM_TYPE)==PROGRAM_INDICATOR && Period()==period && Symbol()==symbol) 
      return(-4); 
//--- second attempt 
Print(" === 2.3 === ");
   if(SeriesInfoInteger(symbol,PERIOD_M1,SERIES_TERMINAL_FIRSTDATE,first_date)) 
     { 
      //--- there is loaded data to build timeseries 
      Print(" === 2.4 === ");
      if(first_date>0) 
        { 
         //--- force timeseries build 
         CopyTime(symbol,period,first_date+PeriodSeconds(period),1,times); 
         //--- check date 
         Print(" === 2.5 === ");
         if(SeriesInfoInteger(symbol,period,SERIES_FIRSTDATE,first_date)) 
            if(first_date>0 && first_date<=start_date) return(2); 
        } 
     } 
     
     Print(" === 3 === ");
     
//--- max bars in chart from terminal options 
   int max_bars=TerminalInfoInteger(TERMINAL_MAXBARS); 
//--- load symbol history info 
   datetime first_server_date=0; 
   while(!SeriesInfoInteger(symbol,PERIOD_M1,SERIES_SERVER_FIRSTDATE,first_server_date) && !IsStopped()) 
      Sleep(5); 
//--- fix start date for loading 

Print(" === 4 === ");


   if(first_server_date>start_date) start_date=first_server_date; 
   if(first_date>0 && first_date<first_server_date) 
      Print("Warning: first server date ",first_server_date," for ",symbol, 
            " does not match to first series date ",first_date); 
//--- load data step by step 

Print(" === 5 === ");

   int fail_cnt=0; 
   while(!IsStopped()) 
     { 
      //--- wait for timeseries build 
      while(!SeriesInfoInteger(symbol,period,SERIES_SYNCHRONIZED) && !IsStopped()) 
         Sleep(5); 
      //--- ask for built bars 
      int bars=Bars(symbol,period); 
      if(bars>0) 
        { 
         if(bars>=max_bars) return(-2); 
         //--- ask for first date 
         if(SeriesInfoInteger(symbol,period,SERIES_FIRSTDATE,first_date)) 
            if(first_date>0 && first_date<=start_date) return(0); 
        } 
      //--- copying of next part forces data loading 
      int copied=CopyTime(symbol,period,bars,100,times); 
      if(copied>0) 
        { 
         //--- check for data 
         if(times[0]<=start_date)  return(0); 
         if(bars+copied>=max_bars) return(-2); 
         fail_cnt=0; 
        } 
      else 
        { 
         //--- no more than 100 failed attempts 
         fail_cnt++; 
         if(fail_cnt>=100) return(-5); 
         Sleep(10); 
        } 
     } 
     
     Print(" === 6 === ");
//--- stopped 
   return(-3); 
  } 
//+------------------------------------------------------------------+ 
//| Возвращает строкое значение периода                              | 
//+------------------------------------------------------------------+ 
string GetPeriodName(ENUM_TIMEFRAMES period) 
  { 
   if(period==PERIOD_CURRENT) period=Period(); 
//--- 
   switch(period) 
     { 
      case PERIOD_M1:  return("M1"); 
      case PERIOD_M2:  return("M2"); 
      case PERIOD_M3:  return("M3"); 
      case PERIOD_M4:  return("M4"); 
      case PERIOD_M5:  return("M5"); 
      case PERIOD_M6:  return("M6"); 
      case PERIOD_M10: return("M10"); 
      case PERIOD_M12: return("M12"); 
      case PERIOD_M15: return("M15"); 
      case PERIOD_M20: return("M20"); 
      case PERIOD_M30: return("M30"); 
      case PERIOD_H1:  return("H1"); 
      case PERIOD_H2:  return("H2"); 
      case PERIOD_H3:  return("H3"); 
      case PERIOD_H4:  return("H4"); 
      case PERIOD_H6:  return("H6"); 
      case PERIOD_H8:  return("H8"); 
      case PERIOD_H12: return("H12"); 
      case PERIOD_D1:  return("Daily"); 
      case PERIOD_W1:  return("Weekly"); 
      case PERIOD_MN1: return("Monthly"); 
     } 
//--- 
   return("unknown period"); 
  }

Result of the work:

2020.10.28 11:18:08.067 Test (FUTBRNJAN21,M1)   Start loadRTSCHH1,H1from2006.01.01 00:00:00
2020.10.28 11:18:08.067 Test (FUTBRNJAN21,M1)    === 1 === 
2020.10.28 11:18:08.067 Test (FUTBRNJAN21,M1)    === 2 === 1970.01.01 00:00:00
2020.10.28 11:22:01.741 Test (FUTBRNJAN21,M1)    === 2.1 === 18446744073475877138    Время выполнения SeriesInfoInteger(symbol,period,SERIES_FIRSTDATE,first_date); 
2020.10.28 11:22:01.741 Test (FUTBRNJAN21,M1)    === 2.2 === 
2020.10.28 11:22:01.741 Test (FUTBRNJAN21,M1)    === 2.3 === 
2020.10.28 11:22:01.741 Test (FUTBRNJAN21,M1)    === 3 ===

I never waited for the script to finish running.

As long as the terminal has such bugs, no battle execution is out of the question!!!

It was expected that when a security gets into the market review, at least all of its properties and the start date of the history are pulled up for it. If there is no history on the server, then

SeriesInfoInteger(symbol,period,SERIES_FIRSTDATE,first_date)

Should return NULL instantly, why the execution time 18446744073475877138 ?

Maybe i don't know something, CopyXXX function also hangs for 16-29 seconds !!!

It's not normal for a broker to have 3000-6000 financial instruments.

Errors, bugs, questions Questions from Beginners MQL5 How do I get

fxsaber 2020.10.28 10:01 #616

Vladimir Pastushak:

Maybe I don't know what, the CopyXXX functions also hang for 16-29 seconds!!!

It's not normal for a broker to have 3000-6000 financial instruments.

Bars are evil. So please post about them in another thread.

Vladimir Pastushak 2020.10.28 10:04 #617

fxsaber:

Bars are evil. So please post about them in another thread.

Maybe you know how to programmatically select a financial instrument and not get hung up for ages?

fxsaber 2020.10.28 10:07 #618

Vladimir Pastushak:

Maybe you know how to programmatically select a financial instrument and not get hung up for ages?

I haven't come across such a task.

fxsaber 2020.10.28 10:38 #619

Aleksey Vyazmikin:

fxsaber, what if you set a low priority for agents in task manager and a high one for MT5?

I can't find a utility that would block allocation of a specific CPU thread to all programs/OS threads, otherwise it would be possible to reserve a thread for MT5 and automatically block its occupation by other programs, which could in theory reduce lags.

Set all Agents to lowest priority.

It does not work.

ZZZ The number of EAs running affects the result.

fxsaber 2020.10.28 12:30 #620

Dear developers, could you please inform me how MQL_MEMORY_USED is calculated?

I have calculated the memory that all EA's variables are taking up.

MQL_MEMORY_USED = 60 MB, Virtual = 3.40 MB ( 5.7%)

It is less than 10%. If I understand correctly, MQL_MEMORY_USED includes the History cache and CopyTicks cache. But it is still much less.

At the same time, the parallel Expert Advisor consumes several times less. But the principle is the same.

In general, what is included to this value?

I have saved a template with Expert Advisor and applied it to the same chart by causing reloading. I have seen it like this.

MQL_MEMORY_USED = 7 MB, Virtual = 3.40 MB ( 48.6%)

The memory usage has changed by almost an order of magnitude. It's hard to explain what this means at the moment.

Big projects. How to clear memory max n. of EA

MT5 and speed in action - page 62