Libraries: TesterBenchmark - page 4

[Vasiliy Sokolov]

The reason for the main retardation of MT5 is the imitation of the full trading environment. In the vast majority of Expert Advisors, such an accurate simulation is not necessary. An excellent solution for such Expert Advisors would be to switch to a simplified testing mode, with simple modelling of the trading environment. However, there is no such possibility. But if you write your own, simplified strategy tester and run it in the mode of mathematical calculations, the speed of optimisation will be fire! This is what I suggest that everyone who wants to dramatically increase the speed of EA optimisation should think about.

[fxsaber]

Vasiliy Sokolov:

In CStrategy, trading operations are performed through CTrade directly. I.e. CStrategy has no trading logic of its own at all. In the test Expert Advisor I did not see any other trading logic than opening/closing a position after N seconds. CStrategy also does not store historical positions, so this example is unfortunately not realisable in CStrategy.

CTrade doesn't store history either, but that didn't prevent us from implementing the variant through it.

The fact that OOP code is slower than procedural code does not tell me anything. Let's get to the point: which CTrade methods are a bottle neck? Why? What does the profiling of these methods tell you? How do you identify slow code sections in the tester?

I haven't analysed the reasons of SB brakes, but I don't think it's OOP, because MT4Orders is also written through OOP, even if it's minimal. After a request to the BOD, the developers accelerated the SB. I don't know whether they have tweaked the SB itself or the compiler.

I myself analyse absolute brakes first, and then - relative ones

MetaTrader 5 has an excellent function of profiling an Expert Advisor on historical data. But, besides the fact that it works slowly (in visual mode), the final result is provided in relative units, i.e. it is impossible to compare performance in absolute terms.

[fxsaber]

Vasiliy Sokolov:

The reason for the main slowdown of MT5 is the imitation of the full trading environment. In the vast majority of Expert Advisors, such an accurate simulation is not necessary. An excellent solution for such Expert Advisors would be to switch to a simplified testing mode, with simple modelling of the trading environment. However, there is no such possibility. But if you write your own, simplified strategy tester and run it in the mode of mathematical calculations, the speed of optimisation will be fire! This is what I suggest you to think about for all those who want to dramatically increase the speed of EA optimisation.

Through custom symbols you can achieve orders of magnitude speedup with full simulation. Actually, even now in the beta mode of custom symbols it is possible to achieve this. I will do it later.

[fxsaber]

MQL5:

i = 5 Pass = 5 OnTester = 8.521 s.: Count = 15925124, 1868926.7 unit/sec, Agent = C:\Program Files\Alpari Limited MT5\Tester\Agent-127.0.0.1-3000 build = 1653

MT4Orders:

i = 2 Pass = 2 OnTester = 8.001 s.: Count = 15925124, 1990391.7 unit/sec, Agent = C:\Program Files\Alpari Limited MT5\Tester\Agent-127.0.0.1-3000 build = 1653

MQL5 turned out to be slower than the library written in the same MQL5! I started to investigate and found the reason by making this substitution

// if (!PositionSelect(_Symbol))
  if (!PositionGetTicket(0))

MQL5 variant of the Expert Advisor started to show such performance.

i = 4 Pass = 4 OnTester = 7.931 s.: Count = 15925124, 2007959.1 unit/sec, Agent = C:\Program Files\Alpari Limited MT5\Tester\Agent-127.0.0.1-3000 build = 1653

Stupid replacement of PositionSelect with PositionGetTicket raises the backtest speed by 7%!

MT4Orders lags behind the maximally optimised pure MQL5 by less than a percent.

[fxsaber]

fxsaber:

The places were distributed as follows

1. Pure MQL5 - 100% performance.
2. 
   
3. SB Trade\Trade.mqh - ~84% performance.

The SB has become less laggy.

I tweaked the SB itself a little bit (pieces of SB used by the Expert Advisor), without changing the functionality and versatility. It became not 84%, but 97%.

Those who use SB, take note of the potential possibility of acceleration.

[fxsaber]

It became interesting to compare the performance of the same MT4 multicurrency Expert Advisor converted in different ways.

Full conversion

#include <TesterBenchmark.mqh> // https://www.mql5.com/en/code/18804
#include "Spreader 2.mq5"      // https://www.mql5.com/en/code/19402

and artisanal (fast) conversion

Forum on trading, automated trading systems and testing trading strategies.

Expert Advisors: Spreader

fxsaber, 2016.09.03 11:18 AM.

// https://www.mql5.com/en/code/16006
#include <MT4Orders.mqh>
#include <MQL4_to_MQL5.mqh>

#include "Spreader_v2.mq4" // https://www.mql5.com/en/code/9495

void OnTick()
{
  start();
}

Results of performance measurement.

Full-fledged version

i = 1 Pass = 1 OnTester = 3.465 s.: Count = 897532, 259028.0 unit/sec, Agent = C:\Program Files\Alpari Limited MT5\Tester\Agent-127.0.0.1-3000 build = 1653

Artisanal

i = 1 Pass = 1 OnTester = 4.815 s.: Count = 897532, 186403.3 unit/sec, Agent = C:\Program Files\Alpari Limited MT5\Tester\Agent-127.0.0.1-3000 build = 1653

The artisanal one loses significantly. The reason is simple - if you want fast timeseries, you can't convert MT4 timeseries head-on

Forum on trading, automated trading systems and testing trading strategies.

Libraries: CPrice

fxsaber, 2016.08.28 10:36 AM

Look at such an option

#define  DEFINE_TIMESERIE(NAME,FUNC,T)                                                                         \
  class CLASS##NAME                                                                                           \
  {                                                                                                           \
  public:                                                                                                     \
    static T Get(const string Symb,const int TimeFrame,const int iShift)                                      \
    {                                                                                                         \
      T tValue[];                                                                                             \
                                                                                                              \
      return((Copy##FUNC((Symb == NULL) ? _Symbol : Symb, _Period, iShift, 1, tValue) > 0) ? tValue[0] : -1); \
    }                                                                                                         \
                                                                                                              \
    T operator[](const int iPos) const                                                                        \
    {                                                                                                         \
      return(CLASS##NAME::Get(_Symbol, _Period, iPos));                                                       \
    }                                                                                                         \
  };                                                                                                          \
                                                                                                              \
  CLASS##NAME  NAME;                                                                                           \
                                                                                                              \
  T i##NAME(const string Symb,const int TimeFrame,const int iShift)                                           \
  {                                                                                                           \
    return(CLASS##NAME::Get(Symb,  TimeFrame, iShift));                                                        \
  }

DEFINE_TIMESERIE(Volume,TickVolume,long)
DEFINE_TIMESERIE(Time,Time,datetime)
DEFINE_TIMESERIE(Open,Open,double)
DEFINE_TIMESERIE(High,High,double)
DEFINE_TIMESERIE(Low,Low,double)
DEFINE_TIMESERIE(Close,Close,double)

You can, as in MT4, call Open[bar], High[bar], Time[bar], Volume[bar], etc. And also iHigh(...), iClose(...) and others.

You need to use (not only for bush, but also for full-fledged variants) solutions like this.

[fxsaber]

fxsaber:

The artisanal one loses significantly. The reason is simple - if you want fast timeseries, you can't convert MT4 timeseries head-on.

Wrong conclusion. The reason for the brakes is quite different. Let's insert only one line into each of MT5 variants

#define Comment(A)

And measure again

• Full-fledged

i = 1 Pass = 1 OnTester = 2.074 s.: Count = 897532, 432754.1 unit/sec, Agent = C:\Program Files\Alpari Limited MT5\Tester\Agent-127.0.0.1-3000 build = 165

• Artisanal.

i = 1 Pass = 1 OnTester = 2.310 s.: Count = 897532, 388542.0 unit/sec, Agent = C:\Program Files\Alpari Limited MT5\Tester\Agent-127.0.0.1-3000 build = 1653

The full-fledged one is 67% faster, the artisanal one is 108% faster! In summary, the gap is now small - full-featured is ~11% faster than artisanal. There, it seems, the reason is in the OOP overhead.

But that's not the main thing. With one line we managed to speed up EAs! And this is in the Optimiser, where Comment doesn't play any role.

[fxsaber]

Forum on trading, automated trading systems and testing trading strategies

Libraries: TesterBenchmark

fxsaber, 2017.07.24 14:13

trading SB is 1.5 times slower than pure MQL5!

This was written in the summer along with the SD application. I'm surprised that everything is even! Or is it logical? How can one talk about super-speed of the tester when almost all EAs (using SB) fail in speed, eat money in the Cloud, etc.! Apparently this is normal.

I simplified the source code by throwing out the work with History and MT4Orders. I left only pure MQL5 and SB

#include <TesterBenchmark.mqh>

#include <Trade\Trade.mqh> // Comment out to enable the variant in pure MQL5

input int Interval = 60;

void OnTick()
{
#ifdef   ERR_USER_INVALID_HANDLE // trade.mqh
  static CTrade Trade;
  static CDealInfo Deal;
  static CPositionInfo Position;

  if (!Position.SelectByIndex(0))
  {
    if (HistorySelect(0, LONG_MAX))
    {
      const int Total = HistoryDealsTotal() - 1;

      if ((Total >= 0) && Deal.SelectByIndex(Total) && (Deal.DealType() == DEAL_TYPE_SELL))
        Trade.Sell(1);
      else
        Trade.Buy(1);
    }
  }
  else if (TimeCurrent() - Position.Time() >= Interval)
    Trade.PositionClose(_Symbol);
#else  // ERR_USER_INVALID_HANDLE
  if (!PositionGetTicket(0))
  {
    if (HistorySelect(0, LONG_MAX))
    {
      const int Total = HistoryDealsTotal() - 1;
      MqlTradeRequest Request = {0};

      Request.action = TRADE_ACTION_DEAL;

      Request.symbol = _Symbol;
      Request.type = ((Total >= 0) && ((ENUM_DEAL_TYPE)HistoryDealGetInteger(HistoryDealGetTicket(Total), DEAL_TYPE) == DEAL_TYPE_SELL)) ?
                     ORDER_TYPE_SELL : ORDER_TYPE_BUY;;

      Request.volume = 1;
      Request.price = SymbolInfoDouble(Request.symbol, (Request.type == ORDER_TYPE_BUY) ? SYMBOL_ASK : SYMBOL_BID);

      MqlTradeCheckResult CheckResult;
      if (OrderCheck(Request, CheckResult))
      {
        MqlTradeResult Result;

        const bool AntiWarning = OrderSend(Request, Result);
      }
    }
  }
  else if (TimeCurrent() - PositionGetInteger(POSITION_TIME) >= Interval)
  {
    MqlTradeRequest Request = {0};
    MqlTradeResult Result;

    Request.action = TRADE_ACTION_DEAL;
    Request.position = PositionGetInteger(POSITION_TICKET);

    Request.symbol = PositionGetString(POSITION_SYMBOL);
    Request.type = (ENUM_ORDER_TYPE)(1 - PositionGetInteger(POSITION_TYPE));

    Request.volume = PositionGetDouble(POSITION_VOLUME);
    Request.price = PositionGetDouble(POSITION_PRICE_CURRENT);

    const bool AntiWarning = OrderSend(Request, Result);
  }
#endif // ERR_USER_INVALID_HANDLE
}

SB result

------
OnTesterInit
i = 0 Pass = 0 OnTester = 1.898 s.: Count = 2007057, 1057458.9 unit/sec, Agent = C:\Program Files\Alpari Limited MT5\Tester\Agent-127.0.0.1-3000 build = 1730
i = 1 Pass = 1 OnTester = 1.900 s.: Count = 2007057, 1056345.8 unit/sec, Agent = C:\Program Files\Alpari Limited MT5\Tester\Agent-127.0.0.1-3000 build = 1730
iMin = 0 Results[iMin] = 1.898 s.
iMax = 1 Results[iMax] = 1.900 s.
Amount = 2 Mean = 1.899 s. - 81.75%
OnTesterDeinit
------
Interval = 4.646 s., Count = 0, 0.0 unit/sec

Result of pure MQL5

------
OnTesterInit
i = 0 Pass = 0 OnTester = 1.239 s.: Count = 2007057, 1619900.7 unit/sec, Agent = C:\Program Files\Alpari Limited MT5\Tester\Agent-127.0.0.1-3000 build = 1730
i = 1 Pass = 1 OnTester = 1.243 s.: Count = 2007057, 1614687.9 unit/sec, Agent = C:\Program Files\Alpari Limited MT5\Tester\Agent-127.0.0.1-3000 build = 1730
iMin = 0 Results[iMin] = 1.239 s.
iMax = 1 Results[iMax] = 1.243 s.
Amount = 2 Mean = 1.241 s. - 75.10%
OnTesterDeinit
------
Interval = 3.305 s., Count = 0, 0.0 unit/sec

All the same 1.5 times! But the suckers don't care, let's continue to use super-SB.

[fxsaber]

fxsaber:

Same 1.5 times! But the suckers don't care, let's keep using the super-SB.

This is the kind of SB tweaks

//+------------------------------------------------------------------+
//| Send order|
//+------------------------------------------------------------------+
bool CTrade::OrderSend(const MqlTradeRequest &request,MqlTradeResult &result)
  {
   static const bool IsTester = (::MQLInfoInteger(MQL_TESTER) || ::MQLInfoInteger(MQL_OPTIMIZATION));
   
   bool res;
   string action="";
   string fmt   ="";
//--- action
   if(m_async_mode)
      res=::OrderSendAsync(request,result);
   else
      res=::OrderSend(request,result);
//--- check
   if(res)
     {
      if(!IsTester &&  m_log_level>LOG_LEVEL_ERRORS)
         PrintFormat(__FUNCTION__+": %s [%s]",FormatRequest(action,request),FormatRequestResult(fmt,request,result));
     }
   else
     {
      if(!IsTester &&  m_log_level>LOG_LEVEL_NO)
         PrintFormat(__FUNCTION__+": %s [%s]",FormatRequest(action,request),FormatRequestResult(fmt,request,result));
     }
//--- return the result
   return(res);
  }

//+------------------------------------------------------------------+
//| Select a position on the index|
//+------------------------------------------------------------------+
bool CPositionInfo::SelectByIndex(const int index)
  {
   return(PositionGetTicket(index));
   
   ENUM_ACCOUNT_MARGIN_MODE margin_mode=(ENUM_ACCOUNT_MARGIN_MODE)AccountInfoInteger(ACCOUNT_MARGIN_MODE);
//---
   if(margin_mode==ACCOUNT_MARGIN_MODE_RETAIL_HEDGING)
     {
      ulong ticket=PositionGetTicket(index);
      if(ticket==0)
         return(false);
     }
   else
     {
      string name=PositionGetSymbol(index);
      if(name=="")
         return(false);
     }
//---
   return(true);
  }

miraculously speeds up the SB advisors. Is it complicated?

Frankly speaking, the trading SB, without changing the API logic, should be rewritten almost completely. Any piece of it is bad.