Strategy optimization result do not match single run test result - page 5

[Stanislav Korotky]

Here are some details (from Russian forum, autotranslated into English).

Forum on trading, automated trading systems and testing of trading strategies

Features of the mql5 language, subtleties and working techniques

fxsaber , 2019.01.17 14:37

In addition to what was said earlier , one of the most common reasons for the discrepancy between identical runs in the Tester is incorrect initialization or its absence.

If the lack of initialization of variables is simple, then with arrays it is somewhat more complicated. Most often, the problematic place can be indicated by finding situations where the numberof array elements increases.

To catch such potential problems, you can insert the following lines at the beginning of the advisor

 // Помогает найти причину потенциальной ошибочной инициализации
template < typename T>
int ArrayResize2( T &Array[], const int NewSize, const string Str )
{
   const int PrevSize = ArraySize (Array);
   const int Res = ArrayResize (Array, NewSize);
  
   if ((PrevSize < NewSize) || (Res != NewSize))
  {
   #define TOSTRING(A) ", " + #A + " = " + ( string )(A)
     Print (Str + ( string )Res + TOSTRING(PrevSize) + TOSTRING(NewSize));
   #undef TOSTRING     
  
    TesterStop();
  }
  
   return (Res);
}

// Вариант с Reserve != 0 не предусмотрен
#define ArrayResize (A,B) ArrayResize2(A, B, __FUNCSIG__ + ", Line = " + ( string ) __LINE__ + \
                                             ": ArrayResize(" + #A + ", " + #B + ") = " )

 // Помогает найти причину потенциальной ошибочной инициализации
template < typename T1, typename T2>
int ArrayInitialize2( T1 &Array[], const T2 Value, const string Str )
{
   const int Res = ArrayInitialize (Array, Value);
  
   if (! ArraySize (Array) || !Res)
  {
   #define TOSTRING(A) ", " + #A + " = " + ( string )(A)
     Print (Str + ( string )Res + TOSTRING( ArraySize (Array)));
   #undef TOSTRING     
  
    TesterStop();
  }
  
   return (Res);
}

#define ArrayInitialize (A,B) ArrayInitialize2(A, B, __FUNCSIG__ + ", Line = " + ( string ) __LINE__ +  \
                                                     ": ArrayInitialize(" + #A + ", " + #B + ") = " )

If the situation is caught, detailed information will be output to the log and the run will be stopped.

P.S. Example of application.

Forum on trading, automated trading systems and testing of trading strategies

Optimization and single run results do not match

fxsaber , 2021.11.25 12:05

I have never used the "Forward Optimization" mode. I can only speak for the regular optimization.

This thing automatically checked for matches between many thousands of single run results and their values in optimization tables on hundreds of symbols. Everything matches.

So almost 100%, the reason for the differences is in the source code.

The Tester itself was also checked for bugs (you never know where it worked incorrectly). Didn't find any problems among millions of transactions. Perfect match with another tester. The only BUT - I thoroughly checked only the single-currency mode.

If there are any bugs in the Tester in future MT5 builds, I will most likely find them.

[Francesco Baldi]

I've done extensive testing and logging.

I've isolated a set of lines in the MQL_Easy library where the difference between optimization and single test can be noted.

There's a function in UtilitiesMT5.mqh that is defined as follows:

ENUM_ORDER_TYPE_FILLING CUtilities::FillingOrder()
{
   //-- Find the filling mode
   uint fillingTemp=(uint)SymbolInfoInteger(this.Symbol,SYMBOL_FILLING_MODE);
   //-- check if any error occurs
   if(this.Error.CheckLastError(true,__FUNCTION__))
      return WRONG_VALUE;
   //-- ORDER_FILLING_FOK
   if((fillingTemp&SYMBOL_FILLING_FOK)==SYMBOL_FILLING_FOK){
      return ORDER_FILLING_FOK;
   }//-- ORDER_FILLING_IOC
   else if((fillingTemp&SYMBOL_FILLING_IOC)==SYMBOL_FILLING_IOC){
      return ORDER_FILLING_IOC;
   }else
      return ORDER_FILLING_RETURN;
}

This gets called every time an order to open or to close a position is created (in order to fill MqlTradeRequest::type_filling property).

Well, during the optimization process the call

if (this.Error.CheckLastError(true,__FUNCTION__))

returns true, like there was an error, and the whole operation (open or close) fails.

The problem is that there's no error, the previous line

   uint fillingTemp=(uint)SymbolInfoInteger(this.Symbol,SYMBOL_FILLING_MODE);

gets executed with no error, but, here is the difference between optimization and single test:

during optimization, when the CUtilities::FillingOrder() gets called, for some reason _LastError is not 0, but it has the value 4014:  ERR_FUNCTION_NOT_ALLOWED, set elsewhere before entering the function,

so when GetLastError() is called inside the CheckLastError function, that error code is read and the function returns true making the whole operation fail.

If you run the single test instead, error 4014 never happens.

Given that the code is the same I don't know how can this happen.

Moreover I don't know how to identify which is the statement that triggers that error.

Any suggestion is appreciated.

Attached is the sample ea, setfile and settings that I used for testing.

Running the optimization should produce four results with just one trade each, running a single test from one of those results should produce more than 150 trades instead.

The MQL_Easy library used is this one: https://github.com/Denn1Ro/MQL_Easy

[Alain Verleyen]

Francesco Baldi #:
...

The problem is that there's no error, the previous line

gets executed with no error, but, here is the difference between optimization and single test:

during optimization, when the CUtilities::FillingOrder() gets called, for some reason _LastError is not 0, but it has the value 4014:  ERR_FUNCTION_NOT_ALLOWED, set elsewhere before entering the function,

so when GetLastError() is called inside the CheckLastError function, that error code is read and the function returns true making the whole operation fail.

...

If you want to check for error (_LastError) on a function call, you need to call ResetLastError() just before this call.

ENUM_ORDER_TYPE_FILLING CUtilities::FillingOrder()
{
   // ALWAYS reset the lastError before a function when you want to check for error AFTER the function call.
   ResetLastError();
   //-- Find the filling mode
   uint fillingTemp=(uint)SymbolInfoInteger(this.Symbol,SYMBOL_FILLING_MODE);
   //-- check if any error occurs
   if(this.Error.CheckLastError(true,__FUNCTION__))
      return WRONG_VALUE;
   //-- ORDER_FILLING_FOK
   if((fillingTemp&SYMBOL_FILLING_FOK)==SYMBOL_FILLING_FOK){
      return ORDER_FILLING_FOK;
   }//-- ORDER_FILLING_IOC
   else if((fillingTemp&SYMBOL_FILLING_IOC)==SYMBOL_FILLING_IOC){
      return ORDER_FILLING_IOC;
   }else
      return ORDER_FILLING_RETURN;
}

And it's even better to use the SymbolInfoInteger() overload which returns a bool if you want to check for error.

The MQL_Easy library used is this one: https://github.com/Denn1Ro/MQL_Easy

Great, an other "framework" to NOT use if it contains such trivial bug.

[Stanislav Korotky]

Alain Verleyen #:

If you want to check for error (_LastError) on a function call, you need to call ResetLastError() just before this call.

The problem is that this all happens (not happens, actually) inside Easy library, and it lacks ResetLastError call in huge number of places - almost everywhere where it should be.

[Stanislav Korotky]

Francesco Baldi #:

I've done extensive testing and logging.

I've isolated a set of lines in the MQL_Easy library where the difference between optimization and single test can be noted.

...

Excellent work.

[Alain Verleyen]

Stanislav Korotky #:

The problem is that this all happens (not happens, actually) inside Easy library, and it lacks ResetLastError call in huge number of places - almost everywhere where it should be.

Yes, so my conclusion...

Great, an other "framework" to NOT use if it contains such trivial bug.

[Francesco Baldi]

I'm sorry but I see no real solution yet, not to the root problem.

Yes, ResetLastError() should be called on top of all functions checking for errors, still we have a different behavior that has no explanation between optimization and single tests.

In particular I would like to know where and why 4014: ERR_FUNCTION_NOT_ALLOWED error is generated.

That will keep on happening and will just be hidden after adding the ResetLastError() call, that might lead to errors somewhere else, also outside the library.

I don't know If everyone is fine knowing that random and uncaught errors happen behind the scenes, I am not.

If someone has suggestions on how to catch which statement is generating that error, I will really appreciate it.

Thanks to everybody who has spent time here.

[Alain Verleyen]

Francesco Baldi #:

I'm sorry but I see no real solution yet, not to the root problem.

Yes, ResetLastError() should be called on top of all functions checking for errors, still we have a different behavior that has no explanation between optimization and single tests.

In particular I would like to know where and why 4014: ERR_FUNCTION_NOT_ALLOWED error is generated.

That will keep on happening and will just be hidden after adding the ResetLastError() call, that might lead to errors somewhere else, also outside the library.

I don't know If everyone is fine knowing that random and uncaught errors happen behind the scenes, I am not.

If someone has suggestions on how to catch which statement is generating that error, I will really appreciate it.

Thanks to everybody who has spent time here.

That's not correct perception.

1. There is absolutely no reason why the behaviour should be exactly the same between optimization and single tests. The context is different, some things (function or environment) are possible on a single test and not optimization, so the platform can deal (internally) differently in some circumstances.

2. Nobody can help with your error 4014, as we don't have a source code to test. Anyway, it doesn't matter, it will not give you any useful information even if you identify the exact source. Error 4014 is irrelevant, it could be any other error any where else in the code.

3. It's the responsibility of the coder to have an EA that will have the exact same behaviour in any context (Live, Backtest or optimization). If you are using a function that leads to different internal state of the platform (a not allowed function like your error 4014 seems to suggest, or some uninitialized variables or whatever else), the platform is NOT responsible about it.

4. The root of the problem in your case, seems to be the usage of a poorly coded framework, the code is checking for error on _LastError when there is NO error. As already explained it should either use a variant of the function which return false when there is an error, OR reset the last error before. It's crystal clear.

Conclusion :

NEVER check for an error when there is no error. There is an error when the function return false, or -1, or any other documented way. It's always possible (and it happens effectively very often) that _LastError is set somewhere internally (by the platform) without any actual error.

[fxsaber]

Alain Verleyen #:

NEVER check for an error when there is no error. There is an error when the function return false, or -1, or any other documented way. It's always possible (and it happens effectively very often) that _LastError is set somewhere internally (by the platform) without any actual error.

Форум по трейдингу, автоматическим торговым системам и тестированию торговых стратегий

Библиотеки: Input_Struct

fxsaber, 2024.09.05 09:52

void OnStart()
{
  string Str = "Hello World!";
  int i = (int)Str;
  
  Print(_LastError); // 5040
}

[rodsibin]

I was having the same issue when optimizing symbol WIN$D in Brazilian market, which has tick size of 5.

But it was intermittent. At the result table, some results when I ran single test, the output was the same, some were different. Sometimes they were all correct.

I did several fixes but none worked, like deleting some cache files, only using local farm, or only local processor, checking my code for uninitialized variables or structures, etc.

Until I saw on test log that some tests had this error of invalid price, due to price no multiple of 5 (tick size):

failed exchange buy 5 WIN$D at 136712, close #2 sell 5 WIN$D.15R 136500 [Invalid price]

The weird part was this didn't happen all the times, considering same symbol, same EA, same parameters. I don't understand why sometimes MT5 sends correct price and other times don't. Testing with market closed, in history, with no prices changing. So seems like a bug to me.

Well, the fix was to adjust the price to me multiple of tick size, to use in all calls, like position open, position modify, etc. A function as simple as this:

double AdjustPriceTick(double price)
{
   double tick_size = SymbolInfoDouble(_Symbol, SYMBOL_TRADE_TICK_SIZE);
   return MathRound(price/tick_size)*tick_size;
}
  

I always used this fix so I didn't understand why this issue was happening. I use CTrade class, so all prices in Buy, BuyLimit, BuyStop, Sell, SellLimit, SellStop, PositionModify functions I pass it adjusted.

The problem was on PositionClose function, which doesn't have a price argument, and it was sending orders with incorrect price.

So to fix it, I copied Trade/Trade.mqh to Include folder with a different name and modified PositionClose function to adjust price to close positions.

And DONE, no more errors or different results!!