MT5 build 2007 - optimisation is broken. Rerun of tests silently fail. Often optimisation results give wildly different results when ‘Run Single Test’. Number of CPU cores affects results.

New comment
 

Having become quite frustrated with MT5 builds 2006 and 2007, I decided to create a reproducible case using MQL example code (Moving Averages EA) for illustrating the problems with the optimiser and test run environment.

I welcome others to reproduce these tests and establish any patterns that can be fed back to development.  

MT5 Build 2007 Optimiser Bugs Repeatable Case With Brand New MT5 Installation

Objective: Clean, reproducible case of broken MT5 Build 2007

To remove variables that could affect the reproducible case, the test setup needs to be reproducible by anyone:

  • Windows 10, latest updates applied, and latest drivers.
  • Two test environments: With and without graphics card, i.e. with and without OpenCL capabilities (two machines might be easier)
  • I tested with 
    • Windows 10 Home x64 on i7(4770, 8 cores) + NVIDIA Geforce GT1030 (OpenCL 1.2)
    • Windows 10 Pro x64 in Parallels VM (2 cores) on Mac(Xeon E5-1650) with No OpenCL

  • New installations of MT5, fresh download from broker
  • Use the same (live) broker account to connect to for all tests
  • No pre-loads of tick history - connect to live account, then run tests
  • Use MQL supplied demo EA: Moving Averages, to avoid source code concerns
  • Defined optimisation configuration 
  • No use of cloud or network agents: all optimisation on the test machine

Findin gs:

In my testing across 2 different computers with brand new MT5 installations:

  • Pass 0 is always correct. (Bugs in boundary conditions?)
  • Many Optimisation Results show zero trades, yet they do trade.
  • Many Optimisation Results showing zero trades should trade, yet don’t.
  • Often results showing trades give wildly different results when ‘Run Single Test’
    • Zero trade results do trade.
    • Trading results give different results to expected
  • There seems to be a connection between the number of cores and the number of recorded results from the optimisation process.
  • The Journal confirms the parameters used for a given single test, but the Inputs tab gets it wrong and can no longer be relied upon to show what parameters were just used. 
  • Rerun tests and the outcomes can change from showing trades to showing no trades.  Inconsistent results cause further confusion.
  • In the end I found no difference due to the presence or absence of OpenCL (graphics card compute). In fact I don't think OpenCL is used during optimisation.

MT5 Build 2006/2007 is broken and raises some key questions:

How can community members develop and optimise EAs if the testing environment is so broken?  

How can we pick the best parameters for analysis and trading? 

Parameters picked in good faith from this build of the Optimiser may in fact be disastrous when trading.  

Given how bad this release is, I question the commitment to improving quality of MT5 releases. What regression testing and test driven development (TDD) is in place?

In my conclusion, I believe MT5 builds 2006/2007 to be fundamentally broken.  I think it would take a brave soul to use this software for trading, development or EA optimisation.

I urge Development to get a fixed release out ASAP.

Full screen shot based report showing the repeatable case attached.

 

I have Version 2007.

Optimization showed one result, then when I ran it the result was almost twice the optimization. seemed weird.

I am doing a test presently......... will see if similar occurs.

 
I observed the same behaviour with Build 1995 on a fresh Windows 7. A have tried so many things in vain that I think it's better to let it be and wait for a fix. Until then I'll optimize with MT4 and port the results to the mql5 EA.
New comment
Reason: