All Blogs / Other

MQL's OOP notes: On The Fly Self-Optimization of Expert Advisers: part 1

14 November 2016, 14:23

1 083

Are you ready for a very very long story? It will be probably most complicated and prolonged publication in the series of MQL OOP so far, but it's surely worth reading, because its main theme is automatic self-optimization of expert advisers on the fly. This is one of the most frequently asked questions of MetaTrader users. And the answer so far is NO.

Ideally, MetaTrader could provide automatic self-optimization of EAs by design. The built-in tester does work, and only thing which is missing is an API, exposing the settings of the tester for MQL code. For example, there could be a couple of functions - TesterSetSettings, TesterGetSettings, accepting a structure with date ranges (how many bars from history to use), optimization mode (open prices, control points, ticks), spread, method and goal of optimization (straightforward/genetic, balance/profit-factor/drawdown/etc.), and a reference to a file with parameters optimization set (which parameters to optimize, in which ranges, with which increments). This set-file can be prepared by user only once (just by clicking Save button on Inputs tab of the optimization dialog), and then he can start optimization by preferred schedule from MQL code by calling a dedicated method like this: TesterStartOptimization. The results of optimization could be then accessed via similar API, and EA could select a better parameter set on the fly. Unfortunately, neither MT4 nor MT5 can afford this.

To overcome the problem we will create a library to perform virtual trading on history with different parameter sets and choose best candidates. It sounds simple, but actually requires to write more or less precise replica of the built-in optimizer. We can't do exact copy, because, for example, we can't generate ticks, but open prices mode is feasible, and this is our task. Control points mode is also doable, but with much more efforts and some limitations (for indicators), so we'll leave it to enthusiasts.

The library will work in MetaTrader 4. I have many reasons to choose it over MetaTrader 5 - its API is more simple and I use it more - just to name a few. If you like MetaTrader 5, you can develop similar library - it should be easy considering the fact that this library provides a detailed roadmap.

Interface Design

It would be great to design the library in such a way, that its embedding into existing EA would require minimal alterations. As far as most natural and popular method of coding EAs for MetaTrader 4 is based on the set of functions from core APIs (such as, account and order related functions), the simpliest approach would be using the same functions both for online and virtual trading. In other words, if EA contains some code blocks calculating signals, price levels, and lots, they should be kept intact as much as possible, and the code should not even feel a difference between optimization cycles and real-time trading. Is this possible at all? Yes, it is.

The code of ordinary EA is running in the global context, where API functions such as OrderSend, OrderClose, iOpen or iClose are defined and provided by the platform. If we move the code inside a class and define our own methods imitating OrderSend, OrderClose, iOpen, iClose and all the others, inside the class, then the original algorithm will work as usual, without noticing the deception.

For that purpose we need to implement the full trading context similar to what the platform provides, as a class. In this class we can define a virtual callback method (analogue of OnTick) to be invoked from the library. EA should create a derived class for the trade context and override the virtual callback, where the code from existing OnTick handler should be moved to. Here is an example:

class LibraryContext
{
  public:
    int OrdersTotal()
    {
      return 0; // stub
    }

    virtual void trade() = 0;
};

class EAContext: public LibraryContext
{
  public
    virtual void trade()
    {
      int n = OrdersTotal(); // will execute LibraryContext::OrdersTotal
    }
};

This is just a simplified scheme for explanation. In reality we'll need to use slightly different hierarchy of classes.

The library will manage automatic switching between 2 modes: virtual optimization and real trading. When optimization is on (by some given schedule), all functions of the trade context will be redirected to internal virtual placeholders, collecting information about every order opening, modification, or closure. The library will run this virtual trading for different parameter sets and find a best set. When optimization is off, all functions of the trade context will be redirected to the real trade context of the platform, and all orders will go to the market.

Interface blueprint

For the start let us make a draft of the programming interface. It should contain public declarations of all classes and methods available for EA programmers. All the stuff will go to a header file (mqh), which should be included into EA. Then we'll develop internal implementation of the APIs in the library source file (mq4), and add an import directive for it into the header.

Which information do we need to control an optimization process from within EA? Here is the main points:

a simple way to enable/disable the library as a whole
history depth to use for virtual trading (ending datetime is always now)
how often to repeat optimization (for example, if optimization runs on 2 months history, it probably makes sense to update it every week)
initial deposit size (remember, some EAs prefer to take risks only on a part of account balance)
work spread (current spread can be inappropriate if optimization is performed on weekends)
optimization goal (which value to optimize: balance, profit factor, sharpe ratio, etc)
and finally, parameters - an array of named entities (inputs) with a range of values to probe, and increments

Let's put it in a class. I chose the name Optimystic for the library, so use it for the main class (and for the header file name as well).

class Optimystic
{
  public:
    virtual void setEnabled(const bool enabled) = 0;
    virtual void setHistoryDepth(const int size, const TIME_UNIT unit) = 0;
    virtual void setReoptimizationPeriod(const int size, const TIME_UNIT unit) = 0;
    virtual void setDeposit(const double deposit) = 0;
    virtual void setSpread(const int points = 0) = 0;

    virtual void setOptimizationCriterion(const PERFORMANCE_ESTIMATOR type) = 0;

    virtual void addParameter(const string name, const double min, const double max, const double step) = 0;
};

All the methods are pure virtual, because we're defining an abstract interface. If it wouldn't be so, then after the header is included into EA source code, compiler would give the error "function must have a body" for every non-pure virtual method. Compiler needs a complete implementation of every concrete method (function) or requires an import directive pointing to a library, where to get it from. But in MQL it's not possible to import classes. Libraries can only export plain old functions. This is why we need to introduce a so-called factory function, which can create an instance of our class and provide it for EA.

#import "Optimystic.ex4"
Optimystic *createInstance();
#import

This "says" to compiler: this function will return a runtime object with specified virtual functions, but their implementation is not your bussiness at the moment.

But let's go back to the class. It uses 2 enumerations, which are not yet defined.

enum TIME_UNIT
{
UNIT_BAR,
UNIT_DAY
};

We can specify time ranges either in bars or in days.

enum PERFORMANCE_ESTIMATOR
{
  ESTIMATOR_DEFAULT = -1,
  ESTIMATOR_NET_PROFIT, // default
  ESTIMATOR_PROFIT_FACTOR,
  ESTIMATOR_SHARPE,
  ESTIMATOR_DRAWDOWN_PERCENT,
  ESTIMATOR_WIN_PERCENT,
  ESTIMATOR_AVERAGE_PROFIT,
  ESTIMATOR_TRADES_COUNT,
  ESTIMATOR_CUSTOM
};

These constants allow us to choose one of most popular optimization criteria. I think their names are self-explaining.

Since the parameters are not only added by EA, but also updated by the library, they seem also requiring a dedicated programming interface. Indeed, we want, for example, to read current values during optimization and to apply final optimal values after optimization. So add a class with all parameter properties and then bind it with Optimystic class.

class OptimysticParameter
{
  public:
    virtual string getName() const = 0;
    virtual double getMin() const = 0;
    virtual double getMax() const = 0;
    virtual double getStep() const = 0;
    virtual double getValue() const = 0;
    virtual double getBest() const = 0;
    virtual bool getEnabled() const = 0;
    virtual void setEnabled(const bool e) = 0;
};

The first 4 methods just returns the properties that we passed to the library via addParameter method. getValue is a current value, and getBest is an optimal value (if any). Also we should have an option to exclude (disable) specific parameter from optimization temporary. This will allow us to add all parameters to the library in a batch and then play with them without recompiling EA - just by switching inputs.

Remember, we need a way to access objects of the class OptimysticParameter. So add a pair of methods.

class Optimystic
{
  public:
    virtual OptimysticParameter *getParameter(const int i) const = 0;
    OptimysticParameter *operator[](int i) const
    {
      return getParameter(i);
    }

The method getParameter is also pure virtual (and its implementation is postponed for a while), but the helper overload of operator[] is defined inline using the former method.

When optimization is made, it's usually important to know all performance indicators for the best parameter set. We can provide the following method for this:

virtual double getOptimalPerformance(PERFORMANCE_ESTIMATOR type = ESTIMATOR_DEFAULT) = 0;

Finally, let's add a method which will be an entry point for the library on every tick.

virtual void onTick() = 0;

It should be called inside EA's OnTick event handler.

When the library gets control from EA via this onTick method, it should prepare trading context (no matter how - we'll come back to this a moment later), and then execute a callback method, which EA should provide, as we described above (in the section Interface Design). The method is going to be a part of a class, so let's outline one.

class OptimysticCallback
{
  protected:
    datetime TimeCurrent()
    {
      //...
    }

    double AccountBalance()
    {
      //...
    }

    int OrdersTotal()
    {
      //...
    }

    int OrderSend(...)
    {
      //...
    }

    // ...

  public:
    virtual void trade() = 0;
};

The class contains not only the callback trade, but also stubs for most of standard API functions. It's not yet important what will be inside them. The main point to note here, is that from the method trade's point of view, all standard names will lead to the protected substitutes.

Every surrogate method should call corresponding platform implementation during normal trading mode, and provide an alternative virtual counterpart for optimization.

If you remember, we created OptimysticCallback in order to provide the callback method in EA, which would be called from the library. It means that we need to pass a reference to the object OptimysticCallback into the library somehow. The best solution is to add it as parameter into our factory function:

#import "Optimystic.ex4"
Optimystic *createInstance(OptimysticCallback &callback);
#import

Interface refinement

To put it all together, let's imagine how EA code could look right now.

input int HistoryDepth = 500;
input int ReoptimizationPeriod = 100;

extern int MAPeriod = 10;

input int MAPeriodMin = 5;
input int MAPeriodMax = 25;
input int MAPeriodStep = 5;

#include <Optimystic.mqh>

class MyOptimysticCallback: public OptimysticCallback
{
  public:
    virtual void trade() // override
    {
      // some code analyzing iMA with MAPeriod
      // the code of this method was originally placed in EA's OnTick
      // ...
      OrderSend(...);
    }
};

MyOptimysticCallback myc; // instantiate MyOptimysticCallback object

Optimystic *p;

void OnInit()
{
  p = createInstance(GetPointer(myc));
  p.setHistoryDepth(HistoryDepth, UNIT_BAR);
  p.setReoptimizationPeriod(ReoptimizationPeriod, UNIT_BAR);
  p.addParameter("MAPeriod", MAPeriodMin, MAPeriodMax, MAPeriodStep);
}

void OnDeinit(const int reason)
{
  delete p;
}

void OnTick()
{
  p.onTick();
}

The control flow is normally as follows:

EA creates a custom callback object derived from OptimysticCallback;
EA creates an instance of Optimystic and passes the callback object to it;
EA adjusts settings of the library;
On every tick EA calls the library via Optimystic::onTick method;
The library does some work behind the scenes and calls MyOptimysticCallback::trade method once for current datetime and parameter set;
EA trades using OrderSend inside MyOptimysticCallback::trade;
As far as the library is in real trading mode, OrderSend goes to the market;

During virtual optimization the flow changes on the steps 5-7:

The library does some work behind the scenes and calls MyOptimysticCallback::trade method many times for different bars in past and different parameter sets;
EA trades using OrderSend inside MyOptimysticCallback::trade;
As far as the library is in optimization mode, OrderSend performs virtual trades;
The library calculates performance indicators for all trades and finds best parameter set;
...

Here we stopped because we're missing some stuff - specifically a method to apply new parameters to EA. Let's extend OptimysticCallback.

class OptimysticCallback
{
  public:
    virtual bool onBeforeOptimization() = 0;
    virtual bool onApplyParameters() = 0;
    virtual void trade(const int bar, const double Ask, const double Bid) = 0;
    virtual void onReadyParameters()
    {
    }
    virtual void onAfterOptimization() = 0;

Actually, we have added 4 methods at once, because they are logically interconnected. It's likely that EA will need to make some preparations before every optimization, this is why we introduced onBeforeOptimization. And we do already know what to place inside its concrete implementation:

bool MyOptimysticCallback::onBeforeOptimization()
{
  p.setDeposit(::AccountBalance());
  p.setSpread((int)MarketInfo(Symbol(), MODE_SPREAD));
  return true;
}

As you remember, we have reserved setDeposit and setSpread in Optimystic. The method onBeforeOptimization returns boolean to let the library know if EA is ready to start optimization: by returning false EA can deny pending optimization.

The method onApplyParameters should be called by the library when a new parameter set is about to be tested. And again, we do already know what to place inside:

bool MyOptimysticCallback::onApplyParameters()
{
MAPeriod = (int)p[0].getValue();
return true;
}

As you remember, operator[] is overloaded for Optimystic to return specific parameter, and it returns an object OptimysticParameter. In turn, it provides current parameter value via getValue.

The method onReadyParameters is called by the library after single pass of optimization. This is the place, where EA can read current performance indicators:

void MyOptimysticCallback::onReadyParameters()
{
  double net = p.getOptimalPerformance(ESTIMATOR_NET_PROFIT);
  double pf = p.getOptimalPerformance(ESTIMATOR_PROFIT_FACTOR);
  double ddp = p.getOptimalPerformance(ESTIMATOR_DRAWDOWN_PERCENT);
  int n = (int)p.getOptimalPerformance(ESTIMATOR_TRADES_COUNT);
  Print("Performance: profit=", DoubleToString(net, 2), " PF=", (pf == DBL_MAX ? "n/a" : DoubleToString(pf, 2)), " DD%=", DoubleToString(ddp, 2), " N=", n);
}

Just as a brief reminder, the method getOptimalPerformance has been added to Optimystic a few paragraphs above.

Finally, the method onAfterOptimization is the place where the best parameter set is already known and can be applied to EA.

void MyOptimysticCallback::onAfterOptimization()
{
  if(p.getOptimalPerformance(ESTIMATOR_CUSTOM) > 0) // if have positive result
  {
    MAPeriod = (int)p[0].getBest();
  }
}

So get back to the control flow and complement it:

1. EA creates a custom callback object derived from OptimysticCallback;

2. EA creates an instance of Optimystic and passes the callback object to it;

3. EA adjusts settings of the library;

4. On a next tick EA calls the library via Optimystic::onTick method;

5. The library does some work behind the scenes and decides if it's time to start optimization according to the settings given in setReoptimizationPeriod;

if it's time:

6.1. The library calls MyOptimysticCallback::trade method many times for different bars in past and different parameter sets;

6.2. EA trades using OrderSend inside MyOptimysticCallback::trade;

6.3. As far as the library is in optimization mode, OrderSend performs virtual trades;

6.4. The library calculates performance indicators for all trades and finds best parameter set;

6.5. EA applies the best parameters for real trading and even "jumps" to the step 7.1;

if it is not:

7.1. The library calls MyOptimysticCallback::trade method once for current datetime and active parameter set;

7.2. EA trades using OrderSend inside MyOptimysticCallback::trade;

7.3. As far as the library is in real trading mode, OrderSend (if it's actually called according to strategy signals) goes to the market;

What is the work behind the scenes that we mention all the time? This implies switching the trading context between real and virtual one. Most simple way to explain this is to look inside any wrapper function we have added into OptimysticCallback.

class OptimysticCallback
{
  protected:
    int OrdersTotal()
    {
      //...
    }

What should we write instead the ellipsis?

To get the answer, we need first to recall that we're still speaking about the header file with abstract interfaces. We should not write much in there. All fat implementation details should go to internals of the library, that is to mq4 source and its ex4 compiled release. This means that all routine and scrupulous work of keeping track of virtual orders, prices and other stuff, which is actually forming the simulated trade context, should be packed into another worker class. Let's name it OptimysticTradeContext and define its interface as the complete abstract reflection of protected methods from OptimysticCallback:

class OptimysticTradeContext
{
  public:
    virtual datetime TimeCurrent() = 0;
    virtual double AccountBalance() = 0;

    //...

    virtual int OrdersTotal() = 0;
    virtual int OrderSend(string symbol, int cmd, double volume, double price, int deviation, double stoploss, double takeprofit, string comment = NULL, int magic = 0, datetime expiration = 0, color arrow = clrNONE) = 0;

    //...
};

This is a contract for concrete implemention of our simulated trade context (I mean we will someday create a class, derived from the interface and populate all its method with complete meaning). We should somehow acquire a pointer to such implementation in OptimysticCallback and use it every time when optimization mode is on. For the simplicity, we can use the pointer itself as a flag: if it's empty - real trading is on, and if it's assigned to a virtual implementation - optimization mode is on.

class OptimysticCallback
{
  private:
    OptimysticTradeContext *context;

  protected:
    datetime TimeCurrent()
    {
      return context != NULL ? context.TimeCurrent() : ::TimeCurrent();
    }

    double AccountBalance()
    {
      return context != NULL ? context.AccountBalance() : ::AccountBalance();
    }

    // ...

  public:
    void setContext(OptimysticTradeContext *c)
    {
      context = c;
    }

    OptimysticTradeContext *getContext() const
    {
      return context;
    }

    // ...
};

The implementation of OptimysticTradeContext is hidden inside the library, and the library should call special setContext method to control current trade context. This is the work behind the scenes.

MetaTrader 4 expert adviser on the fly optimization: Optimystic library interface diagram

At the moment we have discussed all classes/interfaces of the library. The code snippets in the blogpost do not include full definitions of the classes - they comprise much more methods wrapping standard functions from trade context, but you may find the complete file Optimystic.mqh attached below.

There is only one last touch that should be made upon the header. As you remember, we have the import directive which binds our factory method with ex4 library file. We can't leave it as is, because we're going to write implemetation for our classes, and the same header file should be included into the library source Optimystic.mq4 as well. But it can't import from itself. To solve the problem we add a macro definition:

#ifndef OPTIMYSTIC_LIBRARY
#import "Optimystic.ex4"
Optimystic *createInstance(OptimysticCallback &callback);
#import
#endif

In EA source code OPTIMYSTIC_LIBRARY is undefined, so the library is imported. In our library surce code we'll define it as simple as:

#define OPTIMYSTIC_LIBRARY

#include <Optimystic.mqh>

and the header will be processed without an issue.

Implementation of the interfaces and example of EA with on the fly optimization will be studied in the part 2.

Table of contents

Files:

Optimystic.mqh 11 kb