MQL's OOP notes: On The Fly Self-Optimization of Expert Advisers: part 2

14 November 2016, 14:23
Stanislav Korotky
5
55
In the part 1 we have developed a set of programming interfaces (abstract classes) constituting Optimystic library. In this part we'll implement all classes and embed the library into example EA.

The library source code resides in the file Optimystic.mq4, and it's currently almost empty: in the part 1 we've left it on:

#property library
#property strict

#define OPTIMYSTIC_LIBRARY

#include <Optimystic.mqh>

The only function that is exported from the library is the factory:

Optimystic *createInstance(OptimysticCallback &callback) export
{
  return NULL; // ???
}

It should return an object which implements Optimystic interface. Let's start coding it.

class OptimysticImplementation: public Optimystic
{
  private:
    OptimysticCallback *callback;
    
    bool enabled;
    int rangeInBars;
    int resetInBars;
    PERFORMANCE_ESTIMATOR estimator;
    double deposit;
    int spread;
    double commission;
    
  public:
    OptimysticImplementation(OptimysticCallback &cb)
    {
      callback = GetPointer(cb);
    }

    virtual void setEnabled(const bool _enabled)
    {
      enabled = _enabled;
    }

    virtual OptimysticParameter * addParameter(const string name, const double min, const double max, const double step)
    {
      return NULL;
    }

    virtual OptimysticParameter *getParameter(const int i) const
    {
      return NULL;
    }
    
    virtual void setHistoryDepth(const int size, const TIME_UNIT unit)
    {
      if(unit == UNIT_DAY)
      {
        rangeInBars = size * (PeriodSeconds(PERIOD_D1) / PeriodSeconds());
      }
      else
      {
        rangeInBars = size;
      }
    }
    
    virtual void setReoptimizationPeriod(const int size, const TIME_UNIT unit)
    {
      if(unit == UNIT_DAY)
      {
        resetInBars = size * (PeriodSeconds(PERIOD_D1) / PeriodSeconds());
      }
      else
      {
        resetInBars = size;
      }
    }
    
    virtual void setOptimizationCriterion(const PERFORMANCE_ESTIMATOR type)
    {
      estimator = type;
    }

    virtual void setDeposit(const double d)
    {
      deposit = d;
    }
    
    virtual void setSpread(const int points = 0)
    {
      spread = points;
    }
    
    virtual void setCommission(const double c)
    {
      commission = c;
    }
    
    virtual void onTick()
    {
    }
    
    virtual double getOptimalPerformance(PERFORMANCE_ESTIMATOR type = ESTIMATOR_DEFAULT)
    {
      return 0;
    }
};

We're currently missing another class - a class with an implementation of OptimysticParameter. Let's code it as well.

class OptimysticParameterImplementation: public OptimysticParameter
{
  protected:
    string name;
    double min;
    double max;
    double step;
    double value;
    double best;
    bool enabled;
  
  public:
    OptimysticParameterImplementation(const string _name, const double _min, const double _max, const double _step)
    {
      name = _name;
      min = MathMin(_min, _max);
      max = MathMax(_max, _min);
      enabled = true;
      step = _step <= 0 ? 1 : _step;
      best = EMPTY_VALUE;
    }
    
    void setValue(const double v)
    {
      value = v;
    }
    
    void increment()
    {
      value += step;
    }
    
    void markAsBest()
    {
      best = value;
    }

    virtual string getName() const
    {
      return name;
    }
    virtual double getMin() const
    {
      return min;
    }
    virtual double getMax() const
    {
      return max;
    }
    virtual double getStep() const
    {
      return step;
    }
    virtual double getValue() const
    {
      return value;
    }
    virtual double getBest() const
    {
      return best;
    }
    virtual bool getEnabled() const
    {
      return enabled;
    }
    virtual void setEnabled(const bool e)
    {
      enabled = e;
    }
};

To store array of parameters inside OptimysticImplementation I'll use helper class RubbArray (it was described in the blog ealier, but also attached below).

class OptimysticImplementation: public Optimystic
{
  private:
    RubbArray<OptimysticParameterImplementation> parameters;
    int parameterSpace[];
    int parameterCursor[];
    double best;

Now we can complete implementation of the methods addParameter and getParameter:

    virtual OptimysticParameter * addParameter(const string name, const double min, const double max, const double step)
    {
      parameters << new OptimysticParameterImplementation(name, min, max, step);
      return parameters.top();
    }

    virtual OptimysticParameter *getParameter(const int i) const
    {
      return parameters[i];
    }

Now we can focus on the main method onTick. We can't write it at once, so start with a schematic draft. 

    virtual void onTick()
    {
      static datetime lastBar;
      if(enabled)
      {
        if(lastBar != Time[0] && TimeCurrent() > lastOptimization + PeriodSeconds() * resetInBars)
        {
          callback.setContext(context); // enable virtual trading
          optimize();
        }
      }
      callback.setContext(NULL);   // enable real trading
      callback.trade(0, Ask, Bid); // actual trading
      lastBar = Time[0];
    }

This will not compile, because some things here are undefined yet. For example, we need a private variable lastOptimization in the class to store a time of last optimization. We will assign it in the private method optimize, which should be also added to perform actual work. And we don't yet know what is the context. Let's fix the first 2 points:

  private:
    datetime lastOptimization;

  protected:
    bool optimize()
    {
      // ...
      lastOptimization = TimeCurrent();
      return true;
    }

As for the context, it should be a variable with implementation of the interface OptimysticTradeContext, which is left the only unimplemented one.

class OptimysticTradeContextImplementation: public OptimysticTradeContext
{
  // ...
};

It must contain a lot of methods. You can just copy and paste the declaration of the interface into the class and code all methods scrupulously, one by one. Here, in the blog, it's not possible to cover all the changes, so please consult with the attached source code to get the final picture.

One important thing which should be noted regarding implementation of any abstract interface is that it does normally require (a lot of) underlying (hidden) code. In this case, we'are implementing a back-testing engine, where current time is changing according to our own rules. This is why we need a variable holding a bar in past, which looks like current time for an EA being optimized. And we need a method to change the time.

class OptimysticTradeContextImplementation: public OptimysticTradeContext
{
  private:
    int bar;

  public:
    OptimysticTradeContextImplementation()
    {
    }
    
    void setBar(const int b)
    {
      bar = b;
    }

If you remember, we have a method optimize in OptimysticImplementation. This is where we place a cycle through bars on backtest period, and it will call setBar. Something like this:

class OptimysticImplementation: public Optimystic
{
  protected:
    bool optimize()
    {
      if(!callback.onBeforeOptimization()) return false;
      Print("Starting on-the-fly optimization at ", TimeToString(Time[rangeInBars]));
      
      if(spread == 0) spread = (int)MarketInfo(Symbol(), MODE_SPREAD);
      
      /* TODO: init parameters */
      do
      {
        if(!callback.onApplyParameters()) return false;
        
        // build balance and equity curves on the fly
        for(int i = rangeInBars; i >= 0; i--)
        {
          context.setBar(i);
          
          callback.trade(i, iOpen(Symbol(), Period(), i) + spread * Point, iOpen(Symbol(), Period(), i));
        }
        
        // TODO: calculate performance
        
        callback.onReadyParameters();
      }
      while(/* TODO: loop through possible parameter combinations */);

      callback.onAfterOptimization();
      
      lastOptimization = TimeCurrent();
      return true;
    }

Having the current bar in the context we can implement specific methods, for example, TimeCurrent:

class OptimysticTradeContextImplementation: public OptimysticTradeContext
{
  public:
    virtual datetime TimeCurrent()
    {
      return iTime(::Symbol(), ::Period(), bar);
    }

This is the first method of the abstract interface OptimysticTradeContext which got a concrete implementation in OptimysticTradeContextImplementation. Some of the other methods can be implemented in the similar way:

    virtual double iOpen(string symbol, int tf, int offset)
    {
      int b = bar + offset;
      if(symbol != Symbol())
      {
        b = ::iBarShift(symbol, tf, Time[b]);
      }
      return ::iOpen(symbol, tf, b);
    }

But the others require additional work to be done. For example, before we can create a new order with OrderSend we surely need to define a special class.

class Order
{
  private:
    static int tickets;
    
  public:
    
    int type;
    string symbol;
    double openPrice;
    datetime openTime;
    double lots;
    double takeProfit;
    double stopLoss;
    string comment;
    datetime expiration;
    int magicNumber;
    color arrow;

    int ticket;

    double closePrice;
    datetime closeTime;

    
    Order(datetime now, string s, int cmd, double volume, double price, int deviation, double stoploss, double takeprofit, string cmnt = NULL, int magic = 0, datetime exp = 0, color clr = clrNONE)
    {
      symbol = s;
      type = cmd;
      lots = volume;
      openPrice = price;
      openTime = now;
      stopLoss = stoploss;
      takeProfit = takeprofit;
      comment = cmnt;
      magicNumber = magic;
      expiration = exp;
      arrow = clr;
      ticket = ++tickets;
    }

The context should maintain an array of all market and history orders. So let's add them using RubbArray again:

class OptimysticTradeContextImplementation: public OptimysticTradeContext
{
  private:
    RubbArray<Order> orders;
    RubbArray<Order> history;

And implement OrderSend:

    virtual int OrderSend(string symbol, int cmd, double volume, double price, int deviation, double stoploss, double takeprofit, string comment = NULL, int magic = 0, datetime expiration = 0, color arrow = clrNONE)
    {
      Order *order = new Order(TimeCurrent(), symbol, cmd, volume, price, deviation, stoploss, takeprofit, comment, magic, expiration, arrow);
      orders << order;
      return order.ticket;
    }

To close an order we could code something like this:

    virtual bool OrderClose(int ticket, double lots, double price, int slippage, color arrow = clrNONE)
    {
      Order *order = findByTicket(ticket, orders);
      if(order != NULL && (order.type == OP_BUY || order.type == OP_SELL))
      {
        orders >> ticket; // remove from open orders
        history << order; // and add it to the history

        order.closePrice = price;
        order.closeTime = TimeCurrent();

        // TODO: calculate profit and other statistics
        
        return true;
      }
      return false;
    }

We use new private helper here - findByTicket. It's code is more or less obvious:

    Order * findByTicket(int &ticket, RubbArray<Order> &array)
    {
      int n = array.size();
      for(int i = 0; i < n; i++)
      {
        Order *order = array[i];
        if(order.ticket == ticket)
        {
          ticket = i;
          return order;
        }
      }
      return NULL;
    }

The calculation of order's profit (that shown as TODO in OrderClose) can be coded inside the Order class itself:

class Order
{
  public:
    double profit(double price, const int spread)
    {
      if(type == OP_BUY || type == OP_SELL)
      {
        if(closePrice != 0) price = closePrice;
        
        double ptv = MarketInfo(symbol, MODE_POINT) * MarketInfo(symbol, MODE_TICKVALUE) / MarketInfo(symbol, MODE_TICKSIZE);
        double pts = (type == OP_BUY ? +1 : -1) * (price - openPrice) / MarketInfo(symbol, MODE_POINT);
        
        return (pts - spread) * ptv * lots;
      }
      return 0;
    }

Now we can calculate profit in OrderClose. The profit should update balance curve, which we don't have yet. Actually we need to collect many more statistics about trading, such as order counter, equity curve, etc. Let's add them to the context:

class OptimysticTradeContextImplementation: public OptimysticTradeContext
{
  private:
    RubbArray<BarState> income;
    double profit;
    double plusProfit;
    double minusProfit;
    int plusCount;
    int minusCount;

Unfortunately we use yet another new class here - BatState. The good news is that this is the last class in our implementation. And it's simple.

class BarState
{
  public:
    datetime timestamp;
    double balance;
    double equity;
    
    BarState(const datetime dt, double b): timestamp(dt), balance(b), equity(b) {}
    BarState(const datetime dt, double b, double e): timestamp(dt), balance(b), equity(e) {}
};

Get back to the context, and we can finally complete OrderClose (only profit-related part is shown):

        double amount = order.profit(price, spread); // TODO: + order.swap(order.closeTime) + order.lots * commission;
        income.top().balance += amount;
        
        if(amount > 0)
        {
          plusProfit += amount;
          plusCount++;
        }
        else
        {
          minusProfit -= amount;
          minusCount++;
        }

You may see how we update balance and the counters on current bar. Of course, they should be initalized somewhere. So we need to add such a method in the context:

class OptimysticTradeContextImplementation: public OptimysticTradeContext
{
  public:
    void initialize(const double d, const int s, const double c)
    {
      deposit = d;
      spread = s;
      commission = c;
      plusProfit = minusProfit = 0;
      plusCount = minusCount = 0;
      income.clear();
      income << new BarState(TimeCurrent(), deposit);
    }

And it should be called from OptimysticImplementation::optimize method before we run the cycle through history bars. All input values are already known there.

        ...
        if(!callback.onApplyParameters()) return false;
        context.initialize(deposit, spread, commission);
        // build balance and equity curves on the fly
        for(int i = rangeInBars; i >= 0; i--)
        {
          context.setBar(i);
          callback.trade(i, iOpen(Symbol(), Period(), i) + spread * Point, iOpen(Symbol(), Period(), i));
          context.activatePendingOrdersAndStops();
          context.calculateEquity();
        }
        context.closeAll();

        // calculate performance
        context.calculateStats();
        callback.onReadyParameters();

In addition to initialize you see a bunch of other new methods called on the context. I hope their names are self-explaining. We need to implement them all, but let us skip details here - you may consult with the source code. The only thing which can fit is a note about income array. We need to add new element into it on every bar, this is why we extend the function setBar a bit:

    void setBar(const int b)
    {
      bar = b;
      income << new BarState(TimeCurrent(), income.top().balance);
    }

This code simply copies last known balance to the new bar, and if some orders will be closed (as you saw in OrderClose), their profit will update the balance.

BTW, we need to implement AccountBalance method in the context (if you remember), and since we have income array, this can be done easily:

    virtual double AccountBalance()
    {
      return income.size() > 0 ? income.top().balance : 0;
    }

As soon as we have total profit and other stats after the loop through the bars, the context can calculate performance indicators. So let's add a method which return them.

    double getPerformance(const PERFORMANCE_ESTIMATOR type = ESTIMATOR_DEFAULT)
    {
      if(type == ESTIMATOR_PROFIT_FACTOR)
      {
        return minusProfit != 0 ? plusProfit / minusProfit : DBL_MAX;
      }
      else if(type == ESTIMATOR_WIN_PERCENT)
      {
        return plusCount + minusCount != 0 ? 100.0 * plusCount / (plusCount + minusCount) : 0;
      }
      else if(type == ESTIMATOR_AVERAGE_PROFIT)
      {
        return plusCount + minusCount != 0 ? AccountBalance() / (plusCount + minusCount) : 0;
      }
      else if(type == ESTIMATOR_TRADES_COUNT)
      {
        return plusCount + minusCount;
      }
      
      // default: (type == ESTIMATOR_DEFAULT || type == ESTIMATOR_NET_PROFIT)
      return AccountBalance() - deposit;
    }

This method can be used to provide implementation for Optimystic::getOptimalPerformance, which is very important for the client EA.

With the help of getPerformance optimization process (in OptimysticImplementation::optimize) can be refined further. 

      best = 0;
      /* TODO: resetParameters(); */
      do
      {
        // loop through bars goes here
        ...
        // calculate performance (please find details in the sources)
        context.calculateStats();
        
        double gain = context.getPerformance(estimator);
        
        // if estimator is better than previous, then save current parameters
        if(gain > best && context.getPerformance(ESTIMATOR_TRADES_COUNT) > 0)
        {
          best = gain;
          /* TODO: storeBestParameters(); call markAsBest for every parameter */
        }
      }
      while(/* TODO: iterateParameters() */);

All calls related to the parameter set enumeration and selection are still marked as TODO. Due to the lack of space their implementation is not covered here and can be studied in the source code.

And that's actually all. The library is ready. It works.

MetaTrader 4 expert adviser on the fly optimization: Optimystic library class diagram 

Nevertheless, the presented version is very minimal and mostly demonstrates OOP power in such complex tasks. The limitations include (but not limited to):
  • incomplete multicurrency support: specifically we need to have separate spread values for every symbol;
  • no swaps calculation; 
  • no fast optimization by means of genetics or other algorithm (so don't specify too many combinations of parameters to check, because straightforward approach is currently used, and it's slow);
  • no validation of function parameters and error generation (such as, invalid stops);
  • tick values are taken for current exchange rates, whereas they should be recalculated using historic quotes;
  • rough price simulation mode (by open prices);
Also please note that our trade context interface does not provide wrappers for indicator functions, such as iMA, iMACD, iCustom, etc. That does not mean that we can't use indicators, but we need to adjust last parameter in every indicator call made in EA (shifting it to a given bar in past). And this is the reason why we do actually pass current bar number to the method OptimysticCallback::trade. Probably you have already noticed this. An example of EA with indicators will follow. Other 2 paramaters of the method trade reproduce current Ask and Bid prices as prices for i-th bar. They are defined in the platform as special variables, this is why the extensively used so far approach with interface methods is not suited.


Expert Adviser Example

Let's embed the library into example EA - MACD Sample from MetaTrader install. The file is attached at the bottom of the page.

After you add the include:

#include <Optimystic.mqh>

Declare your own class derived from OptimysticCallback:

class MyOptimysticCallback: public OptimysticCallback
{
  private:
    void print(const string title);

  public:
    virtual bool onBeforeOptimization();
    virtual bool onApplyParameters();
    virtual void trade(const int bar, const double Ask, const double Bid);
    virtual void onReadyParameters();
    virtual void onAfterOptimization();
};

and implement all the methods. Part of them was already described in the part 1. Most interesting is that code for the method trade should be taken completely from the former OnTick event handler. The lines that invoke indicators should be changed to take specific bar number passed via the first parameter:

void MyOptimysticCallback::trade(const int bar, const double Ask, const double Bid)
{
  double MacdCurrent, MacdPrevious;
  double SignalCurrent, SignalPrevious;
  double MaCurrent, MaPrevious;

  MacdCurrent = iMACD(NULL, 0, 12, 26, 9, PRICE_CLOSE, MODE_MAIN, bar + 0);
  MacdPrevious = iMACD(NULL, 0, 12, 26, 9, PRICE_CLOSE, MODE_MAIN, bar + 1);
  SignalCurrent = iMACD(NULL, 0, 12, 26, 9, PRICE_CLOSE, MODE_SIGNAL, bar + 0);
  SignalPrevious = iMACD(NULL, 0, 12, 26, 9, PRICE_CLOSE, MODE_SIGNAL, bar + 1);
  MaCurrent = iMA(NULL, 0, MATrendPeriod, 0, MODE_EMA, PRICE_CLOSE, bar + 0);
  MaPrevious = iMA(NULL, 0, MATrendPeriod, 0, MODE_EMA, PRICE_CLOSE, bar + 1);

All other calls made in the source to seemingly standard functions are intercepted by OptimysticCallback and redirected to appropriate backend: either the platfrom or our virtual trade context. And even Ask and Bid are replaced with correct values for the bar bar

Another nuance added into the example EA in relation to optimization is automatic disabling of trading for cases when optimal parameter set is not found (specified performance estimator is not positive).

bool TradeEnabled = true;

void MyOptimysticCallback::onAfterOptimization()
{
  if(p.getOptimalPerformance(Estimator) > 0)
  {
    // apply new parameters for real trading
    MACDOpenLevel = p[0].getBest();
    MACDCloseLevel = p[1].getBest();
    MATrendPeriod = (int)p[2].getBest();
    Print("Best parameters: ", (string)MACDOpenLevel, " ", (string)MACDCloseLevel, " ", (string)MATrendPeriod);
    TradeEnabled = true;
  }
  else
  {
    Print("No optimum");
    TradeEnabled = false;
  }
}

Then in the method trade:

void MyOptimysticCallback::trade(const int bar, const double Ask, const double Bid)
{
  // ...
  
  int cnt, ticket, total;

  total = OrdersTotal();
  if(total < 1 && ((getContext() != NULL) || TradeEnabled))
  {

Hence, new orders can be opened only if virtual context is not NULL (that is in optimization mode) or an applicable parameter set exists (TradeEnabled is true in real trading mode). 

Of course, this flag is automatically selected after every optimization batch, that is after trading was temporary disabled, it can be restored automaticaly as soon as suitable parameter set is found during next optimization.

Please, note, that since we need to change input parameters from MQL code they should be defined as extern instead of input. Also when on the fly optimization is enabled, input parameters which undergo optimization specify minimal value and step for optimization (they are the same for simplicity), and their maximal value is 10 times larger the minimal value. As a result, if you set 3 parameters for optimization, they will make 1000 combinations.

And here is the final warning: the MACD Sample was taken as example due to its simplicity and availability, but the EA itself is not very good (for example, stoploss is applied on orders only after they go in sufficient profit, so if price moves in unfavourable direction from very beginning, you'll get large loss), as a result you should not expect much from the EA performance. Try the library with your own EAs.

 

I'm sure there is still much to discuss on the subject, but the blog post can't last for ever, even if it's a very very long story. You may explore the library yourself and try MQL OOP in the action.

 

Table of contents 


Files:
macd.mq4 9 kb
Share it with friends: