MQL's OOP notes: Online Analytical Processing of trading hypercubes: part 2

12 December 2016, 19:28
Stanislav Korotky
0
92

In the first part we have described common OLAP principles and designed a set of classes bringing hypercube functionality in MetaTrader.

At the moment we have the base Aggregator class, which does almost all the job. Now we can implement many specific aggregators with minimal efforts. For example, to calculate a sum, we define:

template<typename E>
class SumAggregator: public Aggregator<E>
{
  public:
    SumAggregator(const E f, const Selector<E> *&s[]): Aggregator(f, s)
    {
      _typename = typename(this);
    }
    
    virtual void update(const int index, const float value)
    {
      totals[index] += value;
    }
};

And to calculate average, we code:

template<typename E>
class AverageAggregator: public Aggregator<E>
{
  protected:
    int counters[];
    
  public:
    AverageAggregator(const E f, const Selector<E> *&s[]): Aggregator(f, s)
    {
      _typename = typename(this);
    }
    
    virtual void setSelectorBounds()
    {
      Aggregator<E>::setSelectorBounds();
      ArrayResize(counters, ArraySize(totals));
      ArrayInitialize(counters, 0);
    }

    virtual void update(const int index, const float value)
    {
      totals[index] = (totals[index] * counters[index] + value) / (counters[index] + 1);
      counters[index]++;
    }
};

As far as we have aggregators defined now, we can return to the main class Analyst and complement it. 

template<typename E>
class Analyst
{
  private:
    ...
    Aggregator<E> *aggregator;
    
  public:
    Analyst(DataAdapter &a, Aggregator<E> &g): adapter(&a), aggregator(&g)
    {
      ...
    }
    
    void build()
    {
      aggregator.calculate(data);
    }
};

The method build builds meta cube with statistics using records from the data adapter as input.

The only one last thing which is still missing in the class Analyst is a facility for outputting results. It can be extracted to another class - Display. And it's very simple one.

class Display
{
  public:
    virtual void display(MetaCube *metaData) = 0;
};

It contains a pure virtual method accepting meta cube as data source. Specific ways of visualization and probably additional settings should be introduced in derived classes. 

Now we can finalize the class Analyst.

template<typename E>
class Analyst
{
  private:
    ...
    Display *output;
    
  public:
    Analyst(DataAdapter &a, Aggregator<E> &g, Display &d): adapter(&a), aggregator(&g), output(&d)
    {
      ...
    }
    
    void display()
    {
      output.display(aggregator);
    }
};

For testing purposes we need at least one concrete implementation of a Display. Let's code one for printing results in expert logs - LogDisplay. It will loop through entire meta cube and output all values along with corresponding coordinates. Of course, this is not so descriptive as a graph could be. But building various kinds of 2D or 3D graphs is a long story itself, not to mention a diversity of technologies which can be used for graph generation - objects, canvas, Google charts, etc., so put it out of consideration here. You may find complete source codes, including LogDisplay, attached below.

Finally, having everything in stock, we can plot overall workflow as follows:

  • Create HistoryDataAdapter object;
  • Create several specific selectors and place them in an array;
  • Create specific aggregator object, for example SumAggregator, using the array of selectors and a field, which values should be aggregated;
  • Create LogDisplay object;
  • Create Analyst object using the adapter, the aggregator, and the display;
  • Then call successively:

  analyst.acquireData();
  analyst.build();
  analyst.display();

Don't forget to delete the objects at the end.

We'are almost done. There is one small omission that was intentionally made to simplify the text, and this is the time to elaborate on the matter.

All selector we have discussed so far have had constant ranges. For example, there exist only 7 week days, and market orders are always either buy or sell. But what if the range is not known beforehand? This is a quite often situation.

It's perfectly legal to ask metacube for profits splitted by work symbols, or by magic number (expert advisers). To address such technical requirement, we should stack all unique symbols or magic numbers in an internal array, and then use its size as the range for corresponding selector. 

Let's create a dedicated class Vocabulary for managing such internal arrays and demonstrate how it is used in conjunction with, say, SymbolSelector.

The Vocabulaty implementation is straightforward.

template<typename T>
class Vocabulary
{
  protected:
    T index[];

We reserved the array index for unique values.

  public:
    int get(const T &text) const
    {
      int n = ArraySize(index);
      for(int i = 0; i < n; i++)
      {
        if(index[i] == text) return i;
      }
      return -(n + 1);
    }

We can check if a value is already in the index. If it is, the method get returns existing index. If it's not, the method returns new required size, negated. This is done so for a small optimization used in the following method adding new value into the index.

    int add(const T text)
    {
      int n = get(text);
      if(n < 0)
      {
        n = -n;
        ArrayResize(index, n);
        index[n - 1] = text;
        return n - 1;
      }
      return n;
    }

Finally we should provide methods to get total size of the index and retrieve its values.

    int size() const
    {
      return ArraySize(index);
    }
    
    T operator[](const int position) const
    {
      return index[position];
    }
};

As far as work symbols belong to orders, we embed the vocabulary into TradeRecord.

class TradeRecord: public Record
{
  private:
    ...
    static Vocabulary<string> symbols;

  protected:
    void fillByOrder(const double balance)
    {
      ...
      set(FIELD_SYMBOL, symbols.add(OrderSymbol())); // symbols are stored as indices from vocabulary
    }

  public:
    static int getSymbolCount()
    {
      return symbols.size();
    }
    
    static string getSymbol(const int index)
    {
      return symbols[index];
    }
    
    static int getSymbolIndex(const string s)
    {
      return symbols.get(s);
    }

Now we can implement SymbolSelector.

class SymbolSelector: public TradeSelector
{
  public:
    SymbolSelector(): TradeSelector(FIELD_SYMBOL)
    {
      _typename = typename(this);
    }
    
    virtual bool select(const Record *r, int &index) const
    {
      index = (int)r.get(selector);
      return (index >= 0);
    }
    
    virtual int getRange() const
    {
      return TradeRecord::getSymbolCount();
    }
    
    virtual string getLabel(const int index) const
    {
      return TradeRecord::getSymbol(index);
    }
};

The selector for magic numbers is implemented in a similar way. If you look into the sources you may see that the following selectors are provided: TypeSelector, WeekDaySelector, DayHourSelector, HourMinuteSelector, SymbolSelector, SerialNumberSelector, MagicSelector, ProfitableSelector. Also the following aggregators are available: SumAggregator, AverageAggregator, MaxAggregator, MinAggregator, CountAggregator.

That's all. The meta cube is ready.

Please note, that for simplicity we skipped some nuances of implementation of the classes. For example, we omitted all details about filters, which were announced in the blueprint (in the part 1). Also we needed to cumulate profits and losses in a special balance variable in order to calculate the field FIELD_PROFIT_PERCENT. It was not described in the text. You may find full source codes in the file OLAPcube.mqh attached below.

It's time to try the meta cube in action.  


The Example

Let's create a non-trading expert adviser capable of performing OLAP on MetaTrader's account history (the complete source code - THA.mq4 -  is attached at the bottom).

#property strict

#include <OLAPcube.mqh>

Although meta cube can handle any number of dimensions, let's limit it to 3 dimensions for simplicity. This means that we should allow up to 3 selectors. Types of supported selectors are listed in the enumeration:

enum SELECTORS
{
  SELECTOR_NONE,       // none
  SELECTOR_TYPE,       // type
  SELECTOR_SYMBOL,     // symbol
  SELECTOR_SERIAL,     // ordinal
  SELECTOR_MAGIC,      // magic
  SELECTOR_PROFITABLE, // profitable
  /* all the next require a field as parameter */
  SELECTOR_WEEKDAY,    // day-of-week(datetime field)
  SELECTOR_DAYHOUR,    // hour-of-day(datetime field)
  SELECTOR_HOURMINUTE, // minute-of-hour(datetime field)
  SELECTOR_SCALAR      // scalar(field)
};

It is used to define corresponding inputs:

sinput string X = "————— X axis —————";
input SELECTORS SelectorX = SELECTOR_SYMBOL;
input TRADE_RECORD_FIELDS FieldX = FIELD_NONE /* field does matter only for some selectors */;

sinput string Y = "————— Y axis —————";
input SELECTORS SelectorY = SELECTOR_NONE;
input TRADE_RECORD_FIELDS FieldY = FIELD_NONE;

sinput string Z = "————— Z axis —————";
input SELECTORS SelectorZ = SELECTOR_NONE;
input TRADE_RECORD_FIELDS FieldZ = FIELD_NONE;

The filter is going to be only one (though it's possible to have more), and it's off by default.

sinput string F = "————— Filter —————";
input SELECTORS Filter1 = SELECTOR_NONE;
input TRADE_RECORD_FIELDS Filter1Field = FIELD_NONE;
input float Filter1value1 = 0;
input float Filter1value2 = 0;

Also supported aggregators are listed in another enumeration:

enum AGGREGATORS
{
  AGGREGATOR_SUM,      // SUM
  AGGREGATOR_AVERAGE,  // AVERAGE
  AGGREGATOR_MAX,      // MAX
  AGGREGATOR_MIN,      // MIN
  AGGREGATOR_COUNT     // COUNT
};

And it's used for setting up work aggregator:

sinput string A = "————— Aggregator —————";
input AGGREGATORS AggregatorType = AGGREGATOR_SUM;
input TRADE_RECORD_FIELDS AggregatorField = FIELD_PROFIT_AMOUNT;

All selectors, including the one which is used for optional filter, are initialized in OnInit.

int selectorCount;
SELECTORS selectorArray[4];
TRADE_RECORD_FIELDS selectorField[4];

int OnInit()
{
  selectorCount = (SelectorX != SELECTOR_NONE) + (SelectorY != SELECTOR_NONE) + (SelectorZ != SELECTOR_NONE);
  selectorArray[0] = SelectorX;
  selectorArray[1] = SelectorY;
  selectorArray[2] = SelectorZ;
  selectorArray[3] = Filter1;
  selectorField[0] = FieldX;
  selectorField[1] = FieldY;
  selectorField[2] = FieldZ;
  selectorField[3] = Filter1Field;

  EventSetTimer(1);
  return(INIT_SUCCEEDED);
}

OLAP is performed only once in OnTimer event.

void OnTimer()
{
  process();
  EventKillTimer();
}

void process()
{
  HistoryDataAdapter history;
  Analyst<TRADE_RECORD_FIELDS> *analyst;
  
  Selector<TRADE_RECORD_FIELDS> *selectors[];
  ArrayResize(selectors, selectorCount);
  
  for(int i = 0; i < selectorCount; i++)
  {
    selectors[i] = createSelector(i);
  }
  Filter<TRADE_RECORD_FIELDS> *filters[];
  if(Filter1 != SELECTOR_NONE)
  {
    ArrayResize(filters, 1);
    Selector<TRADE_RECORD_FIELDS> *filterSelector = createSelector(3);
    if(Filter1value1 != Filter1value2)
    {
      filters[0] = new FilterRange<TRADE_RECORD_FIELDS>(filterSelector, Filter1value1, Filter1value2);
    }
    else
    {
      filters[0] = new Filter<TRADE_RECORD_FIELDS>(filterSelector, Filter1value1);
    }
  }
  
  Aggregator<TRADE_RECORD_FIELDS> *aggregator;
  
  // MQL does not support a 'class info' metaclass.
  // Otherwise we could use an array of classes instead of the switch
  switch(AggregatorType)
  {
    case AGGREGATOR_SUM:
      aggregator = new SumAggregator<TRADE_RECORD_FIELDS>(AggregatorField, selectors, filters);
      break;
    case AGGREGATOR_AVERAGE:
      aggregator = new AverageAggregator<TRADE_RECORD_FIELDS>(AggregatorField, selectors, filters);
      break;
    case AGGREGATOR_MAX:
      aggregator = new MaxAggregator<TRADE_RECORD_FIELDS>(AggregatorField, selectors, filters);
      break;
    case AGGREGATOR_MIN:
      aggregator = new MinAggregator<TRADE_RECORD_FIELDS>(AggregatorField, selectors, filters);
      break;
    case AGGREGATOR_COUNT:
      aggregator = new CountAggregator<TRADE_RECORD_FIELDS>(AggregatorField, selectors, filters);
      break;
  }
  
  LogDisplay display;
  
  analyst = new Analyst<TRADE_RECORD_FIELDS>(history, aggregator, display);
  
  analyst.acquireData();
  
  Print("Symbol number: ", TradeRecord::getSymbolCount());
  for(int i = 0; i < TradeRecord::getSymbolCount(); i++)
  {
    Print(i, "] ", TradeRecord::getSymbol(i));
  }

  Print("Magic number: ", TradeRecord::getMagicCount());
  for(int i = 0; i < TradeRecord::getMagicCount(); i++)
  {
    Print(i, "] ", TradeRecord::getMagic(i));
  }
  
  analyst.build();
  analyst.display();
  
  delete analyst;
  delete aggregator;
  for(int i = 0; i < selectorCount; i++)
  {
    delete selectors[i];
  }
  for(int i = 0; i < ArraySize(filters); i++)
  {
    delete filters[i].getSelector();
    delete filters[i];
  }
}

The helper function createSelector is defined as follows.

Selector<TRADE_RECORD_FIELDS> *createSelector(int i)
{
  switch(selectorArray[i])
  {
    case SELECTOR_TYPE:
      return new TypeSelector();
    case SELECTOR_SYMBOL:
      return new SymbolSelector();
    case SELECTOR_SERIAL:
      return new SerialNumberSelector();
    case SELECTOR_MAGIC:
      return new MagicSelector();
    case SELECTOR_PROFITABLE:
      return new ProfitableSelector();
    case SELECTOR_WEEKDAY:
      return new WeekDaySelector(selectorField[i]);
    case SELECTOR_DAYHOUR:
      return new DayHourSelector(selectorField[i]);
    case SELECTOR_HOURMINUTE:
      return new DayHourSelector(selectorField[i]);
    case SELECTOR_SCALAR:
      return new TradeSelector(selectorField[i]);
  }
  return NULL;
}

All the classes comes from the header file.

If you run the EA on a real account and choose to use 2 selectors - SymbolSelector and WeekDaySelector - you can get results in the log, something like this: 

Symbol number: 4
0] XAUUSD
1] XAGUSD
2] USDRUB
3] CADJPY
Magic number: 1
0] 0
FIELD_PROFIT_AMOUNT [28]
X: SymbolSelector(FIELD_SYMBOL)
Y: WeekDaySelector(FIELD_DATETIME2)
indices: 4 7
Filters: 
0: XAUUSD Sunday
0: XAGUSD Sunday
0: USDRUB Sunday
0: CADJPY Sunday
-2.470000028610229: XAUUSD Monday
3: XAGUSD Monday
-7.600001335144043: USDRUB Monday
0: CADJPY Monday
0: XAUUSD Tuesday
0: XAGUSD Tuesday
-4.25: USDRUB Tuesday
0: CADJPY Tuesday
-1.470000028610229: XAUUSD Wednesday
4.650000095367432: XAGUSD Wednesday
-1.950000107288361: USDRUB Wednesday
0: CADJPY Wednesday
8.590000152587891: XAUUSD Thursday
-15.44999980926514: XAGUSD Thursday
-29.97999978065491: USDRUB Thursday
3.369999885559082: CADJPY Thursday
52.7700012922287: XAUUSD Friday
-76.20000553131104: XAGUSD Friday
12.42000007629395: USDRUB Friday
0: CADJPY Friday
0: XAUUSD Saturday
0: XAGUSD Saturday
0: USDRUB Saturday
0: CADJPY Saturday

In this case, 4 work symbols were traded in the account. The meta cube size is therefore 28. All combinations of work symbols and days of week are listed along with corresponding profit/loss values. Please note, that WeekDaySelector requires a field to be specified explicitly, because every trade can be logged either by open datetime (FIELD_DATETIME1) or close datetime (FIELD_DATETIME2). In this case we used FIELD_DATETIME2.

 

Table of contents 

Files:
THA.mq4 7 kb
OLAPcube.mqh 23 kb
Share it with friends: