MQL's OOP notes: Online Analytical Processing of trading hypercubes: part 1

Before we start we should decide what data to analyse. There are many options to choose from, such as trading statements, optimization reports, custom indicators readings. Actually this is not so important which one to use, because we're going to design a universal object-oriented framework suitable for any purpose. But since we need something to test the framework on, we should elect a concrete task. One of the most popular tasks is analysis of an account trading history. So let's take it.

The blueprint

The record is basically a vector with an arbitrary number of fields. Since this is an abstract thing, we don't need to know what every field means. For every specific use case we can derive a concrete record class which will know every field meaning and process it accordingly.

All the records should be somehow mapped into a meta cube. We don't yet know how this can be done, but this is the essense of the task - to slice and aggregate the data into a generalized view with various meaningful statistics. As an abstraction, the meta cube provides only basic properties such as number of dimensions, their names, and size of each dimension. Descendant classes should fill the cube with specific statistics. For example, we can imagine such aggregators as sum, or average value of selected field in input records. 

In addition, it's sometimes useful to filter out some records. This is why we should provide a filter. It's much like selector with a restriction applied on values (for example, if you have a day of week selector, producing indices 0-6, you can form a filter on it to exclude specific day from calculations).

Finally, there should be a core class which binds all the abovementioned stuff together. Let's call it analyst.

The whole picture looks like this.

The Implementation

It does nothing more than storing values in the data array (vector). We use float type to save memory. Data cubes can be very large, so using float instead of double allows for sparing 50% of memory at the expense of a bit lower accuracy.

The method getNext should be called in a loop until it returns NULL (no more records). Until then all the records should be saved somewhere (see details below). The method reservedSize is provided for optimizing memory allocation.

And if we'd like to analyze MetaTrader's optimization results, we could use the following enumeration.

For every specific use case we should elaborate specific enumeration. Any one of such enumerations should be used as a template paramater of the templatized class Selector.

The field selector will hold specific value - an element of enumeration. For example, if TRADE_RECORD_FIELDS is used, one could create selector for buy/sell trade operations like so:

The field _typename is auxiliary. It will be overriden in derived classes to identify selectors by class names, which can be useful in logs or resulting graphs. The field is used in the virtual method getTitle.

The method getLabel returns a user-friendly description of the given index. For example, for buy/sell operations, index 0 should produce "buy" and index 1 - "sell".

By default, it maps all records into a single cell in the cube. For example, you can get total profit using this selector.

We define the class using FIELD_TYPE element in the base constructor. getRange returns 2 because we have only 2 possible values for the type: OP_BUY or OP_SELL. getMin and getMax return appropriate values. What should we write inside select method?

The helper method fillByOrder demonstrates how most of the fields can be filled from current order. Of course, the order should be previously selected somewhere in the code. The number of fields TRADE_RECORD_FIELDS_NUMBER can be either hardcoded in a macro definition or determined dynamically from TRADE_RECORD_FIELDS enumeration (you may find details in the source codes attached at the end of the story, which continues in the part 2).

Here, we read the field from incoming record and assign its value (which can be OP_BUY or OP_SELL) as the index. Only market orders are counted, so select returns false for any other types. We'll consider some other selectors later.

The adapter just iterates through all orders in the history and creates TradeRecord for every market order. There must be a core class which instantiates the adapter and invokes its getNext method. Moreover, the core class should store the returned records in an internal array. This is how we came to the class Analyst.

The class does not actually instantiate the adapter but accepts it as a parameter of constructor. This is a well-known principle of dependency injection. It allows us to decouple Analyst from concrete implementation of a DataAdapter. In other words, we can exchange different classes for history adapter freely without any modification of Analyst.

As you remember, aggregators are classes capable of calculating specific statistics for selected record fields. And the base class for aggregators is a meta cube - multidimensional array storing statistics. 

The array dimensions describes the structure of the MetaCube. Its size should be equal to number of selectors used, and every element should contain the size of corresponding dimension, which comes from selector range. For example, if we want to see profits by day of week, we should craft a selector returning day indices in range 0-6 according to order open (or close) time. Since this is the only selector we use, dimensions will have 1 element, and its value will be 7. If we want to add another selector - say TypeSelector, described above - in order to see profits splitted both by day of week and operation type, then dimensions will contain 2 elements: 7 and 2. This also means that there should be 14 cells with statistics.

The base class does not resize or initialize the member arrays, because all this stuff will be implemented in derived classes.

The Aggregator should process specific field of records. For example, this can be profit (FIELD_PROFIT_AMOUNT). We'll initialize the variable field in constructor below.

The calculations should be performed in a multidimensional space formed by arbitrary number of selectors. We have considered the example of counting profits splitted by day of week and buy/sell operation, which requires 2 selectors. They should be stored in the array selectors. Selectors themselves are passed into the object again via constructor (see below).

As you remember we have one dimensional array totals as a storage of values in the multidimensional space of selectors. To convert indices of multiple dimensions into a single position in the one dimensional array we use the following helper function.

It accepts an array with indices and returns ordinal position. You'll see how the array offsets is filled a few lines below.

Finally we came to the task of calculating statistics.

This method will be called for array of records. Every record is presented to each selector in the loop, and if it's not rejected by any one of them, then corresponding indices are saved in the local array k. If all selectors determined valid indices for the record, we invoke method update. It accepts an offset of element in the array totals (the offset is calculated by the helper function mixIndex shown above, based on the array of indices), and value from the specified field of the record. If we continue the example of analyzing profits distribution, then the field will be FIELD_PROFIT_AMOUNT and its value comes from OrderProfit, as you remember.

The method is a pure virtual and should be overriden in descendant classes.

The base Aggregator class does almost all the job. Now we can implement many specific aggregators with minimal efforts. But apart from this there is still a lot of work, so let's continue with all the routine tasks in the part 2.