Tweaking renko indicators for testing and optimizing robots dependent on them

In the previous blogpost of the series dedicated to renko-driven trading systems we've discovered that MetaTrader 5 does not provide a native support for testing and optimization of trading robots based on renko indicators. Now we'll try to design and implement a workaround to overcome this limitation.

Our purposes are:

• provide a Proof Of Concept (POC) of backtesting renkos without custom symbols;
• identify pros and cons of using indicators vs custom symbols;
• obtain an alternative method of implementation of renko-based trading system which will allows us to compare results from different methods.

The last point is very important, because when you use 3-rd party renko generators or built-in indicators you can get sufficiently different results, no one of which can be treated as a reference point. The only method of validation here is cross-validation of different methods: only similar results obtained from different sources are trustworthy, and can help to select a reliable renko provider.

We'll still use the indicator Blue Renko Bars (fixed and extended) as a ground.

On top of it we'll implement the same test expert adviser MA2Cross using 2 MAs crossing as trade signals. The previous versions of this EA work on custom renko symbols.

Tweaking renko indicator

As we know, number of renko boxes and their position relative to regular bars are changing unsynchronously. In the previous blogpost we've tried 3 methods of handling this in renko indicator: copying boxes in indicator buffers on-the-fly, shifting plots on-the-fly, writing boxes at the beginning of the chart (without copies or shifts). On-the-fly changes can not be detected by MetaTrader's tester, so our last resort is the writing boxes at the beginning.

This mode is enabled in the Blue Renko Bars indicator when parameter Reserve is negative (this indicator was also presented in the previous blogpost and attached here as well).

Even this approach has a flaw: calculated number of renko boxes are not passed correctly into dependent indicators. Just to remind you - MetaTrader always sends a total number of regular bars as rates_total parameter of OnCalculate.

So, first of all, we need to find a way to pass actual number of renko boxes from the underlying Blue Renko Bars indicator to other indicators applied on top of it. In our case this is Moving Average.

You may think to use global variables for this, but we'll utilize the indicator buffer itself. All elements to the right from existing boxes are vacant. Let us write the number of boxes into 0-th (latest) element. Here is a diagram:

rates_total-1 |    ...    | n | ... | 5 | 4 | 3 | 2 | 1 | 0    <- rates_total regular bars
box_total-1   | 3 | 2 | 1 | 0 | ... | . | . | . | . | . | *    <- box_total renko boxes

The top row denotes regular bar indexing (as series, from right to the left). The bottom row holds indexes of renko boxes. They are also indexed from right to the left, but aligned to the left, because number of boxes is usually much less than number of bars. box_total leftmost elements are occupied by valid data, whereas the others are empty (denoted by separate dots). The asterisk at index 0 would be also an empty slot, but we'll write the number box_total into it. Hence it's denoted by *.

This data organization will be used in the tester only! During online trading the renko indicator and all dependent indicators can be updated on-the-fly via ChartSetSymbolPeriod (which does not work in the tester but works online).

Of course, the new meta data analysis must be embedded into any indicator which is used in your trading system on top of the renko. We'll show how to do this by example of moving averages.

In order to flag this new mode and distinguish the box_total number from ordinary value in the indicator buffer, we'll wrap the box_total into Not a Number (explained in the MQL5 programming book).

All helper stuff is available in the file NaNs.mqh. Specifically, we're interested in constants of standard NaN classes (such as quiet NaNs - NAN_QUIET - which do not produce errors) and in conversion the ulong constants into double and vice versa.

#define NAN_QUIET     0x7FF8000000000000
#define NAN_QUIET_1   0x7FF8000000000001
#define NAN_QUIET_2   0x7FF8000000000002

Among other things, it's interested how valid doubles and NaNs are encoded on the bitwise level.

     sign
      |   exponent                       mantissa
 NaN: 0 11111111111 1000000000000000000000000000000000000000000000000000
-Inf: 1 11111111111 0000000000000000000000000000000000000000000000000000
+Inf: 0 11111111111 0000000000000000000000000000000000000000000000000000
-Max: 1 11111111110 1111111111111111111111111111111111111111111111111111
+Max: 0 11111111110 1111111111111111111111111111111111111111111111111111

When the exponent field has all 11 bits set to 1, the number is either Inf (infinite) or NaN. Inf has all bits of the mantissa zero. NaN has at least one bit in the mantissa set to 1. The sign bit retains its normal meaning for Inf but is not meaningful for NaN. As you may see, available number of different NaNs equals to 52-bits integer range which gives 2^52 NaNs and 2^52-1 possible box numbers to encode. That is 4 503 599 627 370 496 (~4.5 Quadrillions), and covers any needs.

In the indicator we encode the size of renkoBuffer into quiet NaN and write it into OpenBuffer[0]. When a new regular bar is added we clear up OpenBuffer[1], replacing the NaN with EMPTY_VALUE.

#include <NaNs.mqh>
   
int OnCalculate(...)
{
   ...
   if(lastBar != time[0]) // new bar added
   {
      if(Reserve < 0)
      {
         OpenBuffer[1] = EMPTY_VALUE; // clear NaN
      }
   }
   ...
   size = ArraySize(renkoBuffer);
   ...
   if(Reserve < 0)
   {
     OpenBuffer[0] = NaNs[NAN_QUIET + size]; // convert quiet NaN constant into double
   }
   
   return rates_total; // (Reserve < 0 ? size + 1 : rates_total);
}

We return rates_total from OnCalculate, because previous straightforward attempt to return the size was tampered by MetaTrader 5 and now we have different method of returning the value.

Tweaking custom indicators

Now we need to adapt other custom indicators for working on top of the renko indicator. Basically we need to read the size from 0-th element, convert it from NaN to the size, and apply instead of rates_total.

To underline the generality of approach we collect these operations in a header file BRB.int.mqh (BRB means Blue Renko Bars).

A wrapper of OnCalculate function does the job and finally calls original OnCalculate function, renamed to brbOnCalculate via define.

int OnCalculate(const int rates_total,
                const int prev_calculated,
                const int begin,
                const double &price[])
{
  ArraySetAsSeries(price, true);
  static bool surrogate = false; 
  // NB: _AppliedTo is not set correctly in tester, always equal 0!
  if(/*_AppliedTo >= 8 &&*/surrogate || !MathIsValidNumber(price[0]))
  {
    surrogate = true;
    // 51 bits (6+ bytes) are available for size, only 4 needed
    const int size = (int)(NaNs[price[0]] - NAN_QUIET); // convert NaN double back into ulong and finally int
    static int prev;
    if(!prev_calculated) prev = 0;
    
    static int prevsize;
    if(size != prevsize)
    {
      // element at the index size does not exist!
      Print(iBars(_Symbol, _Period), " ", size, " ", iTime(_Symbol, _Period, iBars(_Symbol, _Period) - size - 1 + 1), " ",
        DoubleToString(price[iBars(_Symbol, _Period) - size - 1 + 2], _Digits), " ",  // previous 1-st
        DoubleToString(price[iBars(_Symbol, _Period) - size - 1 + 1], _Digits));      // current 0-th
      prevsize = size;
    }
    
    if(!MQLInfoInteger(MQL_TESTER) || MQLInfoInteger(MQL_VISUAL_MODE))
    {
      // output rightmost valid element
      Comment(TimeCurrent(), " ", iBars(_Symbol, _Period), " ", size, " ", iTime(_Symbol, _Period, iBars(_Symbol, _Period) - size));
    }
    
    ArraySetAsSeries(price, false); // restore default indexing
    prev = brbOnCalculate(size, prev, begin, price);
    return prev;
  }
  else
  {
    // Comment(price[10000000]); // throw an exception and force unload
  }
  
  ArraySetAsSeries(price, false); // restore default indexing
  return brbOnCalculate(rates_total, prev_calculated, begin, price);
}
   
#define OnCalculate brbOnCalculate

Also some debug prints are added, for example, for every new renko box formation.

Having this file, create a custom moving average indicator from Examples/Custom Moving Average.mq5, provided with MetaTrader 5 installation. New indicator name is CMA.brb.mq5, it should be placed in the same directory MQL5/Indicators/Examples/, as well as abovementioned mqh-file.

#include "BRB.int.mqh" // helper worker

// MQL5 does not allow properties in includes, so need to repeat them here
#property indicator_chart_window
#property indicator_buffers 1
#property indicator_plots   1
#property indicator_type1   DRAW_LINE
#property indicator_color1  Red
#property indicator_width1  1

#include "Custom Moving Average.mq5" // original indicator "as is"

Unfortunately, MetaTrader 5 does not respect properties from included files, even if it's mq5-file (as in our case). This is why we need to duplicate all the properties from Custom Moving Average.mq5 in CMA.brb.mq5.

After compilation of both indicators we can apply them one by one on a chart. When applying CMA.brb don't forget to visit Parameters tab and choose Apply to First indicator data. Here is what we get.

Blue Renko Bars indicator and CMA.brb applied on top of it

Please note that this is the very beginning of the chart, not the rightmost ending. You will probably need to disable automatic chart scrolling to the end.

In the main window there is a comment which outputs debug information: 2024.03.05 13:14:56 is current time of the chart, and 2022.08.01 13:00:00 is the timestamp of the bar where the latest renko box is mapped. 77 boxes are generated, 10000 regular bars are shown.

In the subwindow we see a strange large number followed by 3 zeros: they are OHLC values for nonexisting box which is mapped to the latest regular bar, and the strange large number is the NaN - unfortunately MetaTrader 5 does not display it as NaN, despite the fact that it recognizes +Inf and -Inf normally.

Adapting EA for the renko indicator

Now we can implement a signal module based on CMA.brb indicator. It's attached as Signal2MACrossCustom.mqh. The name of custom indicator is passed to constructor, but currently the method InitMAs is only applicable for CMA.brb (or custom MAs with the same inputs) - you'll need to adjust it for other indicators.

class Signal2MACross : public CExpertSignal
{
  protected:
    const string      m_custom;
    CiCustom          m_maSlow;         // object-indicator
    CiCustom          m_maFast;         // object-indicator
    ...
  public:
    Signal2MACross::Signal2MACross(const string name) : m_custom(name),...
    {
    }
};
...
bool Signal2MACross::InitMAs(CIndicators *indicators)
{
  ...
  MqlParam params_fast[5] =
  {
    {TYPE_STRING, 0, 0.0, m_custom},
    {TYPE_INT, m_fast, 0.0, NULL},
    {TYPE_INT, m_shift, 0.0, NULL},
    {TYPE_INT, m_method, 0.0, NULL},
    {TYPE_INT, m_type, 0.0, NULL}
  };
  MqlParam params_slow[5] =
  {
    {TYPE_STRING, 0, 0.0, m_custom},
    {TYPE_INT, m_slow, 0.0, NULL},
    {TYPE_INT, m_shift, 0.0, NULL},
    {TYPE_INT, m_method, 0.0, NULL},
    {TYPE_INT, m_type, 0.0, NULL}
  };
  // initialize object
  if(!m_maFast.Create(m_symbol.Name(), m_period, IND_CUSTOM, 5, params_fast)
  || !m_maSlow.Create(m_symbol.Name(), m_period, IND_CUSTOM, 5, params_slow))
  {
    printf(__FUNCTION__ + ": error initializing object");
    return(false);
  }
  ...
}

The groundbreaking changes are taking place in the virtual method StartIndex.

class Signal2MACross : public CExpertSignal
{
  ...
    int StartIndex(void) override
    {
      static double value[1] = {};
      static bool surrogate = false;
      int size = -1;
      if(m_type >= 10 && CopyBuffer(m_type, 0, 0, 1, value) == 1 && (surrogate || !MathIsValidNumber(value[0])))
      {
        size = (int)(NaNs[value[0]] - NAN_QUIET);
        surrogate = true;
      }
      
      const int base = size > -1 ? iBars(_Symbol, _Period) - size : 0;
      
      return((m_every_tick ? base : base + 1));
    }
  ...
};

Here we extract actual box number from a NaN, received from the buffer of specific indicator, identified by the handle m_type.

It's assigned in MA2CrrossInd.mq5, in the same way as it was in the previous part. The only difference is that now we use a signal based on the new custom indicator.

int OnInit()
{
  ...
  const int handle = iCustom(_Symbol, _Period, "Blue Renko Bars", BrickSize, ShowWicks, TotalBars, SwapOpenClose, -MQLInfoInteger(MQL_TESTER));
  if(handle == INVALID_HANDLE)
  {
    Print("Can't create indicator, ", _LastError);
    return(INIT_FAILED);
  }
  ChartIndicatorAdd(0, 1, handle);
  ...
  // Create filter Signal2MACross based on custom CMA.brb indicator
  Signal2MACross *filter0 = new Signal2MACross("Examples/CMA.brb");
  if(filter0 == NULL)
  {
    printf(__FUNCTION__ + ": error creating filter0");
    ExtExpert.Deinit();
    return(INIT_FAILED);
  }
  
  // Set filter parameters
  filter0.SlowPeriod(Signal_2MACross_SlowPeriod);
  filter0.FastPeriod(Signal_2MACross_FastPeriod);
  filter0.MAMethod(Signal_2MACross_MAMethod);
  filter0.MAPrice((ENUM_APPLIED_PRICE)handle); // <- renko indicator goes to the signal
  filter0.Shift(Signal_2MACross_Shift);
  filter0.Weight(Signal_2MACross_Weight);
  ...
}

Please note how MQL_TESTER flag, if present, enables the mode with negative Reserve, which is required for the POC to work in the tester.

Backtests

The original expert adviser MA2Cross, based on renko custom symbol, and new expert adviser MA2CrossInd, based on renko indicator, have been tested on the same period and settings. Here are the results.

Backtest of 2 MA crossing EA implemented on custom renko chart

Backtest of 2 MA crossing EA implemented on custom renko indicators

As you may see, the results are almost identical. This is a good reason to believe that they are close to reality and both methods provide sufficient reliability.

Which one to choose?

Comparison

According to the logs indicator-based implementation works much slower than based on custom symbol.

// renko custom symbol
EURUSD_r100,M1: 19903296 ticks, 3230 bars generated. Environment synchronized in 0:00:00.076.
                Test passed in 0:00:24.097 (including ticks preprocessing 0:00:01.906).
EURUSD_r100,M1: total time from login to stop testing 0:00:24.173 (including 0:00:00.076 for history data synchronization)
898 Mb memory used including 23 Mb of history data, 448 Mb of tick data
// renko indicator
EURUSD,M1: 19894688 ticks, 403001 bars generated. Environment synchronized in 0:00:00.073.
                Test passed in 0:04:13.394 (including ticks preprocessing 0:00:02.812).
EURUSD,M1: total time from login to stop testing 0:04:13.467 (including 0:00:00.073 for history data synchronization)
958 Mb memory used including 45 Mb of history data, 448 Mb of tick data

This is easily explainable: the custom symbol provides renko boxes as ready-made data, whereas indicator calculates them on-the-fly.

Moreover, every access to indicator data adds overheads. When you apply 2 MAs on renko chart, there is only 2 buffers to read, but when you construct 4 buffers of renko indicator and then apply 2 MAs on top of it, you're definitely increasing data exchanges at least twice as much.

All in all we can sum up characteristics in the table.

	Pros	Cons
Indicators	Does not require standalone renko generator Settings (such as renko size) can be changed on the fly	Relatively slow calculations during testing Modification of indicators required Renko analysis is hard to do
Custom symbols	Faster testing Any indicator can be applied on renko without modifications Technical analysis in conventional way	Requires to generate custom symbol beforehand Renko settings can not be changed on the fly