Data Exchange between Indicators: It's Easy

Introduction

Why we appreciate newbies, is because they are stubbornly unwilling to use search, and there are many topics like "FAQ", "For newbies" or "The one who asks a question from this list, will burn in hell". Their true mission is to ask questions, like "How to...", "Is it possible to..." and often get answers like "No way", "It's impossible".

The eternal phrase "never say never again" has been stimulating scientists, engineers and programmers to think and to create something new well-forgotten old.

1. Problem Definition

For example, here is a quote from one of topics on the MQL4 Community forum (translated from Russian):

 ...there are two indicators (let's call them A and B). The indicator A uses data directly from the price chart as usual, and B uses data of the indicator A.  Here is the question: What should I do to perform the calculation of B dynamically, using the data of indicator A (which is already attached), instead of using iCustom("indicator A", ...), i.e. if I change some settings of the Indicator A, it should be reflected in the parameter changes of the indicator B.

or in other wording:

...Suppose that we have attached a Moving Average to a chart. Now how to get the direct access to its data buffer?

and this:

... If I call some indicator using iCustom, it reloads even if it has been loaded before the call. Is there any way to prevent it?

The list of such questions can be extended, they are asked again and again, and not only by newbies. Generally, the problem is that MetaTrader has no technique to access to the custom indicator data without the use of iCustom (MQL4) or use of the iCustom + CopyBuffer binding (MQL5). But it would be so tempting for the next masterpiece development to write some functions in MQL code, which will get data or calculate something using data from a specified chart.

It isn't convenient to work through the above mentioned standard functions: for example, in MQL4, the copies of the indicator are created for each caller when iCustom is called; in MQL5 the problem was partly solved by using handles, and now calculations are performed only once for a master copy. Still it's not a panacea: if the indicator, to which we refer, consumes a lot of computation resources, the terminal starvation is virtually guaranteed by a dozen of seconds at each reinitialization.

I remember several offered ways to access to the origin, including the following ones:

• mapping of files on a physical disk or memory;
• shared memory use via DLL for data exchange;
• the use of global variables of the client terminal for data exchange and their storage.

And some others methods, which are variations of the above mentioned ones, as well as some exotic ways like sockets, mailslots, etc. (there is a radical method, which is often used in Expert Advisors - the transfer of indicator calculations directly to the Expert Advisor code, but this is well beyond the scope of our article.

In the author's viewpoint, there is something in all these methods, but all they have a common disadvantage: data are first copied into some location and then distributed to others. Firstly it requires some CPU resources, and secondly, it creates a new problem with the relevance of data transmitted (we will not consider it here).

So, let's try to define the problem:

We want to create such an environment, which would provide access to data of indicators attached to a chart, and would have the following properties:

• absence of data copying (and of the problem with its relevance);
• minimal modification of the code of available methods, if we need to use them;
• MQL code is preferable (of course, we have to use DLL, but we will use just a dozen of strings of C++ code).

The author used C++ Builder for DLLs creation and the MetaTrader 4 and MetaTrader 5 client terminals. Source codes presented below are written in MQL5, the MQL4 codes are attached to the article, the main differences between them will be discussed.

2. Arrays

First, we need some theory, because we are going to work with indicator buffers, and we have to know how their data are located in the memory. This information is not documented properly.

The dynamic arrays in MQL are structures with variable size, so it's interesting how MQL solves the data reallocation problem, if the array size has increased and there isn't free memory right after the array. There are two ways to solve it:

1. to reallocate new data in the additional part of available memory (and save the addresses of all parts for this array, for example using a referenced list), or
   
2. to move the entire array as a whole to the new part of memory, which is sufficient to allocate it.
   

The first method leads to some additional problems, because in this case we have to investigate data structures created by MQL compiler. The following consideration still testifies to the second variant (which is still slower): When a dynamic array is passed to an external function (to DLL), the latter one gets the pointer to the first element of data, other array elements are ordered and located right after the first array element.

When passing by reference, the array data can be changed, so it means that in the first method there is a problem to copy the entire array into a separate memory area to transfer it to an external function, and then add the result to the source array, i.e. perform the same actions that are implied by the second method.

Though this conclusion is not 100% logically strict, it still can be considered rather reliable (it is also proved by the correct operation of our product based on this idea).

So, we will assume that the following statements are right:

• At each moment of time, data of dynamic arrays are located  sequentially in memory, one after another; and the array can be re-located in other parts of computer memory, but only as a whole.
• The address of the first array element is passed to the DLL library, when a dynamic array is passed into an external function as a parameter by reference.
  

And yet we take some other assumptions: under moments of time we mean the time moments when the OnCalculate() (for MQL4 - start ()) function of the corresponding indicator is called; in addition, for simplicity, we assume that our data buffers have the same dimension and type of double [].

3. Sine Qua Non

MQL doesn't support pointers (except for the so called object pointers, which are not pointers in its common sense), as it has repeatedly stated and confirmed by representatives of MetaQuotes Software. So let's check whether this is true.

What is a pointer? It's not just an identifier with an asterisk, it's the address of a cell in a computer memory. And what is the cell address? It's the sequence number starting from a certain beginning. And finally, what is a number? It's an integer that corresponds to some cell of the computer memory. And why can't we work with a pointer like with the integer number? Yes, we can, because MQL programs perfectly work with integers!

But how to convert a pointer to an integer? The Dynamic Link Library will help us, we will use the C++ typecasting possibilities. Due to the fact that C++ pointers are four-byte data type, in our case it is convenient to use the int as such four-byte data type.

The sign bit is not important and we will not take it into account (if it is equal to 1, it just means that the integer is negative), the main thing is that we can keep all the pointer bits unchanged. Of course, we can use unsigned integers, but it will be better if the codes for MQL5 and MQL4 are similar, because MQL4 doesn't provide unsigned integer.

So,

extern "C" __declspec(dllexport) int __stdcall GetPtr(double *a)
{
        return((int)a);
}

Wow, now we have the variable of long type with the value of the array beginning address! Now we need to learn how to read the value of the i-th array element:

extern "C" __declspec(dllexport) double __stdcall GetValue(int pointer,int i)
{
        return(((double*) pointer)[i]);
}

... and write the value (it isn't really necessary in our case, still ...)

extern "C" __declspec(dllexport) void __stdcall SetValue(int pointer,int i,double value)
{
        ((double*) pointer)[i]=value;
}

So, that's all. Now we can work with pointers in MQL.

4. Wrapping

We have created the kernel of the system, now we have to prepare it for the convenient use in MQL programs. Nevertheless, the aesthetics fans please don't feel upset, there will be something else.

There is a billion of ways to do it. We will choose the following way. Let's remember that the client terminal has a special feature for the data exchange between separate MQL programs - Global Variables. The most natural way is to use them for storing pointers to indicator buffers, which we will access to. We will consider such variables as a table of descriptors. Each descriptor will have the name represented by a string like:

string_identifier#buffer_number#symbol#period#buffer_length#indexing_direction#random_number,

and its value will be equal to the integer representation of the corresponding buffer pointer in computer memory.

Some details about the descriptor fields.

• string_identifier – any string (for example, the name of the indicator - the short_name variable can be used etc.); it will be used for search of the necessary pointer, we mean that some indicator will register the descriptors with the same identifier and will distinguish between them using the field:
• buffer_number – will be used to distinguish the buffers;
• buffer_length – we need it to control the boundaries, otherwise we could lead to the crash of the client terminal and Windows Blue Screen of Death :);
• symbol, period – symbol and period to specify the chart window;
• ordering_direction – it specifies the direction of ordering of the array elements: 0 – usual, 1 – reversal (AS_SERIES flag is true);
• random_number – it will be used if there are several copies of an indicator though with different windows or with different sets of parameters, attached to a client terminal (they can set the same values of the first and second fields, that's why we need to distinguish between them somehow) - maybe it's not the best solution, but it works.

First, we need some functions to register descriptor and to delete it. Take a look at the first string of the function - the call of the UnregisterBuffer() function is necessary to delete old descriptors in the list of global variables.

At each new bar the buffer size will be increased by 1, so we have to call RegisterBuffer(). And if the buffer size has changed, the new descriptor will be created in the table (the information of its size is contained in its name), the old one will stay in the table. That's why it is used.

void RegisterBuffer(double &Buffer[], string name, int mode) export
{
   UnregisterBuffer(Buffer);                    //first delete the variable just in case
   
   int direction=0;
   if(ArrayGetAsSeries(Buffer)) direction=1;    //set the right ordering_direction

   name=name+"#"+mode+"#"+Symbol()+"#"+Period()+"#"+ArraySize(Buffer)+"#"+direction;
   int ptr=GetPtr(Buffer);                      // get the buffer pointer

   if(ptr==0) return;
   
   MathSrand(ptr);                              //it's convenient to use the pointer value instead
                                                //of the current time for the random numbers base 
   while(true)
   {
      int rnd=MathRand();
      if(!GlobalVariableCheck(name+"#"+rnd))    //check for unique name - we assume that 
      {                                         //nobody will use more RAND_MAX buffers :)
         name=name+"#"+rnd;                     
         GlobalVariableSet(name,ptr);           //and write it to the global variable
         break;
      }
   }   
}

void UnregisterBuffer(double &Buffer[]) export
{
   int ptr=GetPtr(Buffer);                      //we will register by the real address of the buffer
   if(ptr==0) return;
   
   int gt=GlobalVariablesTotal();               
   int i;
   for(i=gt-1;i>=0;i--)                         //just look through all global variables
   {                                            //and delete our buffer from all places
      string name=GlobalVariableName(i);        
      if(GlobalVariableGet(name)==ptr)
         GlobalVariableDel(name);
   }      
}

Detailed comments are superfluous here, we just outline the fact that the first function creates a new descriptor of the above format in the global variable list; the second one searches through all global variables and deletes the variables and delete descriptors with the value equal to the pointer.

Now letэs consider the second task - obtaining data from the indicator. Before we implement the direct access to the data, first we need to find a corresponding descriptor. It can be solved by using the following function. Its algorithm will be the following: we go through all global variables and check them for the presence of field values specified in the descriptor.

If we have found it, we add its name to the array, passed as the last parameter. So, as result the function returns all the memory addresses that match search conditions. The returned value is the number of descriptors found.

int FindBuffers(string name, int mode, string symbol, int period, string &buffers[]) export
{
   int count=0;
   int i;
   bool found;
   string name_i;
   string descriptor[];
   int gt=GlobalVariablesTotal();

   StringTrimLeft(name);                                    //trim string from unnecessary spaces
   StringTrimRight(name);
   
   ArrayResize(buffers,count);                              //reset size to 0

   for(i=gt-1;i>=0;i--)
   {
      found=true;
      name_i=GlobalVariableName(i);
      
      StringExplode(name_i,"#",descriptor);                 //split string to fields
      
      if(StringFind(descriptor[0],name)<0&&name!=NULL) found=false; //check each field for the match
                                                                    //condition
      if(descriptor[1]!=mode&&mode>=0) found=false;
      if(descriptor[2]!=symbol&&symbol!=NULL) found=false;
      if(descriptor[3]!=period&&period>0) found=false;
      
      if(found)
      {
         count++;                                           //conditions met, add it to the list
         ArrayResize(buffers,count);
         buffers[count-1]=name_i;
      }
   }
   
   return(count);
}

As we see from the function code, some of the search conditions can be omitted. For example, if we pass the NULL as name, the checking of the string_identifier will be omitted. The same is for other fields:  mode<0, symbol:=NULL or period<=0. It allows to extend the search options in the table of descriptors.

For example, you can find all the Moving Average indicators in all chart windows, or only in EURUSD charts with period M15 etc. Additional remark: checking of the string_identifier is performed by function StringFind() instead of the strict equality checking. It is done to provide the possibility of searching by a part of the descriptor (say, when several indicators set a string of “MA(xxx)” type, search can be performed by substring “MA” - as result we will find all the Moving Averages registered).

Also we have used the function StringExplode(string s, string separator, string &result[]). It splits the specified string s to substrings, using the separator, and writes results to the result array.

void StringExplode(string s, string separator, string &result[])
{
   int i,pos;
   ArrayResize(result,1);
   
   pos=StringFind(s,separator); 
   if(pos<0) {result[0]=s;return;}
   
   for(i=0;;i++)
   {
      pos=StringFind(s,separator); 
      if(pos>=0)
      {
         result[i]=StringSubstr(s,0,pos);
         s=StringSubstr(s,pos+StringLen(separator));
      }
      else break;
      ArrayResize(result,ArraySize(result)+1);
   }
}

Now, when we have got the list of all the descriptors necessary, we can obtain data from the indicator:

double GetIndicatorValue(string descriptor, int shift) export
{
   int ptr;
   string fields[];
   int size,direction;
   if(GlobalVariableCheck(descriptor)>0)               //check that the descriptor is valid
   {                
      ptr = GlobalVariableGet(descriptor);             //get the pointer value
      if(ptr!=0)
      {
         StringExplode(descriptor,"#",fields);         //split its name to fields
         size = fields[4];                             //we need the current array size
         direction=fields[5];                                 
         if(direction==1) shift=size-1-shift;          //if the ordering_direction is reverse
         if(shift>=0&&shift<size)                      //check for its validity - to prevent crashes
            return(GetValue(MathAbs(ptr),shift));      //ok, return the value
      }   
   } 
   return(EMPTY_VALUE);                                //overwise return empty value 
}

As you see, it's a wrapping of the GetValue() function from the DLL. It is necessary to check for the descriptor validity, for the array boundaries, and to take into account the indicator buffer ordering. If unsuccessful, the function returns the EMPTY_VALUE.

The modification of the indicator buffer is similar:

bool SetIndicatorValue(string descriptor, int shift, double value) export
{
   int ptr;
   string fields[];
   int size,direction;
   if(GlobalVariableCheck(descriptor)>0)               //check for its validity
   {                
      ptr = GlobalVariableGet(descriptor);             //get descriptor value
      if(ptr!=0)
      {
         StringExplode(descriptor,"#",fields);         //split it to fields
         size = fields[4];                             //we need its size
         direction=fields[5];                                 
         if(direction==1) shift=size-1-shift;          //the case of the inverse ordering
         if(shift>=0&&shift<size)                      //check index to prevent the crash of the client terminal
         {
            SetValue(MathAbs(ptr),shift,value);
            return(true);
         }   
      }   
   }
   return(false);
}

If all the values are correct, it calls the SetValue() function from DLL. The returned value corresponds to the result of modification: true if successful and false in case of an error.

5. Checking How It Works

Now let's try to check it. As a target, we will use the Average True Range (ATR) from the standard package and show, what we have to modify in its code to allow to copy its values to another indicator window. Also we will test the data modification of its buffer data.

The first that we have to do is to add some code lines to the OnCalculate() function:

//+------------------------------------------------------------------+

//| Average True Range                                               |
//+------------------------------------------------------------------+
int OnCalculate(const int rates_total,
                const int prev_calculated,
                const datetime &Time[],
                const double &Open[],
                const double &High[],
                const double &Low[],
                const double &Close[],
                const long &TickVolume[],
                const long &Volume[],
                const int &Spread[])
  {
   if(prev_calculated!=rates_total)
      RegisterBuffer(ExtATRBuffer,"ATR",0);
…

As we see, the new buffer registration should be performed every time, when we have new bars - this case occurs as time passes (normally) or in case of the additional history data downloading.

Besides that, we need to delete descriptors from the table after operation of the indicator is over. That's why it is necessary to add some code to the deinit event handler (but remember that it should return void and have one const int reason input parameter):

void OnDeinit(const int reason)
{
   UnregisterBuffer(ExtATRBuffer);
}

Here is our modified indicator in the client terminal chart window (Fig.1) and its descriptor in the list of global variables (Fig. 2):

 

Fig. 1. Average True Range

Fig. 2. Descriptor created in the list of global variables of the client terminal

The next step is to get access to ATR data. Let's create a new indicator (let's name it test) with the following code:

//+------------------------------------------------------------------+
//|                                                         test.mq5 |
//|                                             Copyright 2009, alsu |
//|                                                 [e-mail] |
//+------------------------------------------------------------------+
#property copyright "2009, alsu"
#property link      "[e-mail]"
#property version   "1.00"
#property indicator_separate_window
#property indicator_buffers 1
#property indicator_plots   1

#include <exchng.mqh>

//---- plot ATRCopy
#property indicator_label1  "ATRCopy"
#property indicator_type1   DRAW_LINE
#property indicator_color1  Red
#property indicator_style1  STYLE_SOLID
#property indicator_width1  1
//--- indicator buffers
double ATRCopyBuffer[];

string atr_buffer;
string buffers[];

//+------------------------------------------------------------------+
//| Custom indicator initialization function                         |
//+------------------------------------------------------------------+

int OnInit()
  {
//--- indicator buffers mapping
   SetIndexBuffer(0,ATRCopyBuffer,INDICATOR_DATA);
//---
   return(0);
  }

//+------------------------------------------------------------------+
//| Custom indicator iteration function                              |
//+------------------------------------------------------------------+
int OnCalculate(const int rates_total,
                const int prev_calculated,
                const datetime& time[],
                const double& open[],
                const double& high[],
                const double& low[],
                const double& close[],
                const long& tick_volume[],
                const long& volume[],
                const int& spread[])
  {
//---
   int found=FindBuffers("ATR",0,NULL,0,buffers);   //search for descriptors in global variables
   if(found>0) atr_buffer=buffers[0];               //if we found it, save it to the atr_buffer
   else atr_buffer=NULL;
   int i;
   for(i=prev_calculated;i<rates_total;i++)
   {
      if(atr_buffer==NULL) break;
      ATRCopyBuffer[i]=GetIndicatorValue(atr_buffer,i);  //now it easy to get data
      SetIndicatorValue(atr_buffer,i,i);                 //and easy to record them
   }
//--- return value of prev_calculated for next call
   return(rates_total);
  }
//+------------------------------------------------------------------+

As we see, it's not difficult. Get the descriptors list by a substring and read the values of the indicator buffer using our functions. And finally, lets write some rubbish there (in our example, we write the values that correspond to the array element index).

Now running the test indicator (our target ATR still should be attached to the chart). As we see in Fig.3, the values of ATR are in the lower subwindow (in our test indicator), we see the straight line instead of it - in fact, it's filled with the values corresponding to the array indexes.

 

Fig. 3. Result of Test.mq5 operation

Twist of the wrist - that's all:)

6. Backward Compatibility

The author tried to create a library, written in MQL5, which is maximally close to the MQ4 standards. So, it needs just a few modifications in order to be used in MQL4.

In particular, you should remove the export keyword, appearing only in MQL5, and modify the code where we have used the indirect typecasting (MQL4 isn't so flexible). Actually, the use of functions in indicators is completely identical, the differences are only in method of the new bar and history adding control.

Conclusion

A few words about the advantages.

It seems that the use of the method described in this article may not be limited to its original purpose only. In addition to the construction of high-speed cascade indicators, the library can be successfully applied to Expert Advisors, including its history testing cases. Other applications are difficult to imagine for the author, but, I think, dear users will be happy to work in this direction.

I believe the following technical moments might be necessary:

• improvement of the table of descriptors (in particular, indication of the windows);
• development of the additional table for the descriptors creation ordering - it may be important for the stability of trading systems.
  

Also I will be glad to receive any other suggestions to improve the library.

All the necessary files are attached to the article. The codes for MQL5, MQL4 and the source code of the exchng.dll library are presented. The files have the same locations as it should be in the client terminal folders.

Disclaimer

The materials presented in this article describe programming techniques, which by careless use may cause damage of the software running on your computer. The use of the attached files should be done only at your own risk and can lead to possible side effects, though not revealed till now.

Acknowledgments

The author has used the problems proposed on the forum of MQL4.Community https://forum.mql4.com and ideas of its users: igor.senych, satop, bank, StSpirit, TheXpert, jartmailru, ForexTools, marketeer, IlyaA – sorry if the list is incomplete.