Getting Rid of Self-Made DLLs

25 April 2012, 14:34
9 541

Do you still write your DLLs?
Then we go to you!


There always comes a moment when MQL5 language functional is not enough for fulfilling tasks. In that case an MQL5 programmer has to use additional tools. For example, it is possible to work with a database, use communication sockets or utilize operation system functions. An MQL5 programmer also has to deal with various APIs to expand the possibilities of the MQL5 program he/she uses. But for several reasons, the programmer cannot access the required functions directly from MQL5, as he/she does not know the following things:

  • How to transfer a complex data type (for example, structure) to API function;
  • How to work with the pointer that is returned by the API function.

Therefore, the programmer is forced to use a different programming language and to create an intermediate DLL to work with the required functionality. Although MQL5 has the possibility to present various data types and transfer them to API, unfortunately, MQL5 cannot solve the issue concerning data extraction from the accepted pointer.

In this article we will dot all the "i"s and show simple mechanisms of transferring and receiving complex data types and working with return indices.


1. Memory is Everything

  • Getting the Indices
  • Copying memory areas

2. Transferring the Structures to API Functions

  • Transforming the structures using MQL5
  • Example of transferring the structure for sockets

3. Working with API Functions Pointers

  • Examples for Memory Mapping File,
  • Example for MySQL

4. Reading NULL-Terminated Strings from API Functions

1. Memory Is Everything

As you may know, any variable (including complex data types variables) has its specific address, from which that variable is stored in memory. This address is a four-byte integer value (of int type) equal to the address of the first byte of this variable.

And if all is well defined, it is possible to work with this memory area. C language library (msvcrt.dll) contains memcpy function. Its purpose is the missing element, which binds MQL5 and various API libraries and provides great possibilities for a programmer.

Let's Turn to the Knowledge of Our Ancestors

Memcpy function copies the specified number of bytes from one buffer to another and returns the pointer to a receiver buffer.

void *memcpy(void *dst, const void *src, int cnt);
dst - pointer to the receiver buffer
src - pointer to the source buffer
cnt - number of bytes for copying

In other words, a memory area with a size of cnt bytes beginning from src address is copied to the memory area beginning from dst address.

The data located at src address can be of various types. This may be char one byte variable, double eight byte number, array, any structure and any memory volume. It means that you can freely transmit data from one area to another, if you know addresses and a size.

How Does It Work

Diagram 1 shows the comparative sizes of some data types.

Sizes of various data types in MQL5

Memcpy function is needed to copy the data from one memory area to another.
Figure 2 shows the copying of four bytes.

Example of copying 4 bytes with the use of memcpy

In MQL5 that will look as follows.

Example 1. Using memcpy
#import "msvcrt.dll"
  int memcpy(int &dst, int &src, int cnt);
//| Script program start function                                    |
void OnStart()
  int dst, src=4, cnt=sizeof(int);
  int adr=memcpy(dst, src, cnt);
  Print("dst value="+string(dst)+"   Address dst="+string(adr));

It should be noted that various data types (of the same cnt size) can be used as memory areas dst and src point at. For example, src pointer can refer to double variable (cnt=8 bytes) and dst can refer to the array having the equivalent size char[8] or int[2].

It does not matter for the memory, what idea a programmer has about it at the moment. It does not matter, whether it is an array char[8] or just one long variable or structure { int a1; int a2; }.

Memory data can be considered as data of various types. For example, it is possible to transfer five byte array to {int i; char c;} structure or vice versa. This relationship provides an opportunity to work directly with API functions.

Let's examine memcpy application versions in the definite order.

Getting the Indices

In example 1 we showed that memcpy function returns dst variable address.

This property can be used to get an address of any variable (including the arrays of other complex types). To do this, we need only to specify the same variable as a source and receiver parameters. In cnt it is possible to transfer 0, as the actual copying is not necessary.

For example, we may get the address of double variable and short array:

Example 2. Getting pointers to the variable
#import "msvcrt.dll"
  int memcpy(short &dst[], short &src[], int cnt);
  int memcpy(double &dst,  double &src, int cnt);

//| Script program start function                                    |
void OnStart()
  short src[5];
  //--- getting src array address (i.е., the address of the first element)
  int adr=memcpy(src, src, 0);
  double var;
  //--- getting var variable address
  adr=memcpy(var, var, 0); 

Received address then can be transferred to the required API function or as a structure parameter and also as a parameter of the same memcpy function.

Copying the Arrays

As you know, an array is some dedicated memory chunk. The size of dedicated memory depends on the elements type and their amount. For example, if the short array elements type and the number of the elements is 10, such an array occupies 20 bytes in memory (as short size is 2 bytes).

But these 20 bytes are also shown as arrays consisting of 20 char or 5 int. In any case, they occupy the same 20 bytes in memory.

To copy the arrays, it is necessary to do the following things:

  • Allocate the required amount of the elements (not less than resulting cnt bytes) for dst memory;
  • Specify the number of bytes in cnt that must be copied from src.
Example 3. Copying the arrays
#import "msvcrt.dll"
  int memcpy(double &dst[],  double &src[], int cnt);

//| Script program start function                                    |
void OnStart()
  double src[5];
  //--- calculating the number of bytes!!!
  int cnt=sizeof(double)*ArraySize(src);
  double dst[]; 
  ArrayResize(dst, 5);
  //--- the array has been copied from src to dst
   memcpy(dst, src, cnt); 

2. Transferring the Structures to API Functions

Suppose, you need to transfer the filled structure pointer to API. MQL5 language sets limitations for transmitting the structures. At the beginning of the article I declared that the memory can be presented differently. That means that the required structure can be copied to the data type supported by MQL5. In general, an array is a type that is suitable for the structures. Therefore, we will have to get an array from a structure and then transfer an array to the API function.

The option of copying the memory using the structures is described in the documentation section. We cannot use memcpy function, as it is impossible to transfer the structures as parameters and copying the structures is the only way here.

Figure 3 shows the representation of the structure consisting of 5 variables of different types and its equivalent presented as char array.

Presenting the structure consisting of 5 variables of different types and its equivalent presented as char array

Example 4. Copying the structures by means of MQL5
struct str1
  double d; // 8 bytes
  long l;   // 8 bytes
  int i[3]; // 3*4=12 bytes
struct str2
  uchar c[8+8+12]; // str1 structure size
//| Script program start function                                    |
void OnStart()
  str1 src; 
  //--- filling the structure parameters
  ArrayInitialize(src.i, 0); 
  str2 dst;
  //--- turning the structure into the byte array

In such simple way we have copied the structure into the byte array.

Let's consider the socket creation function to make this example more practical.

int connect(SOCKET s, const struct sockaddr *name, int namelen);

In this function the second parameter is problematic, as it accepts the pointer for the structure. But we already know what to do with that. So, let's begin.

1. Let's write connect function for import by the method permissible in MQL5:

int connect(int s, uchar &name[], int namelen);

2. Let's observe the required structure in the documentation:

struct sockaddr_in
  short   sin_family;
  u_short sin_port;
  in_addr sin_addr; // additional 8 byte structure
  char sin_zero[8];

3. Creating a structure with an array of similar size:

struct ref_sockaddr_in
  uchar c[2+2+8+8];

4. After filling out the required sockaddr_in structure, we transfer it to the byte array and submit as connect parameter.

Below is the code section made according to these steps.

Example 5. Referring of the client socket to the server
#import "Ws2_32.dll"
  ushort htons(ushort hostshort);
  ulong inet_addr(char &cp[]);
  int connect(int s, char &name[], int namelen);
//| Script program start function                                    |
void OnStart()
  //--- connecting the host after the socket initialization

  char ch[];
  StringToCharArray("", ch);
  //--- preparing the structure
  sockaddr_in addrin;
  //--- copying the structure to the array
  ref_sockaddr_in ref=addrin; 
  //--- connecting the host
  res=connect(asock, ref.c, sizeof(addrin)); 

  //--- further work with the socket

As you can see, you do not need to make your DLL at all. The structures are transferred to API directly.

3. Working with API Functions Pointers

In most cases API functions return a pointer to the data: structures and arrays. MQL5 is not suitable for extracting the data, memcpy function can be used here.

Example of working with memory arrays from Memory Mapping File (MMF)

When working with MMF the function is used, which returns a pointer to a dedicated memory array.

int MapViewOfFile(int hFile, int DesiredAccess, int OffsetHigh, int OffsetLow, int NumOfBytesToMap);

Data reading from this array is executed by simple copying of the required amount of bytes by memcpy function.
Writing the data into the array is performed by the same use of memcpy.

Example 6. Recording and reading data from MMF memory
#import "kernel32.dll"
  int OpenFileMappingW(int dwDesiredAccess, int bInheritHandle,  string lpName);
  int MapViewOfFile(int hFileMappingObject, int dwDesiredAccess, 
                      int dwFileOffsetHigh, int dwFileOffsetLow, int dwNumberOfBytesToMap);
  int UnmapViewOfFile(int lpBaseAddress);
  int CloseHandle(int hObject);
#import "msvcrt.dll"
  int memcpy(uchar &Destination[], int Source, int Length);
  int memcpy(int Destination, int &Source, int Length);
  int memcpy(int Destination, uchar &Source[], int Length);

#define FILE_MAP_ALL_ACCESS   0x000F001F

//| Script program start function                                    |
void OnStart()
  //--- opening the memory object
  int hmem=OpenFileMappingW(FILE_MAP_ALL_ACCESS, 0, "Local\\file");
  //--- getting pointer to the memory
  int view=MapViewOfFile(hmem, FILE_MAP_ALL_ACCESS, 0, 0, 0); 
  //--- reading the first 10 bytes from the memory
  uchar src[10];
  memcpy(src, view, 10);
  int num=10;
  //--- recording the 4 byte int number to the memory beginning
  memcpy(view, num, 4);
  //--- unmapping the view
  //--- closing the object

As you can see, it is not so difficult to work with pointers for the memory array. And most importantly, you do not need to create your additional DLL for that.

Example of working with returned structures for MySQL

One of the urgent problems when working with MySQL has been getting data from it. mysql_fetch_row function returns the strings array. Each string is a fields array. So, this function returns the pointer to the pointer. Our task is to extract all these data from the returned pointer.

The task is a bit complicated by the fact that the fields are various data types including binary ones. It means that it will be impossible to present them as string array. The functions mysql_num_rows, mysql_num_fields, mysql_fetch_lengths are used for getting the information about the strings and field sizes.

Figure 4 shows the structure of presenting the result in memory.
The addresses of the beginning of three strings are gathered into the array. And the address of the array beginning (in the example = 94) is what mysql_fetch_row function will return.

The structure of presenting the request result in memory

Below is the example of the code for getting data from a database request.

Example 7. Getting data from MySQL
#import "libmysql.dll"
  int mysql_real_query(int mysql, uchar &query[], int length);
  int mysql_store_result(int mysql);
  int mysql_field_count(int mysql);
  uint mysql_num_rows(int result);
  int mysql_num_fields(int result);
  int mysql_fetch_lengths(int result);
  int mysql_fetch_row(int result);
//| Script program start function                                    |
void OnStart()
  //--- ... preliminarily initialized mysql data base
  //--- request for getting all the strings from the table
  string query="SELECT * FROM table"; 
  uchar aquery[]; 
  StringToCharArray(query, aquery);

  //--- sending the request
  err=mysql_real_query(mysql, aquery, StringLen(query)); 
  int result=mysql_store_result(mysql);

  //--- in case it contains the strings
  if (result>0) 
    ulong num_rows=mysql_num_rows(result);
    int num_fields=mysql_num_fields(result);    

    //--- getting the first string pointer
    int r=0, row_ptr=mysql_fetch_row(result);

       //--- getting the pointer to the current string columns lengths
      int len_ptr=mysql_fetch_lengths(result); 
      int lens[]; 
       ArrayResize(lens, num_fields);
      //--- getting the sizes of the string fields
      memcpy(lens, len_ptr, num_fields*sizeof(int));
      //--- getting the data fields   
      int field_ptr[];
      ArrayResize(field_ptr, num_fields);
      ArrayInitialize(field_ptr, 0);

      //--- getting the pointers to the fields
      memcpy(field_ptr, row_ptr, num_fields*sizeof(int)); 
      for (int f=0; f<num_fields; f++)
        ArrayResize(byte, lens[f]);
        ArrayInitialize(byte, 0);
         //--- copy the field to the byte array
        if (field_ptr[f]>0 && lens[f]>0) memcpy(byte, field_ptr[f], lens[f]);
      //--- getting the pointer to the pointer to the next string

4. Reading NULL-Terminated Strings from API Functions

Some API functions return the pointer to the string but do not show us the length of this string. In this situation we deal with strings that end in zero. This zero helps to determine the end of the string. This means that its size can be specified.

Presenting the NULL-terminated string in memory

C (msvcrt.dll) library already has the function that copies the contents of the NULL-terminated string from the appropriate pointer to another string. The size of the string is defined by the function. It is better to use a byte array as a receiver, as APIs often return multibyte strings instead of Unicode.

strcpy - copies NULL-terminated strings

char *strcpy(char *dst, const char *src);
dst - the pointer to the destination string
src - the pointer to the Null-terminated source string

In fact, it is a special case of memcpy function. The system stops the copying on the found zero in a string. This function will always be used when working with such pointers.

For example, there are several functions in API from MySQL that return the pointers to strings. And getting data from them using strcpy is a trivial task.

Example 8. Getting the strings from the pointers
#import "libmysql.dll"
  int mysql_init(int mysql);
  int mysql_real_connect(int mysql, uchar &host[], uchar &user[], uchar &password[], 
                            uchar &DB[], uint port, uchar &socket[], int clientflag);
  int mysql_get_client_info();
  int mysql_get_host_info(int mysql);
  int mysql_get_server_info(int mysql);
  int mysql_character_set_name(int mysql);
  int mysql_stat(int mysql);
#import "msvcrt.dll"
  int strcpy(uchar &dst[], int src);
//| Script program start function                                    |
void OnStart()
  uchar byte[];
  ArrayResize(byte, 300);

  int ptr;
  string st;
  //--- pointer to the string

  if (ptr>0) strcpy(byte, ptr);
  //--- initializing the base
  int mysql=mysql_init(mysql);

  //--- transferring the strings to the byte arrays
  uchar ahost[]; 
  StringToCharArray("localhost", ahost);
  uchar auser[];
  StringToCharArray("root", auser);
  uchar apwd[];
  StringToCharArray("", apwd);
  uchar adb[];
  StringToCharArray("some_db", adb);
  uchar asocket[];
  StringToCharArray("", asocket);
  //--- connecting the base
  int rez=mysql_real_connect(mysql, ahost, auser, apwd, adb, port, asocket, 0);
  //--- determining the connection and the base status
  if (ptr>0) strcpy(byte, ptr);
  if (ptr>0) strcpy(byte, ptr);
  if (ptr>0) strcpy(byte, ptr);
  if (ptr>0) strcpy(byte, ptr);


Thus, the use of three basic mechanisms of working with memory (copying the structures, getting pointers and their data on memcpy and getting strcpy strings) covers virtually all tasks when working with various API functions.

Warning. It may be unsafe to work with memcpy and strcpy, unless a sufficient amount of data has been allocated for the receiver buffer. Therefore, be careful about the size of the amounts allocated for receiving data.

Translated from Russian by MetaQuotes Software Corp.
Original article:

Last comments | Go to discussion (1)
krisn | 22 Nov 2014 at 05:23
Most useful and informative article regarding working with external DLL APIs. Thank you so much.
How to Post a Product in the Market How to Post a Product in the Market

Publish your interesting applications in the Market, and they will be immediately available to all traders who use MetaTrader 5 worldwide. The Market is a great opportunity to earn, with an immediate transfer to the account and convenient statistics to help you analyze purchases and downloads of demo versions of your products. All MQL5 programs in the Market are automatically encrypted for a specific buyer, allowing 5 or more activations, and do not require additional protection from your side.

How to Develop an Expert Advisor using UML Tools How to Develop an Expert Advisor using UML Tools

This article discusses creation of Expert Advisors using the UML graphical language, which is used for visual modeling of object-oriented software systems. The main advantage of this approach is the visualization of the modeling process. The article contains an example that shows modeling of the structure and properties of an Expert Advisor using the Software Ideas Modeler.

Who Is Who in Who Is Who in

The website remembers all of you quite well! How many of your threads are epic, how popular your articles are and how often your programs in the Code Base are downloaded – this is only a small part of what is remembered at Your achievements are available in your profile, but what about the overall picture? In this article we will show the general picture of all members achievements.

AutoElliottWaveMaker - MetaTrader 5 Tool for Semi-Automatic Analysis of Elliott Waves AutoElliottWaveMaker - MetaTrader 5 Tool for Semi-Automatic Analysis of Elliott Waves

The article provides a review of AutoElliottWaveMaker - the first development for Elliott Wave analysis in MetaTrader 5 that represents a combination of manual and automatic wave labeling. The wave analysis tool is written exclusively in MQL5 and does not include external dll libraries. This is another proof that sophisticated and interesting programs can (and should) be developed in MQL5.