Getting Rid of Self-Made DLLs

--- | 25 April, 2012

Do you still write your DLLs?
Then we go to you!

Introduction

There always comes a moment when MQL5 language functional is not enough for fulfilling tasks. In that case an MQL5 programmer has to use additional tools. For example, it is possible to work with a database, use communication sockets or utilize operation system functions. An MQL5 programmer also has to deal with various APIs to expand the possibilities of the MQL5 program he/she uses. But for several reasons, the programmer cannot access the required functions directly from MQL5, as he/she does not know the following things:

• How to transfer a complex data type (for example, structure) to API function;
• How to work with the pointer that is returned by the API function.

Therefore, the programmer is forced to use a different programming language and to create an intermediate DLL to work with the required functionality. Although MQL5 has the possibility to present various data types and transfer them to API, unfortunately, MQL5 cannot solve the issue concerning data extraction from the accepted pointer.

In this article we will dot all the "i"s and show simple mechanisms of transferring and receiving complex data types and working with return indices.

Contents

1. Memory is Everything

• Getting the Indices
• Copying memory areas

2. Transferring the Structures to API Functions

• Transforming the structures using MQL5
• Example of transferring the structure for sockets

3. Working with API Functions Pointers

• Examples for Memory Mapping File,
• Example for MySQL

4. Reading NULL-Terminated Strings from API Functions

1. Memory Is Everything

As you may know, any variable (including complex data types variables) has its specific address, from which that variable is stored in memory. This address is a four-byte integer value (of int type) equal to the address of the first byte of this variable.

And if all is well defined, it is possible to work with this memory area. C language library (msvcrt.dll) contains memcpy function. Its purpose is the missing element, which binds MQL5 and various API libraries and provides great possibilities for a programmer.

Let's Turn to the Knowledge of Our Ancestors

Memcpy function copies the specified number of bytes from one buffer to another and returns the pointer to a receiver buffer.

void *memcpy(void *dst, const void *src, int cnt);
dst - pointer to the receiver buffer
src - pointer to the source buffer
cnt - number of bytes for copying

In other words, a memory area with a size of cnt bytes beginning from src address is copied to the memory area beginning from dst address.

The data located at src address can be of various types. This may be char one byte variable, double eight byte number, array, any structure and any memory volume. It means that you can freely transmit data from one area to another, if you know addresses and a size.

How Does It Work

Diagram 1 shows the comparative sizes of some data types.

Memcpy function is needed to copy the data from one memory area to another.
Figure 2 shows the copying of four bytes.

In MQL5 that will look as follows.

Example 1. Using memcpy
#import "msvcrt.dll"
  int memcpy(int &dst, int &src, int cnt);
#import
//+------------------------------------------------------------------+
//| Script program start function                                    |
//+------------------------------------------------------------------+
void OnStart()
{
  int dst, src=4, cnt=sizeof(int);
  int adr=memcpy(dst, src, cnt);
  Print("dst value="+string(dst)+"   Address dst="+string(adr));
}

It should be noted that various data types (of the same cnt size) can be used as memory areas dst and src point at. For example, src pointer can refer to double variable (cnt=8 bytes) and dst can refer to the array having the equivalent size char[8] or int[2].

Memory data can be considered as data of various types. For example, it is possible to transfer five byte array to {int i; char c;} structure or vice versa. This relationship provides an opportunity to work directly with API functions.

Let's examine memcpy application versions in the definite order.

Getting the Indices

In example 1 we showed that memcpy function returns dst variable address.

This property can be used to get an address of any variable (including the arrays of other complex types). To do this, we need only to specify the same variable as a source and receiver parameters. In cnt it is possible to transfer 0, as the actual copying is not necessary.

For example, we may get the address of double variable and short array:

Example 2. Getting pointers to the variable
#import "msvcrt.dll"
  int memcpy(short &dst[], short &src[], int cnt);
  int memcpy(double &dst,  double &src, int cnt);
#import

//+------------------------------------------------------------------+
//| Script program start function                                    |
//+------------------------------------------------------------------+
void OnStart()
{
  short src[5];
  //--- getting src array address (i.е., the address of the first element)
  int adr=memcpy(src, src, 0);
  double var;
  //--- getting var variable address
  adr=memcpy(var, var, 0); 
}

Received address then can be transferred to the required API function or as a structure parameter and also as a parameter of the same memcpy function.

Copying the Arrays

As you know, an array is some dedicated memory chunk. The size of dedicated memory depends on the elements type and their amount. For example, if the short array elements type and the number of the elements is 10, such an array occupies 20 bytes in memory (as short size is 2 bytes).

But these 20 bytes are also shown as arrays consisting of 20 char or 5 int. In any case, they occupy the same 20 bytes in memory.

To copy the arrays, it is necessary to do the following things:

• Allocate the required amount of the elements (not less than resulting cnt bytes) for dst memory;
• Specify the number of bytes in cnt that must be copied from src.

Example 3. Copying the arrays
#import "msvcrt.dll"
  int memcpy(double &dst[],  double &src[], int cnt);
#import

//+------------------------------------------------------------------+
//| Script program start function                                    |
//+------------------------------------------------------------------+
void OnStart()
{
  double src[5];
  //--- calculating the number of bytes!!!
  int cnt=sizeof(double)*ArraySize(src);
  double dst[]; 
  ArrayResize(dst, 5);
  //--- the array has been copied from src to dst
   memcpy(dst, src, cnt); 
}

2. Transferring the Structures to API Functions

Suppose, you need to transfer the filled structure pointer to API. MQL5 language sets limitations for transmitting the structures. At the beginning of the article I declared that the memory can be presented differently. That means that the required structure can be copied to the data type supported by MQL5. In general, an array is a type that is suitable for the structures. Therefore, we will have to get an array from a structure and then transfer an array to the API function.

The option of copying the memory using the structures is described in the documentation section. We cannot use memcpy function, as it is impossible to transfer the structures as parameters and copying the structures is the only way here.

Figure 3 shows the representation of the structure consisting of 5 variables of different types and its equivalent presented as char array.

Example 4. Copying the structures by means of MQL5
struct str1
{
  double d; // 8 bytes
  long l;   // 8 bytes
  int i[3]; // 3*4=12 bytes
};
struct str2
{
  uchar c[8+8+12]; // str1 structure size
};
//+------------------------------------------------------------------+
//| Script program start function                                    |
//+------------------------------------------------------------------+
void OnStart()
{
  str1 src; 
  src.d=-1;
  src.l=20;
  //--- filling the structure parameters
  ArrayInitialize(src.i, 0); 
  str2 dst;
  //--- turning the structure into the byte array
  dst=src; 
}

In such simple way we have copied the structure into the byte array.

Let's consider the socket creation function to make this example more practical.

int connect(SOCKET s, const struct sockaddr *name, int namelen);

In this function the second parameter is problematic, as it accepts the pointer for the structure. But we already know what to do with that. So, let's begin.

1. Let's write connect function for import by the method permissible in MQL5:

int connect(int s, uchar &name[], int namelen);

2. Let's observe the required structure in the documentation:

struct sockaddr_in
{
  short   sin_family;
  u_short sin_port;
  in_addr sin_addr; // additional 8 byte structure
  char sin_zero[8];
};

3. Creating a structure with an array of similar size:

struct ref_sockaddr_in
{
  uchar c[2+2+8+8];
};

4. After filling out the required sockaddr_in structure, we transfer it to the byte array and submit as connect parameter.

Below is the code section made according to these steps.

Example 5. Referring of the client socket to the server
#import "Ws2_32.dll"
  ushort htons(ushort hostshort);
  ulong inet_addr(char &cp[]);
  int connect(int s, char &name[], int namelen);
#import
//+------------------------------------------------------------------+
//| Script program start function                                    |
//+------------------------------------------------------------------+
void OnStart()
{
  //--- connecting the host after the socket initialization

  char ch[];
  StringToCharArray("127.0.0.1", ch);
  //--- preparing the structure
  sockaddr_in addrin;
  addrin.sin_family=AF_INET;
  addrin.sin_addr=inet_addr(ch);
  addrin.sin_port=htons(1000);
  //--- copying the structure to the array
  ref_sockaddr_in ref=addrin; 
  //--- connecting the host
  res=connect(asock, ref.c, sizeof(addrin)); 

  //--- further work with the socket
}

As you can see, you do not need to make your DLL at all. The structures are transferred to API directly.

3. Working with API Functions Pointers

In most cases API functions return a pointer to the data: structures and arrays. MQL5 is not suitable for extracting the data, memcpy function can be used here.

Example of working with memory arrays from Memory Mapping File (MMF)

When working with MMF the function is used, which returns a pointer to a dedicated memory array.

int MapViewOfFile(int hFile, int DesiredAccess, int OffsetHigh, int OffsetLow, int NumOfBytesToMap);

Data reading from this array is executed by simple copying of the required amount of bytes by memcpy function.
Writing the data into the array is performed by the same use of memcpy.

Example 6. Recording and reading data from MMF memory
#import "kernel32.dll"
  int OpenFileMappingW(int dwDesiredAccess, int bInheritHandle,  string lpName);
  int MapViewOfFile(int hFileMappingObject, int dwDesiredAccess, 
                      int dwFileOffsetHigh, int dwFileOffsetLow, int dwNumberOfBytesToMap);
  int UnmapViewOfFile(int lpBaseAddress);
  int CloseHandle(int hObject);
#import "msvcrt.dll"
  int memcpy(uchar &Destination[], int Source, int Length);
  int memcpy(int Destination, int &Source, int Length);
  int memcpy(int Destination, uchar &Source[], int Length);
#import

#define FILE_MAP_ALL_ACCESS   0x000F001F

//+------------------------------------------------------------------+
//| Script program start function                                    |
//+------------------------------------------------------------------+
void OnStart()
{
  //--- opening the memory object
  int hmem=OpenFileMappingW(FILE_MAP_ALL_ACCESS, 0, "Local\\file");
  //--- getting pointer to the memory
  int view=MapViewOfFile(hmem, FILE_MAP_ALL_ACCESS, 0, 0, 0); 
  //--- reading the first 10 bytes from the memory
  uchar src[10];
  memcpy(src, view, 10);
  int num=10;
  //--- recording the 4 byte int number to the memory beginning
  memcpy(view, num, 4);
  //--- unmapping the view
  UnmapViewOfFile(view); 
  //--- closing the object
  CloseHandle(hmem); 
}

As you can see, it is not so difficult to work with pointers for the memory array. And most importantly, you do not need to create your additional DLL for that.

Example of working with returned structures for MySQL

One of the urgent problems when working with MySQL has been getting data from it. mysql_fetch_row function returns the strings array. Each string is a fields array. So, this function returns the pointer to the pointer. Our task is to extract all these data from the returned pointer.

The task is a bit complicated by the fact that the fields are various data types including binary ones. It means that it will be impossible to present them as string array. The functions mysql_num_rows, mysql_num_fields, mysql_fetch_lengths are used for getting the information about the strings and field sizes.

Figure 4 shows the structure of presenting the result in memory.
The addresses of the beginning of three strings are gathered into the array. And the address of the array beginning (in the example = 94) is what mysql_fetch_row function will return.

Below is the example of the code for getting data from a database request.

Example 7. Getting data from MySQL
#import "libmysql.dll"
  int mysql_real_query(int mysql, uchar &query[], int length);
  int mysql_store_result(int mysql);
  int mysql_field_count(int mysql);
  uint mysql_num_rows(int result);
  int mysql_num_fields(int result);
  int mysql_fetch_lengths(int result);
  int mysql_fetch_row(int result);
#import 
//+------------------------------------------------------------------+
//| Script program start function                                    |
//+------------------------------------------------------------------+
void OnStart()
{
  //--- ... preliminarily initialized mysql data base
  //--- request for getting all the strings from the table
  string query="SELECT * FROM table"; 
  uchar aquery[]; 
  StringToCharArray(query, aquery);

  //--- sending the request
  err=mysql_real_query(mysql, aquery, StringLen(query)); 
  int result=mysql_store_result(mysql);

  //--- in case it contains the strings
  if (result>0) 
  {
    ulong num_rows=mysql_num_rows(result);
    int num_fields=mysql_num_fields(result);    

    //--- getting the first string pointer
    int r=0, row_ptr=mysql_fetch_row(result);
    while(row_ptr>0)
    {

       //--- getting the pointer to the current string columns lengths
      int len_ptr=mysql_fetch_lengths(result); 
      int lens[]; 
       ArrayResize(lens, num_fields);
      //--- getting the sizes of the string fields
      memcpy(lens, len_ptr, num_fields*sizeof(int));
      //--- getting the data fields   
      int field_ptr[];
      ArrayResize(field_ptr, num_fields);
      ArrayInitialize(field_ptr, 0);

      //--- getting the pointers to the fields
      memcpy(field_ptr, row_ptr, num_fields*sizeof(int)); 
      for (int f=0; f<num_fields; f++)
      {
        ArrayResize(byte, lens[f]);
        ArrayInitialize(byte, 0);
         //--- copy the field to the byte array
        if (field_ptr[f]>0 && lens[f]>0) memcpy(byte, field_ptr[f], lens[f]);
      }
      r++;
      //--- getting the pointer to the pointer to the next string
      row_ptr=mysql_fetch_row(result); 
    }
  }
}

4. Reading NULL-Terminated Strings from API Functions

Some API functions return the pointer to the string but do not show us the length of this string. In this situation we deal with strings that end in zero. This zero helps to determine the end of the string. This means that its size can be specified.

C (msvcrt.dll) library already has the function that copies the contents of the NULL-terminated string from the appropriate pointer to another string. The size of the string is defined by the function. It is better to use a byte array as a receiver, as APIs often return multibyte strings instead of Unicode.

strcpy - copies NULL-terminated strings

char *strcpy(char *dst, const char *src);
dst - the pointer to the destination string
src - the pointer to the Null-terminated source string

In fact, it is a special case of memcpy function. The system stops the copying on the found zero in a string. This function will always be used when working with such pointers.

For example, there are several functions in API from MySQL that return the pointers to strings. And getting data from them using strcpy is a trivial task.

Example 8. Getting the strings from the pointers
#import "libmysql.dll"
  int mysql_init(int mysql);
  int mysql_real_connect(int mysql, uchar &host[], uchar &user[], uchar &password[], 
                            uchar &DB[], uint port, uchar &socket[], int clientflag);
  int mysql_get_client_info();
  int mysql_get_host_info(int mysql);
  int mysql_get_server_info(int mysql);
  int mysql_character_set_name(int mysql);
  int mysql_stat(int mysql);
#import "msvcrt.dll"
  int strcpy(uchar &dst[], int src);
#import
//+------------------------------------------------------------------+
//| Script program start function                                    |
//+------------------------------------------------------------------+
void OnStart()
{
  uchar byte[];
  ArrayResize(byte, 300);

  int ptr;
  string st;
  //--- pointer to the string
  ptr=mysql_get_client_info();

  if (ptr>0) strcpy(byte, ptr);
  Print("client_info="+CharArrayToString(byte));
  //--- initializing the base
  int mysql=mysql_init(mysql);

  //--- transferring the strings to the byte arrays
  uchar ahost[]; 
  StringToCharArray("localhost", ahost);
  uchar auser[];
  StringToCharArray("root", auser);
  uchar apwd[];
  StringToCharArray("", apwd);
  uchar adb[];
  StringToCharArray("some_db", adb);
  uchar asocket[];
  StringToCharArray("", asocket);
  //--- connecting the base
  int rez=mysql_real_connect(mysql, ahost, auser, apwd, adb, port, asocket, 0);
  //--- determining the connection and the base status
  ptr=mysql_get_host_info(mysql);
  if (ptr>0) strcpy(byte, ptr);
  Print("mysql_host_info="+CharArrayToString(byte));
  ptr=mysql_get_server_info(mysql);
  if (ptr>0) strcpy(byte, ptr);
  Print("mysql_server_info="+CharArrayToString(byte));
  ptr=mysql_character_set_name(mysql);
  if (ptr>0) strcpy(byte, ptr);
  Print("mysql_character_set_name="+CharArrayToString(byte));
  ptr=mysql_stat(mysql);
  if (ptr>0) strcpy(byte, ptr);
  Print("mysql_stat="+CharArrayToString(byte));
}

Conclusion

Thus, the use of three basic mechanisms of working with memory (copying the structures, getting pointers and their data on memcpy and getting strcpy strings) covers virtually all tasks when working with various API functions.