Communicating With MetaTrader 5 Using Named Pipes Without Using DLLs

MetaQuotes | 15 October, 2012

Introduction

Many developers face the same problem - how to get to the trading terminal sandbox without using unsafe DLLs.

One of the easiest and safest method is to use standard Named Pipes that work as normal file operations. They allow you to organize interprocessor client-server communication between programs. Although there is an already published article A DLL-free solution to communicate between MetaTrader 5 terminals using Named Pipes on this topic that demonstrates enabling access to DLLs, we will use standard and safe features of client terminal.

You can find more information about named pipes in MSDN library, but we will get down to practical examples in C++ and MQL5. We will implement server, client, data exchange between them and then benchmark performance.

Server Implementation

Let's code a simple server in C++. A script from the terminal will connect to this server and will exchange data with it. The server core has the following set of WinAPI functions:

• CreateNamedPipe - creates a named pipe.
• ConnectNamedPipe - enables server to wait for client connections.
• WriteFile - writes data to pipe.
• ReadFile - reads data from pipe.
• FlushFileBuffers - flushes accumulated buffers.
• DisconnectNamedPipe - disconnects server.
• CloseHandle - closes handle.

Once a named pipe is opened it returns a file handle that can be used for regular read/write file operations. As a result you get a very simple mechanism that don't require any special knowledge in network operations.

Named pipes have one distinctive feature - they can be both local and network. That is, it's easy to implement a remote server that will accept network connections from client terminals.

Here is a simple example of creating a local server as full-duplex channel that works in the bytes exchange mode:

//--- open 
CPipeManager manager;

if(!manager.Create(L"\\\\.\\pipe\\MQL5.Pipe.Server"))
   return(-1);


//+------------------------------------------------------------------+
//| Create named pipe                                                |
//+------------------------------------------------------------------+
bool CPipeManager::Create(LPCWSTR pipename)
  {
//--- check parameters
   if(!pipename || *pipename==0) return(false);
//--- close old
   Close();
//--- create named pipe
   m_handle=CreateNamedPipe(pipename,PIPE_ACCESS_DUPLEX,
                            PIPE_TYPE_BYTE | PIPE_READMODE_BYTE | PIPE_WAIT,
                            PIPE_UNLIMITED_INSTANCES,256*1024,256*1024,1000,NULL);

   if(m_handle==INVALID_HANDLE_VALUE)
     {
      wprintf(L"Creating pipe '%s' failed\n",pipename);
      return(false);
     }
//--- ok
   wprintf(L"Pipe '%s' created\n",pipename);
   return(true);
  }

To get a client connection you have to use the ConnectNamedPipe function:

//+------------------------------------------------------------------+
//| Connect client                                                   |
//+------------------------------------------------------------------+
bool CPipeManager::ConnectClient(void)
  {
//--- pipe exists?
   if(m_handle==INVALID_HANDLE_VALUE) return(false);
//--- connected?
   if(!m_connected)
     {
      //--- connect
      if(ConnectNamedPipe(m_handle,NULL)==0)
        {
         //--- client already connected before ConnectNamedPipe?
         if(GetLastError()!=ERROR_PIPE_CONNECTED)
            return(false);
         //--- ok
        }
      m_connected=true;
     }
//---
   return(true);
  }

Data exchange is organized using 4 simple functions:

• CPipeManager::Send(void *data,size_t data_size)
• CPipeManager::Read(void *data,size_t data_size)
• CPipeManager::SendString(LPCSTR command)
• CPipeManager::ReadString(LPSTR answer,size_t answer_maxlen)

They allow you to send/receive data as binary data or ANSI text strings in MQL5 compatible mode. Moreover, since CFilePipe in MQL5 opens a file in ANSI mode by default, strings are automatically converted to Unicode on receipt and sending. If your MQL5 program opens a file in Unicode mode (FILE_UNICODE), then it can exchange Unicode strings (with BOM starting signature).

Client Implementation

We will write our client in MQL5. It will be able to perform regular file operations using the CFilePipe class from Standard Library. This class is almost identical to the CFileBin, but it contains an important verification of data availability in a virtual file before reading this data.

//+------------------------------------------------------------------+
//| Wait for incoming data                                           |
//+------------------------------------------------------------------+
bool CFilePipe::WaitForRead(const ulong size)
  {
//--- check handle and stop flag
   while(m_handle!=INVALID_HANDLE && !IsStopped())
     {
      //--- enough data?
      if(FileSize(m_handle)>=size)
         return(true);
      //--- wait a little
      Sleep(1);
     }
//--- failure
   return(false);
  }

//+------------------------------------------------------------------+
//| Read an array of variables of double type                        |
//+------------------------------------------------------------------+
uint CFilePipe::ReadDoubleArray(double &array[],const int start_item,const int items_count)
  {
//--- calculate size
   uint size=ArraySize(array);
   if(items_count!=WHOLE_ARRAY) size=items_count;
//--- check for data
   if(WaitForRead(size*sizeof(double)))
      return FileReadArray(m_handle,array,start_item,items_count);
//--- failure
   return(0);
  }

Named pipes have significant differences in implementation of their local and network modes. Without such a verification, network mode operations will always return a read error when sending large amounts of data (over 64K).

Let's connect to the server with two checks: either to remote computer named 'RemoteServerName' or to local machine.

void OnStart()
  {
//--- wait for pipe server
   while(!IsStopped())
     {
      if(ExtPipe.Open("\\\\RemoteServerName\\pipe\\MQL5.Pipe.Server",FILE_READ|FILE_WRITE|FILE_BIN)!=INVALID_HANDLE) break;
      if(ExtPipe.Open("\\\\.\\pipe\\MQL5.Pipe.Server",FILE_READ|FILE_WRITE|FILE_BIN)!=INVALID_HANDLE) break;
      Sleep(250);
     }
   Print("Client: pipe opened");

Data Exchange

After successful connection let's send a text string with identification info to the server. Unicode string will be automatically converted into ANSI, since the file is opened in ANSI mode.

//--- send welcome message
   if(!ExtPipe.WriteString(__FILE__+" on MQL5 build "+IntegerToString(__MQ5BUILD__)))
     {
      Print("Client: sending welcome message failed");
      return;
     }

In response, the server will send its string "Hello from pipe server" and the integer 1234567890. The client once again will send string "Test string" and the integer 1234567890.

//--- read data from server
   string        str;
   int           value=0;

   if(!ExtPipe.ReadString(str))
     {
      Print("Client: reading string failed");
      return;
     }
   Print("Server: ",str," received");

   if(!ExtPipe.ReadInteger(value))
     {
      Print("Client: reading integer failed");
      return;
     }
   Print("Server: ",value," received");
//--- send data to server
   if(!ExtPipe.WriteString("Test string"))
     {
      Print("Client: sending string failed");
      return;
     }

   if(!ExtPipe.WriteInteger(value))
     {
      Print("Client: sending integer failed");
      return;
     }

OK, we are finished with simple data exchange. Now it's time for performance benchmark.

Performance Benchmark

As a test, we will send 1 gigabyte of data as an array of the double type numbers in blocks of 8 megabytes from server to client, then check correctness of the blocks and measure the transfer rate.

Here is this code in C++ server:

//--- benchmark
   double  volume=0.0;
   double *buffer=new double[1024*1024];   // 8 Mb

   wprintf(L"Server: start benchmark\n");
   if(buffer)
     {
      //--- fill the buffer
      for(size_t j=0;j<1024*1024;j++)
         buffer[j]=j;
      //--- send 8 Mb * 128 = 1024 Mb to client
      DWORD   ticks=GetTickCount();

      for(size_t i=0;i<128;i++)
        {
         //--- setup guard signatures
         buffer[0]=i;
         buffer[1024*1024-1]=i+1024*1024-1;
         //--- 
         if(!manager.Send(buffer,sizeof(double)*1024*1024))
           {
            wprintf(L"Server: benchmark failed, %d\n",GetLastError());
            break;
           }
         volume+=sizeof(double)*1024*1024;
         wprintf(L".");
        }
      wprintf(L"\n");
      //--- read confirmation
      if(!manager.Read(&value,sizeof(value)) || value!=12345)
         wprintf(L"Server: benchmark confirmation failed\n");
      //--- show statistics
      ticks=GetTickCount()-ticks;
      if(ticks>0)
         wprintf(L"Server: %.0lf Mb sent at %.0lf Mb per second\n",volume/1024/1024,volume/1024/ticks);
      //---
      delete[] buffer;
     }

and in MQL5 client:

//--- benchmark
   double buffer[];
   double volume=0.0;

   if(ArrayResize(buffer,1024*1024,0)==1024*1024)
     {
      uint  ticks=GetTickCount();
      //--- read 8 Mb * 128 = 1024 Mb from server
      for(int i=0;i<128;i++)
        {
         uint items=ExtPipe.ReadDoubleArray(buffer);
         if(items!=1024*1024)
           {
            Print("Client: benchmark failed after ",volume/1024," Kb, ",items," items received");
            break;
           }
         //--- check the data
         if(buffer[0]!=i || buffer[1024*1024-1]!=i+1024*1024-1)
           {
            Print("Client: benchmark invalid content");
            break;
           }
         //---
         volume+=sizeof(double)*1024*1024;
        }
      //--- send confirmation
      value=12345;
      if(!ExtPipe.WriteInteger(value))
         Print("Client: benchmark confirmation failed ");
      //--- show statistics
      ticks=GetTickCount()-ticks;
      if(ticks>0)
         printf("Client: %.0lf Mb received at %.0lf Mb per second\n",volume/1024/1024,volume/1024/ticks);
      //---
      ArrayFree(buffer);
     }

Note, that the first and the last elements of transfered blocks are checked in order to make sure that there were no errors during the transfer. Also, when transfer is complete client sends a confirming signal to the server about successful data receipt. If you won't use final confirmations, you will easily encounter a data loss if one of the parties closes connection too early.

Run the PipeServer.exe server locally and attach the PipeClient.mq5 script to any chart:

MQL5 Pipe Server
Copyright 2012, MetaQuotes Software Corp.
Pipe '\\.\pipe\MQL5.Pipe.Server' created
Client: waiting for connection...
Client: connected as 'PipeClient.mq5 on MQL5 build 705'
Server: send string
Server: send integer
Server: read string
Server: 'Test string' received
Server: read integer
Server: 1234567890 received
Server: start benchmark
......................................................
........
Server: 1024 Mb sent at 2921 Mb per second

PipeClient (EURUSD,H1)  Client: pipe opened
PipeClient (EURUSD,H1)  Server: Hello from pipe server received
PipeClient (EURUSD,H1)  Server: 1234567890 received
PipeClient (EURUSD,H1)  Client: 1024 Mb received at 2921 Mb per second

For local exchange, transfer rate is truly amazing - almost 3 gigabytes per second. This means that named pipes can be used to transfer almost any amount of data into MQL5 programs.

Now let's benchmark data transfer performance in an ordinary 1 gigabit LAN:

MQL5 Pipe Server
Copyright 2012, MetaQuotes Software Corp.
Pipe '\\.\pipe\MQL5.Pipe.Server' created
Client: waiting for connection...
Client: connected as 'PipeClient.mq5 on MQL5 build 705'
Server: send string
Server: send integer
Server: read string
Server: 'Test string' received
Server: read integer
Server: 1234567890 received
Server: start benchmark
......................................................
........
Server: 1024 Mb sent at 63 Mb per second

PipeClient (EURUSD,H1)  Client: pipe opened
PipeClient (EURUSD,H1)  Server: Hello from pipe server received
PipeClient (EURUSD,H1)  Server: 1234567890 received
PipeClient (EURUSD,H1)  Client: 1024 Mb received at 63 Mb per second

In local network, 1 gigabyte of data has been transfered at rate of 63 megabytes per second, which is very good. In fact it is 63% of the gigabit network maximum bandwidth.

Conclusion

Protection system of the MetaTrader 5 trading platform does not allow MQL5 programs run outside their sandbox, guarding traders against threats when using untrusted Expert Advisors. Using named pipes you can easy create integrations with third-party software and manage EAs from outside. Safely.