Download MetaTrader 5

Communicating With MetaTrader 5 Using Named Pipes Without Using DLLs

15 October 2012, 21:16
MetaQuotes Software Corp.
8
12 134

Introduction

Many developers face the same problem - how to get to the trading terminal sandbox without using unsafe DLLs.

One of the easiest and safest method is to use standard Named Pipes that work as normal file operations. They allow you to organize interprocessor client-server communication between programs. Although there is an already published article A DLL-free solution to communicate between MetaTrader 5 terminals using Named Pipes on this topic that demonstrates enabling access to DLLs, we will use standard and safe features of client terminal.

You can find more information about named pipes in MSDN library, but we will get down to practical examples in C++ and MQL5. We will implement server, client, data exchange between them and then benchmark performance.


Server Implementation

Let's code a simple server in C++. A script from the terminal will connect to this server and will exchange data with it. The server core has the following set of WinAPI functions:

Once a named pipe is opened it returns a file handle that can be used for regular read/write file operations. As a result you get a very simple mechanism that don't require any special knowledge in network operations.

Named pipes have one distinctive feature - they can be both local and network. That is, it's easy to implement a remote server that will accept network connections from client terminals.

Here is a simple example of creating a local server as full-duplex channel that works in the bytes exchange mode:

//--- open 
CPipeManager manager;

if(!manager.Create(L"\\\\.\\pipe\\MQL5.Pipe.Server"))
   return(-1);


//+------------------------------------------------------------------+
//| Create named pipe                                                |
//+------------------------------------------------------------------+
bool CPipeManager::Create(LPCWSTR pipename)
  {
//--- check parameters
   if(!pipename || *pipename==0) return(false);
//--- close old
   Close();
//--- create named pipe
   m_handle=CreateNamedPipe(pipename,PIPE_ACCESS_DUPLEX,
                            PIPE_TYPE_BYTE | PIPE_READMODE_BYTE | PIPE_WAIT,
                            PIPE_UNLIMITED_INSTANCES,256*1024,256*1024,1000,NULL);

   if(m_handle==INVALID_HANDLE_VALUE)
     {
      wprintf(L"Creating pipe '%s' failed\n",pipename);
      return(false);
     }
//--- ok
   wprintf(L"Pipe '%s' created\n",pipename);
   return(true);
  }

To get a client connection you have to use the ConnectNamedPipe function:

//+------------------------------------------------------------------+
//| Connect client                                                   |
//+------------------------------------------------------------------+
bool CPipeManager::ConnectClient(void)
  {
//--- pipe exists?
   if(m_handle==INVALID_HANDLE_VALUE) return(false);
//--- connected?
   if(!m_connected)
     {
      //--- connect
      if(ConnectNamedPipe(m_handle,NULL)==0)
        {
         //--- client already connected before ConnectNamedPipe?
         if(GetLastError()!=ERROR_PIPE_CONNECTED)
            return(false);
         //--- ok
        }
      m_connected=true;
     }
//---
   return(true);
  }

Data exchange is organized using 4 simple functions:

  • CPipeManager::Send(void *data,size_t data_size)
  • CPipeManager::Read(void *data,size_t data_size)
  • CPipeManager::SendString(LPCSTR command)
  • CPipeManager::ReadString(LPSTR answer,size_t answer_maxlen)

They allow you to send/receive data as binary data or ANSI text strings in MQL5 compatible mode. Moreover, since CFilePipe in MQL5 opens a file in ANSI mode by default, strings are automatically converted to Unicode on receipt and sending. If your MQL5 program opens a file in Unicode mode (FILE_UNICODE), then it can exchange Unicode strings (with BOM starting signature).


Client Implementation

We will write our client in MQL5. It will be able to perform regular file operations using the CFilePipe class from Standard Library. This class is almost identical to the CFileBin, but it contains an important verification of data availability in a virtual file before reading this data.

//+------------------------------------------------------------------+
//| Wait for incoming data                                           |
//+------------------------------------------------------------------+
bool CFilePipe::WaitForRead(const ulong size)
  {
//--- check handle and stop flag
   while(m_handle!=INVALID_HANDLE && !IsStopped())
     {
      //--- enough data?
      if(FileSize(m_handle)>=size)
         return(true);
      //--- wait a little
      Sleep(1);
     }
//--- failure
   return(false);
  }

//+------------------------------------------------------------------+
//| Read an array of variables of double type                        |
//+------------------------------------------------------------------+
uint CFilePipe::ReadDoubleArray(double &array[],const int start_item,const int items_count)
  {
//--- calculate size
   uint size=ArraySize(array);
   if(items_count!=WHOLE_ARRAY) size=items_count;
//--- check for data
   if(WaitForRead(size*sizeof(double)))
      return FileReadArray(m_handle,array,start_item,items_count);
//--- failure
   return(0);
  }

Named pipes have significant differences in implementation of their local and network modes. Without such a verification, network mode operations will always return a read error when sending large amounts of data (over 64K).

Let's connect to the server with two checks: either to remote computer named 'RemoteServerName' or to local machine.

void OnStart()
  {
//--- wait for pipe server
   while(!IsStopped())
     {
      if(ExtPipe.Open("\\\\RemoteServerName\\pipe\\MQL5.Pipe.Server",FILE_READ|FILE_WRITE|FILE_BIN)!=INVALID_HANDLE) break;
      if(ExtPipe.Open("\\\\.\\pipe\\MQL5.Pipe.Server",FILE_READ|FILE_WRITE|FILE_BIN)!=INVALID_HANDLE) break;
      Sleep(250);
     }
   Print("Client: pipe opened");


Data Exchange

After successful connection let's send a text string with identification info to the server. Unicode string will be automatically converted into ANSI, since the file is opened in ANSI mode.

//--- send welcome message
   if(!ExtPipe.WriteString(__FILE__+" on MQL5 build "+IntegerToString(__MQ5BUILD__)))
     {
      Print("Client: sending welcome message failed");
      return;
     }

In response, the server will send its string "Hello from pipe server" and the integer 1234567890. The client once again will send string "Test string" and the integer 1234567890.

//--- read data from server
   string        str;
   int           value=0;

   if(!ExtPipe.ReadString(str))
     {
      Print("Client: reading string failed");
      return;
     }
   Print("Server: ",str," received");

   if(!ExtPipe.ReadInteger(value))
     {
      Print("Client: reading integer failed");
      return;
     }
   Print("Server: ",value," received");
//--- send data to server
   if(!ExtPipe.WriteString("Test string"))
     {
      Print("Client: sending string failed");
      return;
     }

   if(!ExtPipe.WriteInteger(value))
     {
      Print("Client: sending integer failed");
      return;
     }

OK, we are finished with simple data exchange. Now it's time for performance benchmark.


Performance Benchmark

As a test, we will send 1 gigabyte of data as an array of the double type numbers in blocks of 8 megabytes from server to client, then check correctness of the blocks and measure the transfer rate.

Here is this code in C++ server:

//--- benchmark
   double  volume=0.0;
   double *buffer=new double[1024*1024];   // 8 Mb

   wprintf(L"Server: start benchmark\n");
   if(buffer)
     {
      //--- fill the buffer
      for(size_t j=0;j<1024*1024;j++)
         buffer[j]=j;
      //--- send 8 Mb * 128 = 1024 Mb to client
      DWORD   ticks=GetTickCount();

      for(size_t i=0;i<128;i++)
        {
         //--- setup guard signatures
         buffer[0]=i;
         buffer[1024*1024-1]=i+1024*1024-1;
         //--- 
         if(!manager.Send(buffer,sizeof(double)*1024*1024))
           {
            wprintf(L"Server: benchmark failed, %d\n",GetLastError());
            break;
           }
         volume+=sizeof(double)*1024*1024;
         wprintf(L".");
        }
      wprintf(L"\n");
      //--- read confirmation
      if(!manager.Read(&value,sizeof(value)) || value!=12345)
         wprintf(L"Server: benchmark confirmation failed\n");
      //--- show statistics
      ticks=GetTickCount()-ticks;
      if(ticks>0)
         wprintf(L"Server: %.0lf Mb sent at %.0lf Mb per second\n",volume/1024/1024,volume/1024/ticks);
      //---
      delete[] buffer;
     }

and in MQL5 client:

//--- benchmark
   double buffer[];
   double volume=0.0;

   if(ArrayResize(buffer,1024*1024,0)==1024*1024)
     {
      uint  ticks=GetTickCount();
      //--- read 8 Mb * 128 = 1024 Mb from server
      for(int i=0;i<128;i++)
        {
         uint items=ExtPipe.ReadDoubleArray(buffer);
         if(items!=1024*1024)
           {
            Print("Client: benchmark failed after ",volume/1024," Kb, ",items," items received");
            break;
           }
         //--- check the data
         if(buffer[0]!=i || buffer[1024*1024-1]!=i+1024*1024-1)
           {
            Print("Client: benchmark invalid content");
            break;
           }
         //---
         volume+=sizeof(double)*1024*1024;
        }
      //--- send confirmation
      value=12345;
      if(!ExtPipe.WriteInteger(value))
         Print("Client: benchmark confirmation failed ");
      //--- show statistics
      ticks=GetTickCount()-ticks;
      if(ticks>0)
         printf("Client: %.0lf Mb received at %.0lf Mb per second\n",volume/1024/1024,volume/1024/ticks);
      //---
      ArrayFree(buffer);
     }

Note, that the first and the last elements of transfered blocks are checked in order to make sure that there were no errors during the transfer. Also, when transfer is complete client sends a confirming signal to the server about successful data receipt. If you won't use final confirmations, you will easily encounter a data loss if one of the parties closes connection too early.

Run the PipeServer.exe server locally and attach the PipeClient.mq5 script to any chart:

PipeServer.exe PipeClient.mq5
MQL5 Pipe Server
Copyright 2012, MetaQuotes Software Corp.
Pipe '\\.\pipe\MQL5.Pipe.Server' created
Client: waiting for connection...
Client: connected as 'PipeClient.mq5 on MQL5 build 705'
Server: send string
Server: send integer
Server: read string
Server: 'Test string' received
Server: read integer
Server: 1234567890 received
Server: start benchmark
......................................................
........
Server: 1024 Mb sent at 2921 Mb per second
PipeClient (EURUSD,H1)  Client: pipe opened
PipeClient (EURUSD,H1)  Server: Hello from pipe server received
PipeClient (EURUSD,H1)  Server: 1234567890 received
PipeClient (EURUSD,H1)  Client: 1024 Mb received at 2921 Mb per second


For local exchange, transfer rate is truly amazing - almost 3 gigabytes per second. This means that named pipes can be used to transfer almost any amount of data into MQL5 programs.

Now let's benchmark data transfer performance in an ordinary 1 gigabit LAN:

PipeServer.exe PipeClient.mq5
MQL5 Pipe Server
Copyright 2012, MetaQuotes Software Corp.
Pipe '\\.\pipe\MQL5.Pipe.Server' created
Client: waiting for connection...
Client: connected as 'PipeClient.mq5 on MQL5 build 705'
Server: send string
Server: send integer
Server: read string
Server: 'Test string' received
Server: read integer
Server: 1234567890 received
Server: start benchmark
......................................................
........
Server: 1024 Mb sent at 63 Mb per second
PipeClient (EURUSD,H1)  Client: pipe opened
PipeClient (EURUSD,H1)  Server: Hello from pipe server received
PipeClient (EURUSD,H1)  Server: 1234567890 received
PipeClient (EURUSD,H1)  Client: 1024 Mb received at 63 Mb per second


In local network, 1 gigabyte of data has been transfered at rate of 63 megabytes per second, which is very good. In fact it is 63% of the gigabit network maximum bandwidth.


Conclusion

Protection system of the MetaTrader 5 trading platform does not allow MQL5 programs run outside their sandbox, guarding traders against threats when using untrusted Expert Advisors. Using named pipes you can easy create integrations with third-party software and manage EAs from outside. Safely.

Translated from Russian by MetaQuotes Software Corp.
Original article: https://www.mql5.com/ru/articles/503

Attached files |
pipeclient.mq5 (3.15 KB)
pipeserver.zip (43.59 KB)
Last comments | Go to discussion (8)
Alain Verleyen
Alain Verleyen | 16 Nov 2013 at 13:18
vanvolxem:
I would test pipeclient.mq5 application, but the link load is not correct,

I said that I am not computer.

I would have wanted to create a database for analysis outside of MT5 because my trading system is based on correlation thank you for your response

Ok, I see. Reported to Service Desk.

You can download this file directly from your MT5 platform. Simply use the search tool with pipeclient keyword.

vanvolxem
vanvolxem | 21 Nov 2013 at 16:01
angevoyageur:

Ok, I see. Reported to Service Desk.

You can download this file directly from your MT5 platform. Simply use the search tool with pipeclient keyword.

Thank you, it works 
Simon Gniadkowski
Simon Gniadkowski | 21 Nov 2013 at 17:48
vanvolxem:
I would test pipeclient.mq5 application, but the link load is not correct,

I said that I am not computer.

I would have wanted to create a database for analysis outside of MT5 because my trading system is based on correlation thank you for your response
Try here:  pipeclient.mq5
Carl Schreiber
Carl Schreiber | 6 May 2016 at 09:53

Is any body using this client-sever? For what and how?

I read at the beginning:

"A script from the terminal will connect to this server and will exchange data with it."

May I ask how can I connect two clients running on two different terminals via this server? As far as I understand the server and the tests only one client connects to the server and what can this server do with the messages it has received?

Wouldn't it be necessary to have

1) an independent server waiting for multiple clients,

2) each client to start the connection by sending a unique ID of itself,

3) a server sending to every connected client any message that the server has received - may be except to the client it has got the message from,

4) a server sending each message beginning with the unique ID of the client which has sent the message?

- This way the client can detect its own messages and delete them - if they were send back by the server, by what ever reason.

- This way the client can detect the client(s) it should listen too - must be part of the mq4-code of the client.

newgel
newgel | 23 Aug 2016 at 20:42

Solution presented here does not work with Async Named Pipes. Please see https://www.mql5.com/en/forum/94343

 Newgel 

Step on New Rails: Custom Indicators in MQL5 Step on New Rails: Custom Indicators in MQL5

I will not list all of the new possibilities and features of the new terminal and language. They are numerous, and some novelties are worth the discussion in a separate article. Also there is no code here, written with object-oriented programming, it is a too serous topic to be simply mentioned in a context as additional advantages for developers. In this article we will consider the indicators, their structure, drawing, types and their programming details, as compared to MQL4. I hope that this article will be useful both for beginners and experienced developers, maybe some of them will find something new.

Here Comes the New MetaTrader 5 and MQL5 Here Comes the New MetaTrader 5 and MQL5

This is just a brief review of MetaTrader 5. I can't describe all the system's new features for such a short time period - the testing started on 2009.09.09. This is a symbolical date, and I am sure it will be a lucky number. A few days have passed since I got the beta version of the MetaTrader 5 terminal and MQL5. I haven't managed to try all its features, but I am already impressed.

Using text files for storing input parameters of Expert Advisors, indicators and scripts Using text files for storing input parameters of Expert Advisors, indicators and scripts

The article describes the application of text files for storing dynamic objects, arrays and other variables used as properties of Expert Advisors, indicators and scripts. The files serve as a convenient addition to the functionality of standard tools offered by MQL languages.

How to create an indicator of non-standard charts for MetaTrader Market How to create an indicator of non-standard charts for MetaTrader Market

Through offline charts, programming in MQL4, and reasonable willingness, you can get a variety of chart types: "Point & Figure", "Renko", "Kagi", "Range bars", equivolume charts, etc. In this article, we will show how this can be achieved without using DLL, and therefore such "two-for-one" indicators can be published and purchased from the Market.