Other

Socket library for MT4 and MT5

6 September 2017, 08:59
JC
JC
104
32 955

[Published as a blog entry because submission to the Codebase stalled on the fact that this socket library works with both MT4 and MT5, whereas the Codebase is divided into separate sections for MT4 and MT5. The support desk couldn't work out how to handle this...]

[Updated 2019-07-16. New versions of the code, with support for binary send and receive.]

Socket library, for both MT4 and MT5 (32-bit and 64-bit).

The attached code, though uploaded as .mq4, works as either .mq4 or .mq5

Features:

  • Both client and server sockets
  • Both send and receive
  • Both MT4 and MT5 (32-bit and 64-bit)
  • Optional event-driven handling (EAs only, not scripts or indicators), offering faster responses to socket events than OnTimer()
  • Direct use of Winsock; no need for a custom DLL sitting between this code and ws2_32.dll

Based on the following forum posts:


Client sockets

You create a connection to a server using one of the following two constructors for ClientSocket:

   ClientSocket(ushort localport);
   ClientSocket(string HostnameOrIPAddress, ushort port);

The first connects to a port on localhost (127.0.0.1). The second connects to a remote server, which can be specified either by passing an IP address such as "123.123.123.123" or a hostname such as "www.myserver.com".

After creating the instance of the class, and periodically afterwards, you should check the value of IsSocketConnected(). If false, then the connection has failed (or has later been closed). You then need to destroy the class and create a new connection. One common pattern of usage therefore looks like the following:

   ClientSocket * glbConnection = NULL;

   void OnTick()
   {
      // Create a socket if none already exists
      if (!glbConnection) glbConnection = new ClientSocket(12345);
      
      if (glbConnection.IsSocketConnected()) {
         // Socket is okay. Do some action such as sending or receiving
      }
      
      // Socket may already have been dead, or now detected as failed
      // following the attempt above at sending or receiving.
      // If so, delete the socket and try a new one on the next call to OnTick()
      if (!glbConnection.IsSocketConnected()) {
         delete glbConnection;
         glbConnection = NULL;            
      }
   }

You send data down a socket using the simple Send() method, which takes a string parameter. Any failure to send returns false, which will also mean that IsSocketConnected() then returns false. The format of the data which you are sending to the server is obviously entirely up to you. For example:

      string strMsg = Symbol() + "," + DoubleToString(SymbolInfoDouble(Symbol(), SYMBOL_BID), 6) + "\r\n";
      if (!glbClientSocket.Send(strMsg)) {
         // Send failed. Socket is presumably dead, and
         // .IsSocketConnected() will now return false
      }

You can receive pending incoming data on a socket using Receive(), which returns either the pending data or an empty string. You will normally want to call Receive() from OnTimer(), or using the event handling described below.

A non-blank return value from Receive() does not necessarily mean that the socket is still active. The server may have sent some data  and also closed the socket. 

   string strMessage = MySocket.Receive();
   if (strMessage != "") {
      // Process the message
   }
   
   // Regardless of whether there was any data, the socket may
   // now be dead.
   if (!MySocket.IsSocketConnected()) {
      // ... socket has been closed
   }

You can also give Receive() an optional message terminator, such as "\r\n". It will then store up data, and only return complete messages (minus the terminator). If you use a terminator then there may have been multiple complete messages since your last call to Receive(), and you should keep calling Receive() until it returns an empty string, in order to collect all the messages. For example:

   string strMessage;
   do {
      strMessage = MySocket.Receive("\r\n");
      if (strMessage != "") {
         // Do something with the message
      }
   } while (strMessage != "");

You close a socket simply by destroying the ClientSocket object.


Server sockets

For anyone not used to socket programming: the model is that you create a server socket; you accept connections on it; and each acceptance creates a new socket for communicating with that client. No data is sent or received through the server socket itself.

You create a server socket using an instance of ServerSocket(), telling the constructor a port number to listen on, and whether to accept connections only from the local machine or from any remote computer (subject to firewall rules etc).

   ServerSocket(ushort ServerPort, bool ForLocalhostOnly);

You should check the value of Created() after creating the ServerSocket() object. Any failure will usually be because something is already listening on your chosen port.

   MyServerSocket = new ServerSocket(12345, true);
   if (!MyServerSocket.Created()) {
      // Almost certainly because port 12345 is already in use
   }

You must be careful to destroy any server sockets which you create. If you don't, the port will remain in use and locked until MT4/5 is shut down, and you (or any other program) will not be able to create a new socket on that port. The normal way of handling this is to do destruction in OnDeinit():

   ServerSocket * glbServerSocket;
   ...
   
   void OnDeinit(const int reason)
   {
      if (glbServerSocket) delete glbServerSocket;
   }

You accept incoming connections using Accept(), typically from periodic checks in OnTimer() or using the event handling described below. Accept() returns either NULL if there is no waiting client, or a new instance of ClientSocket() which you then use for communicating with the client. There can be multiple simultaneous new connections, and you will therefore typically want to keep calling Accept() until it returns NULL. For example:

   ClientSocket * pNewClient;
   do {
      pNewClient = MyServerSocket.Accept();
      if (pNewClient) {
         // Store client socket for future use.
         // Must remember to delete it when finished, to avoid memory leaks.
      }
   } (while pNewClient != NULL);

To repeat the overview above: Accept() gives you a new instance of ClientSocket (which you must later delete). You then communicate with the client using the Send() and Receive() on ClientSocket. No data is ever sent or received through the server socket itself.

For an example of storing, and subsequently processing, a list of client connections, see the example server code.


Event-driven handling

The timing infrastructure in Windows does not normally have millisecond granularity. EventSetMillisecondTimer(1) is usually in fact equivalent to EventSetMillisecondTimer(16). Therefore, checks for socket activity in OnTimer() potentially have a delay of at least 16 milliseconds before you respond to a new connection or incoming data.

In an EA (but not a script or indicator) you can achieve <1ms response times using event-driven handling. The way this works is that the socket library generates dummy key-down messages to OnChartEvent() when socket activity occurs. Responding to these events can be significantly faster than a periodic check in OnTimer().

You need to request the event-driven handling by #defining SOCKET_LIBRARY_USE_EVENTS before including the library. For example:

   #define SOCKET_LIBRARY_USE_EVENTS
   #include <socket-library-mt4-mt5.mqh>

(Note that this has no effect in a custom indicator or script. It only works with EAs.)

You then process notifications in OnChartEvent as follows:

   void OnChartEvent(const int id, const long& lparam, const double& dparam, const string& sparam)
   {
      if (id == CHARTEVENT_KEYDOWN) {
         // May be a real key press, or a dummy notification
         // resulting from socket activity. If lparam matches
         // any .GetSocketHandle() then it's from a socket.
         // If not, it's a real key press. (If lparam>256 then
         // it's also pretty reliably a socket message rather 
         // than a real key press.)
         
         if (lparam == MyServerSocket.GetSocketHandle()) {
            // Activity on a server socket
         } else if (lparam == MyClientSocket.GetSocketHandle()) {
            // Activity on a client socket
         } else {
            // Doesn't match a socket. Assume real key pres
         }
      }
   }

For a comprehensive example of using the event-driven handling, see the example socket server code.


Sending and receiving binary data

The socket library has alternative versions of the ::Send() and ::Receive() functions which take/return an array rather than a string. These can be used for sending and receiving binary data over a socket. For example:

   // Read file into arrFile[], and close
   uchar arrFile[];
   int f = FileOpen(FileToSend, FILE_BIN | FILE_READ);
   FileReadArray(f, arrFile);
   FileClose(f);

   // Send the binary data down the socket
   socket.Send(arrFile);

   int bytesReceived = 0
   do {
      uchar receivedData[];
      bytesReceived = socket.Receive(receivedData);
      // Add the receivedData[] to some sort of store...
   } while (bytesReceived > 0);

However... TCP/IP communication is not message-based. When you use the string versions of ::Send() and ::Receive() you have the option of a message terminator such as \r\n. In effect, this is implementing a simple messaging protocol over the top of TCP/IP. If you use the array versions of ::Send() and ::Receive() then you have to implement the messaging protocol yourself.

For example, if you send a file as binary data then the server/receiver needs some way of identifying when the complete file has been received. You would typically do something like the following: send a header meaning "I am about to send a file"; then send the size of the file; and then send the actual data. The server/receiver would then need to run that process in reverse: look for the header; read the file size; and add data to a queue until it has received the entire expected amount of file data.

There is an alternative which is less efficient but a lot simpler - and which, despite inefficiency, will nevertheless be perfectly adequate for most real-life usage.

The example file-send script which is included with the library sends files by doing base64 encoding of the file data, and then sending that as a string. As a result, it can continue to make use of the \r\n terminator and simple messaging protocol which the library provides. The base64 encoding is inefficient because it increases the size of the data-send by 33%, and because it involves repeated string concatenations, but you should nevertheless find that this simple approach is perfectly adequate for almost all conceivable real-life uses.


Comparison with built-in MT5 sockets

After this library was originally written, MT5 (but not MT4) gained native socket handling via new functions such as SocketCreate().

The main advantage of these built-in MT5 functions is that they can support TLS. If you need to be able to send TLS-encrypted data over a socket then they are probably the best option. The only way of using TLS with this socket library is via a proxy such as stunnel: you send from the MQL4/5 code to stunnel, and stunnel then forwards that as a TLS-encrypted connection.

Disadvantages of the built-in MT5 sockets are as follows - may change as time progresses:

  • MT5 only. No support in MT4.
  • Client sockets only. No server sockets.
  • No event-driven handling
  • Binary send and receive only. You have to implement your own messaging over the top of that, rather than making use of the library's string-send and optional message terminator.
  • Doesn't require "Allow DLL imports", but does require each endpoint to be authorised in the same way as for WebRequest() etc


Notes on MT4/5 cross-compatibility

It appears to be safe for a 64-bit application to use 4-byte socket handles, despite the fact that the Win32 SDK defines SOCKET as 8-byte on x64. Nevertheless, this code uses 8-byte handles when running on 64-bit MT5.

The area which definitely does cause problems is gethostbyname(), because it returns a memory block containing pointers whose size depends on the environment. (The issue here is 32-bit vs 64-bit, not MT4 vs MT5.)

This code not only needs to handle 4-byte vs 8-byte memory pointers. The further problem is that MQL5 has no integer data type whose size varies with the environment. Therefore, it's necessary to have two versions of the #import of gethostbyname(), and to force the compiler to use the applicable one despite the fact that they only really vary by their return type. Manipulating the hostent* returned by gethostbyname() is then very ugly, made doubly so by the need for different paths of execution on 32-bit and 64-bit, using a range of different #imports of the RtlMoveMemory() function which the code uses to process the pointers.


#MT4 MT5 MQL4 MQL5 WinSock socket bind listen htons
Share it with friends: