Experts: MQL5 Programming for Traders – Source Codes from the Book. Part 7

 

MQL5 Programming for Traders – Source Codes from the Book. Part 7:

The final seventh part of the book discusses the advanced capabilities of the MQL5 API, which will be useful when developing programs for MetaTrader 5. These include custom financial symbols, built-in economic calendar events, and general-purpose technologies such as networking, databases, and cryptography.

MQL5 Programming for Traders – Source Codes from the Book. Part 7

Author: MetaQuotes

 
Here is small bugfixes and improvements for calendar cache and filter.
MQL5 Book: Advanced language tools / Economic calendar / Transferring calendar database to tester
MQL5 Book: Advanced language tools / Economic calendar / Transferring calendar database to tester
  • www.mql5.com
The calendar is available for MQL programs only online, and therefore testing news trading strategies poses some difficulties. One of the solutions...
Files:
 
More bugfixes and improvements for the economic calendar reading and exporting to CSV are published in the codebase. Specifically, the sorting algorithm is fixed for the case of mostly sorted large arrays (which are usually received from MQL5 calendar API), so that slow-downs and stack-overflows are eliminated.
Economic Calendar CSV
Economic Calendar CSV
  • www.mql5.com
This script saves a predefined set of economic events from the MetaTrader's built-in economic calendar into CSV file.
 
I'm attaching some bugfixes and improvements in the websockets classes.
Files:
wss.zip  16 kb
 
Stanislav Korotky #:
I'm attaching some bugfixes and improvements in the websockets classes.
Thank you for your efforts
 
Stanislav Korotky #:
More bugfixes and improvements for the economic calendar reading and exporting to CSV are published in the codebase. Specifically, the sorting algorithm is fixed for the case of mostly sorted large arrays (which are usually received from MQL5 calendar API), so that slow-downs and stack-overflows are eliminated.

There is a little bud on the msclient.mqh line 126 'return connection.handshake(url, host, origin, custom_headers);' passing the full 'url' instead of 'path' lead to error 403 on python websocket. I had to make that change b4 the EA could successfully connect to my python websocket server. 
I attached the server log from Postman (passed) and EA before I made the change (failed)

But here is the next issue, my server auto sends a keepalive ping at a configured interval but the MT5 websocket implementation doesn't seems to respond and this behavior cause the server to always drop the client connect immediately after the ping timeout.

Please any help with regards to the keepalive ping pong will be appreciated 

Files:
passed.png  88 kb
failed.png  50 kb
 
pauldic #:

There is a little bud on the msclient.mqh line 126 'return connection.handshake(url, host, origin, custom_headers);' passing the full 'url' instead of 'path' lead to error 403 on python websocket. I had to make that change b4 the EA could successfully connect to my python websocket server. 
I attached the server log from Postman (passed) and EA before I made the change (failed)

But here is the next issue, my server auto sends a keepalive ping at a configured interval but the MT5 websocket implementation doesn't seems to respond and this behavior cause the server to always drop the client connect immediately after the ping timeout.

Please any help with regards to the keepalive ping pong will be appreciated 

Hi, yes you can use path in the client call:

template<typename T>
class WebSocketClient: public IWebSocketObserver
{
   ...
   bool open(const string custom_headers = NULL)
   {
      ...
      connection = new T(&this, socket, compression);
      return connection.handshake(path, host, origin, custom_headers);
   }

On the other hand, the HTTP request syntax supports complete URI as a target, and I did tests with the original source codes against different servers (including the wss://ws.postman-echo.com/raw) and they worked normally. So I'm not sure if it's a violation or your python's implementation is too rigid in a sense.

As for ping/pong, it should work. Take a look at wsprotocol.mqh:

   // Handle incoming control frames: sends Pong on Ping and close connection after a Close request
   void processControlFrame(IWebSocketFrame *frame)
   {
      switch(frame.getType())
      {
      case WS_FRAME_OPCODE::WS_CLOSE_FRAME:
         if(closeRequested) // our close was confirmed
         {
            Print("Server close ack");
         }
         else if(!disconnecting) // server initiated close
         {
            if(openMessage) // still not finalized(!)
            {
               owner.onMessage(openMessage);
               openMessage = NULL;
            }
          
            WebSocketFrame temp(WS_FRAME_OPCODE::WS_CLOSE_FRAME); // send our ack
            sendFrame(&temp);
         }
         close();
         break;
      case WS_FRAME_OPCODE::WS_PING_FRAME:
         {
            IWebSocketFrame *temp = WebSocketFrame::create(WS_FRAME_OPCODE::WS_PONG_FRAME, frame.getData());
            sendFrame(temp);
            delete temp;
         }
         break;
      }
   }

So please debug this stuff on your side, since you have the server sending pings to your client.

 
BTW, I forgot to add new version of MQL5Book/StringUtils.mqh used in the ws/sources.
Files:
 
Stanislav Korotky #:
BTW, I forgot to add new version of MQL5Book/StringUtils.mqh used in the ws/sources.
@Stanislav Korotky thanks again for your help. Regarding the ping pong issue with python websocket server, upon further debugging I noticed that the server sends out the ping request as either text or binary data and the MT5 responds successfully only when a text based ping is been sent out. This helped my to track down and fix the issue.

The issue is on line 310 within the ws ping frame switch/case, when the websocket frame is been created:

IWebSocketFrame *temp = WebSocketFrame::create(WS_FRAME_OPCODE::WS_PONG_FRAME, frame.getData());

frame.getData() the server seems to only acknowledge the pong if the ping data is text/string based but fails if a binary ping was sent

thus I got to use the overloaded  frame.getData(uchar &buf[]) to get it work in both cases. I don't know if I would meet other use cases that (my change) will break but so far so good all seems okay for now.


            uchar data[];
            frame.getData(data);
            IWebSocketFrame *temp = WebSocketFrame::create(WS_FRAME_OPCODE::WS_PONG_FRAME, data);

Just out of curiosity, I am wondering why we have to manually check for messages at interval with the OnTimer() event handler, while I see something that seems like an event handler function OnMessage incoming messages. Though the current setup works as is but I will appreciate if you can enlighten more on why this approach 
 
pauldic #:
@Stanislav Korotky thanks again for your help. Regarding the ping pong issue with python websocket server, upon further debugging I noticed that the server sends out the ping request as either text or binary data and the MT5 responds successfully only when a text based ping is been sent out. This helped my to track down and fix the issue.

The issue is on line 310 within the ws ping frame switch/case, when the websocket frame is been created:

IWebSocketFrame *temp = WebSocketFrame::create(WS_FRAME_OPCODE::WS_PONG_FRAME, frame.getData());

frame.getData() the server seems to only acknowledge the pong if the ping data is text/string based but fails if a binary ping was sent

thus I got to use the overloaded  frame.getData(uchar &buf[]) to get it work in both cases. I don't know if I would meet other use cases that (my change) will break but so far so good all seems okay for now.


            uchar data[];
            frame.getData(data);
            IWebSocketFrame *temp = WebSocketFrame::create(WS_FRAME_OPCODE::WS_PONG_FRAME, data);

Just out of curiosity, I am wondering why we have to manually check for messages at interval with the OnTimer() event handler, while I see something that seems like an event handler function OnMessage incoming messages. Though the current setup works as is but I will appreciate if you can enlighten more on why this approach 

Thank you for your efforts. I'll consider including your edits in the source codes.

As for OnTimer and OnMessage, they are provided because you may want to implement different processing logic in your application. The reading from the sockets are blocking on MQL5 API level, with specified timeout. Hence, if you're interested only in waiting for new messages, their processing and generating responses automatically, you can do it in blocking mode (with infinite timeout in reading, and without OnTimer) - that is the app is a sort of frozen while waiting data. But if you need to provide a responsive UI for a user (which is the case with the chat app) then you set a small timeout for short reading attempts in OnTimer-handler (they can produce empty data if timed out) and monitor user actions inbetween.

 
Stanislav Korotky #:

Thank you for your efforts. I'll consider including your edits in the source codes.

As for OnTimer and OnMessage, they are provided because you may want to implement different processing logic in your application. The reading from the sockets are blocking on MQL5 API level, with specified timeout. Hence, if you're interested only in waiting for new messages, their processing and generating responses automatically, you can do it in blocking mode (with infinite timeout in reading, and without OnTimer) - that is the app is a sort of frozen while waiting data. But if you need to provide a responsive UI for a user (which is the case with the chat app) then you set a small timeout for short reading attempts in OnTimer-handler (they can produce empty data if timed out) and monitor user actions inbetween.

Thank you again @Stanislav Korotky for the clarification