WebRequest Ignoring Timeout Parameter in Experts Advisory (MT5 Build 1881)

[bolek42]

Hi there,

Im trying to set up a WebRequest in an Expert Advisory, wheras the server needs ~5-10 min to come up with the response. Even though I set the timeout to 3600 seconds, I get a "read timeout" after exactly 10 seconds.

The following code can be used to reproduce the behavior for MT5 Build 1881:

#property copyright ""
#property link      ""
#property version   "1.00"


int OnInit()
 {
   int ret;
   char data[];
   char result[];
   string result_hdr = NULL;
         
   printf("Request");
   StringToCharArray("random post data", data);
   ret = WebRequest("POST", "http://127.0.0.1/", "", 3600*1000, data, result,result_hdr);
   printf("return: %d %d", ret, GetLastError());
   printf("result: %s", CharArrayToString(result));
   return(INIT_SUCCEEDED);
  }

void OnDeinit(const int reason){}
void OnTick(){}

The output of this script will be the following:

2018.07.19 16:46:08.092 test (Adobe,H1) Request

2018.07.19 16:46:18.095 test (Adobe,H1) return: 1003 5203

2018.07.19 16:46:18.095 test (Adobe,H1) result: read timeout

To address the obvious:

- Error code 5203: "HTTP request failed"  Isn't actually that helpful

-Return Code 1003: The documentary says "Return Value: HTTP server response code or -1 for an error."  Different behavior as documented. This HTTP status code does not exist and the server does not return anything (can be reproduced by using "nc -l -p 80" as a fake Webserver.)

-The timeout is triggered after exactly 10 seconds, so the timeout parameter seems to be ignored.

Are there some other timeout values I don't know about or anything else, that could cause this behavior?

Thanks in advance ;)

[nicholish en]

bolek42:

Are there some other timeout values I don't know about or anything else, that could cause this behavior?

Thanks in advance ;)

WebRequest is a blocking call so I think the default timeout <= 10 seconds, however, no server should take longer than that to generate a response and if it does then the server logic isn't properly implemented. Assuming this is your server, you should return a 503 (server busy) status if the business logic has not completed. Here is an example python server. 

from time import sleep
from concurrent.futures import ThreadPoolExecutor
from flask import Flask
app = Flask(__name__)
@app.route('/')
def hello_mql():
    if hello_mql.future is None:
        #simulate heavy business logic
        hello_mql.future = ThreadPoolExecutor().submit(sleep, 60)
    if hello_mql.future.done():
        return 'Hello MQL!', 200
    return 'SERVER BUSY', 503
hello_mql.future = None

If the server is not yours then you could implement your own middle-man service with python using the same logic. 

import requests
from time import sleep
from concurrent.futures import ThreadPoolExecutor as Tpe
from flask import Flask
app = Flask(__name__)
def url_get():
    sleep(20)
    return requests.get('https://www.mql5.com')

@app.route('/')
def hello_mql():
    if hello_mql.future is None:
        hello_mql.future = Tpe().submit(url_get)
    if hello_mql.future.done():
        return (
            hello_mql.future.result().text, 
            hello_mql.future.result().status_code
        )
    return "BUSY", 503
hello_mql.future = None

[bolek42]

Thanks for your response :)

Unfortunately, returning a 503 would not solve my poroblem.

Is there any way to change this behaviour default timout? I can't find any thing in the docs, in an older MT5 version the timeout was propperly processed... 

A description of problem:

In fact I'm running a Python (django based) webserver/REST-api for interfacing tensorflow.

The idea is to make pair wise comparison of 300-500 stocks, resulting in up to 490k samples to be processed.

The timing behaviour is intended and acceptable for a daily call.

A certain speedup would be definetly possible, but not 60 times faster :/

I see two options here, first implement the following:

1. Client issues job, to be computed

2. Server executes job in an asynchronous way

3. Server returns some id

4. Client Polls until job id is done

5. Client Requests the final results

Or increase the timout...

I saw some references to WinHTTP in terminal64.exe, is that whats used internally for HTTP?

So maybe a call to WinHttpSetTimeouts could fix that problem?

Cheers

[Alain Verleyen]

bolek42:

Thanks for your response :)

Unfortunately, returning a 503 would not solve my poroblem.

Is there any way to change this behaviour default timout? I can't find any thing in the docs, in an older MT5 version the timeout was propperly processed... 

A description of problem:

In fact I'm running a Python (django based) webserver/REST-api for interfacing tensorflow.

The idea is to make pair wise comparison of 300-500 stocks, resulting in up to 490k samples to be processed.

The timing behaviour is intended and acceptable for a daily call.

A certain speedup would be definetly possible, but not 60 times faster :/

I see two options here, first implement the following:

1. Client issues job, to be computed

2. Server executes job in an asynchronous way

3. Server returns some id

4. Client Polls until job id is done

5. Client Requests the final results

Or increase the timout...

I saw some references to WinHTTP in terminal64.exe, is that whats used internally for HTTP?

So maybe a call to WinHttpSetTimeouts could fix that problem?

Cheers

If you are thinking it's a bug, please report it to ServiceDesk.

I don't understand why you need a WebServer to work with TensorFlow.

[bolek42]

Alain Verleyen:

If you are thinking it's a bug, please report it to ServiceDesk.

I don't understand why you need a WebServer to work with TensorFlow.

Thanks, I will ask there.

I'm using REST because it's a convenient way of providing an API, that's human readable and it can also be interfaced with HTML/JS to provide a GUI and any other language. But mainly, I had a lot of code already laying around ;)

Do you know a better way of using MQL with tensorflow?

[Alain Verleyen]

bolek42:

Thanks, I will ask there.

I'm using REST because it's a convenient way of providing an API, that's human readable and it can also be interfaced with HTML/JS to provide a GUI and any other language. But mainly, I had a lot of code already laying around ;)

Do you know a better way of using MQL with tensorflow?

"Better" is all relative. Look at this.

[nicholish en]

bolek42:

Thanks for your response :)

Unfortunately, returning a 503 would not solve my poroblem.

Is there any way to change this behaviour default timout? I can't find any thing in the docs, in an older MT5 version the timeout was propperly processed... 

A description of problem:

In fact I'm running a Python (django based) webserver/REST-api for interfacing tensorflow.

The idea is to make pair wise comparison of 300-500 stocks, resulting in up to 490k samples to be processed.

The timing behaviour is intended and acceptable for a daily call.

A certain speedup would be definetly possible, but not 60 times faster :/

I see two options here, first implement the following:

1. Client issues job, to be computed

2. Server executes job in an asynchronous way

3. Server returns some id

4. Client Polls until job id is done

5. Client Requests the final results

Or increase the timout...

I saw some references to WinHTTP in terminal64.exe, is that whats used internally for HTTP?

So maybe a call to WinHttpSetTimeouts could fix that problem?

Cheers

You absolutely 100% need to avoid your EA hanging on a blocked webrequest call. This is why you need to program concurrency into your business logic. The flow should be something like:

EA:

while(!is_tensor_signal_ready(response))
   Sleep(1000);

[bolek42]

nicholi shen:

You absolutely 100% need to avoid your EA hanging on a blocked webrequest call. This is why you need to program concurrency into your business logic. The flow should be something like:

EA:

Is there any reason for that? The EA runs in a separate thread, as far as I know, therefore waiting on any blocking call should be no problem. Both recv() and sleep() should hand the execution back to the kernel. I guess there is no Metatrader magic happening while sleeping, that does not happen with recv(). Even though if this were true, I could do the following to receive a blocking HTTP request (pseudocode):

//send HTTP request with "Connection: close"

//receive response
result = ""
while (connection_open(socket)) {
        result += recv(socket); //Non blocking recv
        Sleep(1000);
}
return result;

As my service desk request is stucked in the endless depth of the ticket wall for further investigation, I ended up implementing an HTTP client myself, what actually solved my problem... It is based on this nice article on WinSockets in MQL: https://www.mql5.com/ru/articles/2599 . As a positive side effect, I'm not restricted to port 80 anymore and can remove the (useless) proxy server. Here is the code if anyone comes across the same problem:

#property copyright "Based on https://www.mql5.com/ru/articles/2599 (Alexey Sergeev) | Beerware License"
#property link          "bolek42.github.io"

#define BYTE                 uchar
#define WORD                ushort
#define DWORD             int
#define DWORD_PTR     ulong
#define SOCKET            uint

#define MAKEWORD(a, b)      ((WORD)(((BYTE)(((DWORD_PTR)(a)) & 0xff)) | ((WORD)((BYTE)(((DWORD_PTR)(b)) & 0xff))) << 8))

#define WSADESCRIPTION_LEN      256
#define WSASYS_STATUS_LEN       128

#define INVALID_SOCKET  (SOCKET)(~0)
#define SOCKET_ERROR    (-1)
#define NO_ERROR        0
#define SOMAXCONN     128

#define AF_INET         2 // internetwork: UDP, TCP, etc.
#define SOCK_STREAM     1
#define IPPROTO_TCP      6

#define SD_RECEIVE    0x00
#define SD_SEND         0x01
#define SD_BOTH         0x02

#define IOCPARM_MASK    0x7f
#define IOC_IN           0x80000000
#define _IOW(x,y,t)     (IOC_IN|(((int)sizeof(t)&IOCPARM_MASK)<<16)|((x)<<8)|(y))
#define FIONBIO         _IOW('f', 126, int)
//------------------------------------------------------------------    struct WSAData
struct WSAData
  {
   WORD              wVersion;
   WORD              wHighVersion;
   char              szDescription[WSADESCRIPTION_LEN+1];
   char              szSystemStatus[WSASYS_STATUS_LEN+1];
   ushort            iMaxSockets;
   ushort            iMaxUdpDg;
   char              lpVendorInfo[];
  };

#define LPWSADATA               char&
//------------------------------------------------------------------    struct sockaddr_in
struct sockaddr_in
  {
   ushort            sin_family;
   ushort            sin_port;
   ulong             sin_addr; //struct in_addr { ulong s_addr; };
   char              sin_zero[8];
  };
//------------------------------------------------------------------    struct sockaddr
struct sockaddr
  {
   ushort            sa_family;
   char              sa_data[14];
  };
#define LPSOCKADDR      char&

struct ref_sockaddr { char ref[2+14]; };

//------------------------------------------------------------------    import Ws2_32.dll
#import "Ws2_32.dll"
int WSAStartup(WORD wVersionRequested,LPWSADATA lpWSAData[]);
int WSACleanup();
int WSAGetLastError();

ushort htons(ushort hostshort);
ulong inet_addr(char& cp[]);
string inet_ntop(int Family,ulong &pAddr,char &pStringBuf[],uint StringBufSize);
ushort ntohs(ushort netshort);

SOCKET socket(int af,int type,int protocol);
int ioctlsocket(SOCKET s,int cmd,int &argp);
int shutdown(SOCKET s,int how);
int closesocket(SOCKET s);

// Server Stuff
int bind(SOCKET s,LPSOCKADDR name[],int namelen);
int listen(SOCKET s,int backlog);
SOCKET accept(SOCKET s,LPSOCKADDR addr[],int &addrlen);

// Client Stuff
int connect(SOCKET s,LPSOCKADDR name[],int namelen);
int send(SOCKET s,char &buf[],int len,int flags);
int recv(SOCKET s,char &buf[],int len,int flags);

#import

string httpRequest(string host, ushort port, string url, string data) {
   //init WinSock
   char wsaData[];
   ArrayResize(wsaData,sizeof(WSAData));
   int res = WSAStartup(MAKEWORD(2,2), wsaData);
   if(res != 0) {
      Print("-WSAStartup failed: " + string(res));
      return NULL;
   }

   //Create Socket
   SOCKET s = socket(AF_INET,SOCK_STREAM,IPPROTO_TCP);
   if(s == INVALID_SOCKET) {
      Print("-Create failed error: "+ string(WSAGetLastError()));
      WSACleanup(); 
      return NULL;
   }

   //Connect
   char ch[]; StringToCharArray(host, ch);
   sockaddr_in addrin;
   addrin.sin_family = AF_INET;
   addrin.sin_addr = inet_addr(ch);
   addrin.sin_port = htons(port);
   ArrayFree(ch);

   //cast struct
   ref_sockaddr ref;
   ArrayInitialize(ref.ref, (char)0x00);
   ref.ref[0] = addrin.sin_family & 0xff;
   ref.ref[1] = addrin.sin_family >> 8;
   ref.ref[2] = addrin.sin_port & 0xff;
   ref.ref[3] = addrin.sin_port >> 8;
   for (int i=0; i < sizeof(ulong); i++) ref.ref[4+i] = (addrin.sin_addr >> (8*i)) & 0xff;
   res=connect(s, ref.ref, sizeof(addrin));
   
   if(res == SOCKET_ERROR) {
      int err = WSAGetLastError();
      Print("Connection Failed"+string(WSAGetLastError()));
      closesocket(s);
      WSACleanup();
      return NULL;
   }
     
   //Assemble header
   string hdr =  ((StringLen(data) == 0) ? "GET ": "POST ") + url + " HTTP/1.1\r\n";
   hdr +=         "Host: " + host + "\r\n";
   hdr +=         "Connection: close\r\n";
   hdr +=         "Content-Length: " + string(StringLen(data)) + "\r\n";
   hdr +=         "\r\n" + data;
   char buf[];
   StringToCharArray(hdr, buf);
   send(s,buf, StringLen(hdr), 0);
   printf(hdr);
   
   //receive data
   string ret = "";
   int nbytes;
   ArrayResize(buf, 1024);
   do {
      ArrayInitialize(buf, (char)0x00);
      nbytes = recv(s,buf, 1023, 0);
      ret = ret + CharArrayToString(buf);
   } while (nbytes > 0);
     
   closesocket(s);
   WSACleanup();
   ArrayFree(buf);
   
   return StringSubstr(ret, StringFind(ret, "\r\n\r\n")+4);
}

P.S. Why is there a copyright on such tutorials?! Technically I am not allowed to copy&paste anything from this article without permission. That somehow contradicts the purpose of a tutorial...