WebRequest bug - page 3

 

In Chrome:

Request1:

GET https://mql-crypt.herokuapp.com/index.php?param1=Na6dIwFMr4weGnRNTYewxrpffSiYDhl0un02dGYEMv1lThKoYuUEk1Q+Dmfk477k HTTP/1.1
Host: mql-crypt.herokuapp.com
Connection: keep-alive
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.138 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
Sec-Fetch-Site: none
Sec-Fetch-Mode: navigate
Sec-Fetch-User: ?1
Sec-Fetch-Dest: document
Accept-Encoding: gzip, deflate, br
Accept-Language: fa,en-US;q=0.9,en;q=0.8

Response1:

Array
(
    [param1] => Na6dIwFMr4weGnRNTYewxrpffSiYDhl0un02dGYEMv1lThKoYuUEk1Q Dmfk477k
)

Request2:

GET https://mql-crypt.herokuapp.com/index.php?param1=Na6dIwFMr4weGnRNTYewxrpffSiYDhl0un02dGYEMv1lThKoYuUEk1Q%2BDmfk477k HTTP/1.1
Host: mql-crypt.herokuapp.com
Connection: keep-alive
Cache-Control: max-age=0
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.138 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
Sec-Fetch-Site: none
Sec-Fetch-Mode: navigate
Sec-Fetch-User: ?1
Sec-Fetch-Dest: document
Accept-Encoding: gzip, deflate, br
Accept-Language: fa,en-US;q=0.9,en;q=0.8

Response2:

Array
(
    [param1] => Na6dIwFMr4weGnRNTYewxrpffSiYDhl0un02dGYEMv1lThKoYuUEk1Q+Dmfk477k
)
 
Mohammad Hossein Sadeghi:

Both WebRequest calls will result in (from Fiddler):

GET https://mql-crypt.herokuapp.com/index.php?param1=Na6dIwFMr4weGnRNTYewxrpffSiYDhl0un02dGYEMv1lThKoYuUEk1Q+Dmfk477k HTTP/1.1
Cache-Control: no-cache
Connection: Keep-Alive
Proxy-Connection: Keep-Alive
Pragma: no-cache
Content-Type: text/plain
Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, */*
Accept-Charset: *,utf-8
Accept-Language: en
Host: mql-crypt.herokuapp.com
User-Agent: MetaTrader 4 Terminal/4.1260 (Windows NT 10.0; x64)

And web server will return the same response as:

Array
(
    [param1] => Na6dIwFMr4weGnRNTYewxrpffSiYDhl0un02dGYEMv1lThKoYuUEk1Q Dmfk477k
)

What is this server ?

Why is it answering "Na6dIwFMr4weGnRNTYewxrpffSiYDhl0un02dGYEMv1lThKoYuUEk1Q Dmfk477k" to a request with "Na6dIwFMr4weGnRNTYewxrpffSiYDhl0un02dGYEMv1lThKoYuUEk1Q+Dmfk477.

 

Confirmed Bug.


You can easily setup an echo-server in docker. 

docker-compose.yaml

my-http-listener:
    image: mendhak/http-https-echo
    ports:
        - "80:80"
        - "8443:443"

docker-compose -f docker-compose.yaml up

simple test script

void OnStart() {
   string cookie = NULL, headers;
   char   post[], result[];
   string url = "http://127.0.0.1?query=test+1%2B2";
   ResetLastError();
   int res = WebRequest("GET", url, cookie, NULL, 500, post, 0, result, headers);
   if(res == -1) {
      Print("Error in WebRequest. Error code  =", GetLastError());
      MessageBox("Add the address '" + url + "' to the list of allowed URLs on tab 'Expert Advisors'", "Error", MB_ICONINFORMATION);
   } else {
      if(res == 200) {
         PrintFormat("The file has been successfully downloaded, File size %d byte.", ArraySize(result));
         int filehandle = FileOpen("url.htm", FILE_WRITE | FILE_BIN);
         if(filehandle != INVALID_HANDLE) {
            FileWriteArray(filehandle, result, 0, ArraySize(result));
            FileClose(filehandle);
         } else
            Print("Error in FileOpen. Error code =", GetLastError());
      } else
         PrintFormat("Downloading '%s' failed, error code %d", url, res);
   }
}

Echo results: first using python then MT5

my-http-listener_1  | -----------------
my-http-listener_1  | { path: '/hello-world',
my-http-listener_1  |   headers:
my-http-listener_1  |    { host: 'localhost',
my-http-listener_1  |      'user-agent': 'python-requests/2.23.0',
my-http-listener_1  |      'accept-encoding': 'gzip, deflate',
my-http-listener_1  |      accept: '*/*',
my-http-listener_1  |      connection: 'keep-alive' },
my-http-listener_1  |   method: 'GET',
my-http-listener_1  |   body: '',
my-http-listener_1  |   cookies: undefined,
my-http-listener_1  |   fresh: false,
my-http-listener_1  |   hostname: 'localhost',
my-http-listener_1  |   ip: '::ffff:172.17.0.1',
my-http-listener_1  |   ips: [],
my-http-listener_1  |   protocol: 'http',
my-http-listener_1  |   query: { query: 'test 1+2' },
my-http-listener_1  |   subdomains: [],
my-http-listener_1  |   xhr: false,
my-http-listener_1  |   connection: { servername: undefined } }
my-http-listener_1  | "GET /hello-world?query=test+1%2B2 HTTP/1.1"
my-http-listener_1  | -----------------
my-http-listener_1  | { path: '/',
my-http-listener_1  |   headers:
my-http-listener_1  |    { host: '127.0.0.1',
my-http-listener_1  |      accept: '*/*',
my-http-listener_1  |      'accept-encoding': 'gzip, deflate',
my-http-listener_1  |      'accept-language': 'en',
my-http-listener_1  |      'user-agent': 'MetaTrader 5 Terminal/5.2450 (Windows NT 10.0; x64)',
my-http-listener_1  |      connection: 'keep-alive',
my-http-listener_1  |      'content-length': '0' },
my-http-listener_1  |   method: 'GET',
my-http-listener_1  |   body: '',
my-http-listener_1  |   cookies: undefined,
my-http-listener_1  |   fresh: false,
my-http-listener_1  |   hostname: '127.0.0.1',
my-http-listener_1  |   ip: '::ffff:172.17.0.1',
my-http-listener_1  |   ips: [],
my-http-listener_1  |   protocol: 'http',
my-http-listener_1  |   query: { query: 'test 1 2' },
my-http-listener_1  |   subdomains: [],
my-http-listener_1  |   xhr: false,
my-http-listener_1  |   connection: { servername: undefined } }
my-http-listener_1  | "GET /?query=test+1+2 HTTP/1.1"

 
nicholi shen:

Confirmed Bug.


You can easily setup an echo-server in docker. 

docker-compose.yaml

docker-compose -f docker-compose.yaml up

simple test script

Echo results: first using python then MT5

Why confirmed ?

It's confirmed that WebRequest() change the '%2B' to a '+', but why is that a bug ?

And subsidiary question why is the server converting the '+' to a space ?

 
Alain Verleyen:

Why confirmed ?

It's confirmed that WebRequest() change the '%2B' to a '+', but why is that a bug ?

And subsidiary question why is the server converting the '+' to a space ?

The url has to be encoded before sending a request to the server. It confirms that WebRequests is taking an already url-encoded string, parsing it, then incorrectly re-encoding it before sending it to the server. To answer your second question, the echo-server takes the url encoded query string and converts it back to utf-8 machine readable string when echoing it as JSON. Here's an example you can run in a python shell.

>>> from urllib import parse
>>> params = dict(query='Test 1+2')
>>> params
{'query': 'Test 1+2'}
>>> q = parse.urlencode(params)
>>> q
'query=Test+1%2B2'
>>> parse.parse_qsl(q)
[('query', 'Test 1+2')]

This is how it should work, and also how it's not being done in webrequests. 

Percent-encoding - Wikipedia
  • en.wikipedia.org
The characters allowed in a URI are either reserved or unreserved (or a percent character as part of a percent-encoding). Reserved characters are those characters that sometimes have special meaning. For example, forward slash characters are used to separate different parts of a URL (or more generally, a URI). Unreserved characters have no such...
 

I forgot to add... 

This is what WebRequests is doing to the string which is incorrect. 

>>> q = q.replace('%2B', '+')
>>> parse.parse_qsl(q)
[('query', 'Test 1 2')]
 

Furthermore... you can also see the exact string being received by the server from both requests test. 

python = "GET /?query=test+1%2B2 HTTP/1.1"
mql = "GET /?query=test+1+2 HTTP/1.1"
 
nicholi shen:

The url has to be encoded before sending a request to the server. It confirms that WebRequests is taking an already url-encoded string, parsing it, then incorrectly re-encoding it before sending it to the server. To answer your second question, the echo-server takes the url encoded query string and converts it back to utf-8 machine readable string when echoing it as JSON. Here's an example you can run in a python shell.

WebRequest is doing some decoding/encoding. We all agree on that. And it should at least be documented. But that was not my point.

The point could be summarized as : Why "it's how it should work" ? I found my answer :

Now in the query part, spaces may be encoded to either "+" (for backwards compatibility: do not try to search for it in the URI standard) or "%20" while the "+" character (as a result of this ambiguity) has to be escaped to "%2B".

I was searching in the standard, and of course didn't found it.

So it's a WebRequest "bug" because the HTTP servers don't follow the standards ( The standard says '+' should not be encoded). And we all know how Metaquotes is respectful about the standards.

 
Alain Verleyen:

 I found my answer :

I was searching in the standard, and of course didn't found it.

Webservers accept url-encoded strings. WebRequests' url parameter takes a string that's already url-encoded. WebRequests url param should not be mucking about with the encoded string. I'm not sure you are understanding even though everyone ITT is providing very simple MCVE. MQ is not respectful of the "standards" in these regards. MQ is taking a taking a properly encoded string and forwarding it to the webserver as a different (improperly encoded) string. That is a bug.  

 
Alain Verleyen:

What is this server ?

Why is it answering "Na6dIwFMr4weGnRNTYewxrpffSiYDhl0un02dGYEMv1lThKoYuUEk1Q Dmfk477k" to a request with "Na6dIwFMr4weGnRNTYewxrpffSiYDhl0un02dGYEMv1lThKoYuUEk1Q+Dmfk477.

The server is PHP, the most common server (78.6%) according to https://w3techs.com/technologies/overview/programming_language
Usage Statistics and Market Share of Server-side Programming Languages for Websites, May 2020
Usage Statistics and Market Share of Server-side Programming Languages for Websites, May 2020
  • w3techs.com
Technologies > Server-side Languages Usage statistics of server-side programming languages for websites This diagram shows the percentages of websites using various server-side programming languages. See technologies overview for explanations on the methodologies used in the surveys. Our reports are updated daily. How to read the diagram: PHP...
Reason: