WebRequest bug - page 3

To add comments, please log in or register
Mohammad Hossein Sadeghi
3275
Mohammad Hossein Sadeghi  

In Chrome:

Request1:

GET https://mql-crypt.herokuapp.com/index.php?param1=Na6dIwFMr4weGnRNTYewxrpffSiYDhl0un02dGYEMv1lThKoYuUEk1Q+Dmfk477k HTTP/1.1
Host: mql-crypt.herokuapp.com
Connection: keep-alive
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.138 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
Sec-Fetch-Site: none
Sec-Fetch-Mode: navigate
Sec-Fetch-User: ?1
Sec-Fetch-Dest: document
Accept-Encoding: gzip, deflate, br
Accept-Language: fa,en-US;q=0.9,en;q=0.8

Response1:

Array
(
    [param1] => Na6dIwFMr4weGnRNTYewxrpffSiYDhl0un02dGYEMv1lThKoYuUEk1Q Dmfk477k
)

Request2:

GET https://mql-crypt.herokuapp.com/index.php?param1=Na6dIwFMr4weGnRNTYewxrpffSiYDhl0un02dGYEMv1lThKoYuUEk1Q%2BDmfk477k HTTP/1.1
Host: mql-crypt.herokuapp.com
Connection: keep-alive
Cache-Control: max-age=0
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.138 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
Sec-Fetch-Site: none
Sec-Fetch-Mode: navigate
Sec-Fetch-User: ?1
Sec-Fetch-Dest: document
Accept-Encoding: gzip, deflate, br
Accept-Language: fa,en-US;q=0.9,en;q=0.8

Response2:

Array
(
    [param1] => Na6dIwFMr4weGnRNTYewxrpffSiYDhl0un02dGYEMv1lThKoYuUEk1Q+Dmfk477k
)
Alain Verleyen
40436
Alain Verleyen  
Mohammad Hossein Sadeghi:

Both WebRequest calls will result in (from Fiddler):

GET https://mql-crypt.herokuapp.com/index.php?param1=Na6dIwFMr4weGnRNTYewxrpffSiYDhl0un02dGYEMv1lThKoYuUEk1Q+Dmfk477k HTTP/1.1
Cache-Control: no-cache
Connection: Keep-Alive
Proxy-Connection: Keep-Alive
Pragma: no-cache
Content-Type: text/plain
Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, */*
Accept-Charset: *,utf-8
Accept-Language: en
Host: mql-crypt.herokuapp.com
User-Agent: MetaTrader 4 Terminal/4.1260 (Windows NT 10.0; x64)

And web server will return the same response as:

Array
(
    [param1] => Na6dIwFMr4weGnRNTYewxrpffSiYDhl0un02dGYEMv1lThKoYuUEk1Q Dmfk477k
)

What is this server ?

Why is it answering "Na6dIwFMr4weGnRNTYewxrpffSiYDhl0un02dGYEMv1lThKoYuUEk1Q Dmfk477k" to a request with "Na6dIwFMr4weGnRNTYewxrpffSiYDhl0un02dGYEMv1lThKoYuUEk1Q+Dmfk477.

nicholish en
2642
nicholish en  

Confirmed Bug.


You can easily setup an echo-server in docker. 

docker-compose.yaml

my-http-listener:
    image: mendhak/http-https-echo
    ports:
        - "80:80"
        - "8443:443"

docker-compose -f docker-compose.yaml up

simple test script

void OnStart() {
   string cookie = NULL, headers;
   char   post[], result[];
   string url = "http://127.0.0.1?query=test+1%2B2";
   ResetLastError();
   int res = WebRequest("GET", url, cookie, NULL, 500, post, 0, result, headers);
   if(res == -1) {
      Print("Error in WebRequest. Error code  =", GetLastError());
      MessageBox("Add the address '" + url + "' to the list of allowed URLs on tab 'Expert Advisors'", "Error", MB_ICONINFORMATION);
   } else {
      if(res == 200) {
         PrintFormat("The file has been successfully downloaded, File size %d byte.", ArraySize(result));
         int filehandle = FileOpen("url.htm", FILE_WRITE | FILE_BIN);
         if(filehandle != INVALID_HANDLE) {
            FileWriteArray(filehandle, result, 0, ArraySize(result));
            FileClose(filehandle);
         } else
            Print("Error in FileOpen. Error code =", GetLastError());
      } else
         PrintFormat("Downloading '%s' failed, error code %d", url, res);
   }
}

Echo results: first using python then MT5

my-http-listener_1  | -----------------
my-http-listener_1  | { path: '/hello-world',
my-http-listener_1  |   headers:
my-http-listener_1  |    { host: 'localhost',
my-http-listener_1  |      'user-agent': 'python-requests/2.23.0',
my-http-listener_1  |      'accept-encoding': 'gzip, deflate',
my-http-listener_1  |      accept: '*/*',
my-http-listener_1  |      connection: 'keep-alive' },
my-http-listener_1  |   method: 'GET',
my-http-listener_1  |   body: '',
my-http-listener_1  |   cookies: undefined,
my-http-listener_1  |   fresh: false,
my-http-listener_1  |   hostname: 'localhost',
my-http-listener_1  |   ip: '::ffff:172.17.0.1',
my-http-listener_1  |   ips: [],
my-http-listener_1  |   protocol: 'http',
my-http-listener_1  |   query: { query: 'test 1+2' },
my-http-listener_1  |   subdomains: [],
my-http-listener_1  |   xhr: false,
my-http-listener_1  |   connection: { servername: undefined } }
my-http-listener_1  | "GET /hello-world?query=test+1%2B2 HTTP/1.1"
my-http-listener_1  | -----------------
my-http-listener_1  | { path: '/',
my-http-listener_1  |   headers:
my-http-listener_1  |    { host: '127.0.0.1',
my-http-listener_1  |      accept: '*/*',
my-http-listener_1  |      'accept-encoding': 'gzip, deflate',
my-http-listener_1  |      'accept-language': 'en',
my-http-listener_1  |      'user-agent': 'MetaTrader 5 Terminal/5.2450 (Windows NT 10.0; x64)',
my-http-listener_1  |      connection: 'keep-alive',
my-http-listener_1  |      'content-length': '0' },
my-http-listener_1  |   method: 'GET',
my-http-listener_1  |   body: '',
my-http-listener_1  |   cookies: undefined,
my-http-listener_1  |   fresh: false,
my-http-listener_1  |   hostname: '127.0.0.1',
my-http-listener_1  |   ip: '::ffff:172.17.0.1',
my-http-listener_1  |   ips: [],
my-http-listener_1  |   protocol: 'http',
my-http-listener_1  |   query: { query: 'test 1 2' },
my-http-listener_1  |   subdomains: [],
my-http-listener_1  |   xhr: false,
my-http-listener_1  |   connection: { servername: undefined } }
my-http-listener_1  | "GET /?query=test+1+2 HTTP/1.1"

Alain Verleyen
40436
Alain Verleyen  
nicholi shen:

Confirmed Bug.


You can easily setup an echo-server in docker. 

docker-compose.yaml

docker-compose -f docker-compose.yaml up

simple test script

Echo results: first using python then MT5

Why confirmed ?

It's confirmed that WebRequest() change the '%2B' to a '+', but why is that a bug ?

And subsidiary question why is the server converting the '+' to a space ?

nicholish en
2642
nicholish en  
Alain Verleyen:

Why confirmed ?

It's confirmed that WebRequest() change the '%2B' to a '+', but why is that a bug ?

And subsidiary question why is the server converting the '+' to a space ?

The url has to be encoded before sending a request to the server. It confirms that WebRequests is taking an already url-encoded string, parsing it, then incorrectly re-encoding it before sending it to the server. To answer your second question, the echo-server takes the url encoded query string and converts it back to utf-8 machine readable string when echoing it as JSON. Here's an example you can run in a python shell.

>>> from urllib import parse
>>> params = dict(query='Test 1+2')
>>> params
{'query': 'Test 1+2'}
>>> q = parse.urlencode(params)
>>> q
'query=Test+1%2B2'
>>> parse.parse_qsl(q)
[('query', 'Test 1+2')]

This is how it should work, and also how it's not being done in webrequests. 

Percent-encoding - Wikipedia
  • en.wikipedia.org
The characters allowed in a URI are either reserved or unreserved (or a percent character as part of a percent-encoding). Reserved characters are those characters that sometimes have special meaning. For example, forward slash characters are used to separate different parts of a URL (or more generally, a URI). Unreserved characters have no such...
nicholish en
2642
nicholish en  

I forgot to add... 

This is what WebRequests is doing to the string which is incorrect. 

>>> q = q.replace('%2B', '+')
>>> parse.parse_qsl(q)
[('query', 'Test 1 2')]
nicholish en
2642
nicholish en  

Furthermore... you can also see the exact string being received by the server from both requests test. 

python = "GET /?query=test+1%2B2 HTTP/1.1"
mql = "GET /?query=test+1+2 HTTP/1.1"
Alain Verleyen
40436
Alain Verleyen  
nicholi shen:

The url has to be encoded before sending a request to the server. It confirms that WebRequests is taking an already url-encoded string, parsing it, then incorrectly re-encoding it before sending it to the server. To answer your second question, the echo-server takes the url encoded query string and converts it back to utf-8 machine readable string when echoing it as JSON. Here's an example you can run in a python shell.

WebRequest is doing some decoding/encoding. We all agree on that. And it should at least be documented. But that was not my point.

The point could be summarized as : Why "it's how it should work" ? I found my answer :

Now in the query part, spaces may be encoded to either "+" (for backwards compatibility: do not try to search for it in the URI standard) or "%20" while the "+" character (as a result of this ambiguity) has to be escaped to "%2B".

I was searching in the standard, and of course didn't found it.

So it's a WebRequest "bug" because the HTTP servers don't follow the standards ( The standard says '+' should not be encoded). And we all know how Metaquotes is respectful about the standards.

nicholish en
2642
nicholish en  
Alain Verleyen:

 I found my answer :

I was searching in the standard, and of course didn't found it.

Webservers accept url-encoded strings. WebRequests' url parameter takes a string that's already url-encoded. WebRequests url param should not be mucking about with the encoded string. I'm not sure you are understanding even though everyone ITT is providing very simple MCVE. MQ is not respectful of the "standards" in these regards. MQ is taking a taking a properly encoded string and forwarding it to the webserver as a different (improperly encoded) string. That is a bug.  

Mohammad Hossein Sadeghi
3275
Mohammad Hossein Sadeghi  
Alain Verleyen:

What is this server ?

Why is it answering "Na6dIwFMr4weGnRNTYewxrpffSiYDhl0un02dGYEMv1lThKoYuUEk1Q Dmfk477k" to a request with "Na6dIwFMr4weGnRNTYewxrpffSiYDhl0un02dGYEMv1lThKoYuUEk1Q+Dmfk477.

The server is PHP, the most common server (78.6%) according to https://w3techs.com/technologies/overview/programming_language
Usage Statistics and Market Share of Server-side Programming Languages for Websites, May 2020
Usage Statistics and Market Share of Server-side Programming Languages for Websites, May 2020
  • w3techs.com
Technologies > Server-side Languages Usage statistics of server-side programming languages for websites This diagram shows the percentages of websites using various server-side programming languages. See technologies overview for explanations on the methodologies used in the surveys. Our reports are updated daily. How to read the diagram: PHP...
To add comments, please log in or register