WebRequest bug

Javi_VR 2019.08.29 19:44

Hello,

It seems there is a bug in the URL encoding of the function WebRequest, specifically with the character '+'.

For example, if you send the string "Test 1+2" the server should receive "Test+1%2B2" but it receives "Test+1+2", so all ' +' characters are converted to spaces.

This problem persists even if I encode the URL with my own function:

string EncodeURL(string str)
{
    char   array[];
    string result = "";

    const int len = StringToCharArray(str, array, 0, -1, CP_UTF8) - 1;

    for (int i = 0; i < len; i++)
    {
        const unsigned char c = array[i];

        if (((c >= '0') && (c <= '9')) ||
            ((c >= 'A') && (c <= 'Z')) ||
            ((c >= 'a') && (c <= 'z')) ||
            (c == '-') ||
            (c == '_') ||
            (c == '.') ||
            (c == '~'))
        {
            result += CharToString(c);
        }
        else if (c == ' ')
        {
            result += CharToString('+');
        }
        else
        {
            result += StringFormat("%%%02X", c);
        }
    }

    return result;
}

I'm using latest version: MetaTrader 5 build 2128.

Cheers,

Javi

Alain Verleyen 2019.08.29 21:19 #1

Javi_VR:

Hello,

It seems there is a bug in the URL encoding of the function WebRequest, specifically with the character '+'.

For example, if you send the string "Test 1+2" the server should receive "Test+1%2B2" but it receives "Test+1+2", so all ' +' characters are converted to spaces.

This problem persists even if I encode the URL with my own function:

I'm using latest version: MetaTrader 5 build 2128.

Cheers,

Javi

You posted a question about WebRequest, you could at least post the relevant code.

Javi_VR 2019.08.30 01:11 #2

Alain Verleyen:
You posted a question about WebRequest, you could at least post the relevant code.

Read my post again. I didn't post any question, I reported a bug in a function with more than enough information to solve it.

What "relevant code" do you want me to post to show that the URL encoding of WebRequest is incorrect? This? WebRequest("GET", "http://example.com/script.php?text=Test 1+2", ...)

is there difference between Error in WebRequest. Error WebRequest broken in build

Soewono Effendi 2019.08.30 01:57 #3

Alain Verleyen:
You posted a question about WebRequest, you could at least post the relevant code.

***

For example, if you send the string "Test 1+2" the server should receive "Test+1%2B2" but it receives "Test+1+2", so all ' +' characters are converted to spaces.

***

Do I understand you correctly, that you EXPECT WebRequest to do the URL encoding for you ?

It will send data verbatim, as is, whatever you throw in, that is sent to server. No url encoding conversion. Try it out.

WebRequest send always null MQL5 to Telegram How to code?

Javi_VR 2019.08.30 04:02 #4

Soewono Effendi:
***

For example, if you send the string "Test 1+2" the server should receive "Test+1%2B2" but it receives "Test+1+2", so all ' +' characters are converted to spaces.

***

Do I understand you correctly, that you EXPECT WebRequest to do the URL encoding for you ?

It will send data verbatim, as is, whatever you throw in, that is sent to server. No url encoding conversion. Try it out.

No, you're wrong, WebRequest encodes the URL but incorrectly (at least with the GET method, I didn't test POST).

You can capture the TCP/IP packets or, if you have a server, inspect the raw request URI in PHP to check it.

MQL5 to Telegram [Archive c 17.03.2008] Humour WebRequest Function, always return

Soewono Effendi 2019.08.30 04:19 #5

Javi_VR:

No, you're wrong, WebRequest encodes the URL but incorrectly (at least with the GET method, I didn't test POST).

You can capture the TCP/IP packets or, if you have a server, inspect the raw request URI in PHP to check it.

when you say so. :)

Alain Verleyen 2019.08.30 04:43 #6

Javi_VR:

Read my post again. I didn't post any question, I reported a bug in a function with more than enough information to solve it.

What "relevant code" do you want me to post to show that the URL encoding of WebRequest is incorrect? This? WebRequest("GET", "http://example.com/script.php?text=Test 1+2", ...)

At least, now we know you are using GET.

Javi_VR:

Hello,

It seems there is a bug in the URL encoding of the function WebRequest, specifically with the character '+'.

For example, if you send the string "Test 1+2" the server should receive " Test+1%2B2" but it receives "Test+1+2", so all ' +' characters are converted to spaces.

Why is that ? It could but it should not. The replacement of a space by '+' is not a standard.

That's not exact.

  string url="http://www.example.com/mql5.php?value=Test 1+2";
  int res=WebRequest("GET",url,req_headers,10000,post,result,result_headers);

Wireshark capture :

281 8.385354 192.168.0.142 93.184.216.34 HTTP 346 GET /mql5.php?value= Test%201+2 HTTP/1.1

This problem persists even if I encode the URL with my own function:

Using a '+' to encode a space doesn't work. But %20 should be fine, and so a '+' just means a '+', no need to encode it.

Javi_VR:

No, you're wrong, WebRequest encodes the URL but incorrectly (at least with the GET method, I didn't test POST).

You can capture the TCP/IP packets or, if you have a server, inspect the raw request URI in PHP to check it.

You are wrong. The encoding is correct, it just doesn't use or allow encoding a space with a '+'.

If you absolutely want to use a '+' for space and your own encoding, use a POST, the data are untouched.

Reference.

How can I code [WARNING CLOSED!] Any newbie URL encode implementation

Soewono Effendi 2019.08.30 05:03 #7

https://www.urlencoder.org/

try to encode 'Test 1+2' then you'll see what your issue is.

Compare with 'Test 1 2'

Javi_VR 2019.08.30 06:56 #8

Alain Verleyen:

At least, now we know you are using GET.

Why is that ? It could but it should not. The replacement of a space by '+' is not a standard.

That's not exact.

Wireshark capture :

281 8.385354 192.168.0.142 93.184.216.34 HTTP 346 GET /mql5.php?value= Test%201+2 HTTP/1.1

Using a '+' to encode a space doesn't work. But %20 should be fine, and so a '+' just means a '+', no need to encode it.

You are wrong. The encoding is correct, it just doesn't use or allow encoding a space with a '+'.

If you absolutely want to use a '+' for space and your own encoding, use a POST, the data are untouched.

Reference.

The problem is not if WebRequest encodes the spaces as %20 or '+', the problem is that the character '+' should be encoded as %2B, if not it'll be interpreted as a space. Test it in any server with the example you wrote.

The last reference (RFC1866) says the query of a GET must be encoded as "application/x-www-form-urlencoded" then any '+' is interpreted for the servers as a space. And WebRequest specifies this encoding in the header as content-type, check the documentation or the TCP/IP packets.

I've implemented my own version of WebRequest with the sockets functions so for me there is no problem at all if this bug is solved or not. But I think it will be a good idea to specify this in the documentation, or even better, add a version of WebRequest without any encoding to let you use your own encoding instead of saying "use a POST".

and again dll and WebRequest How can i send

Javi_VR 2019.08.30 06:59 #9

Soewono Effendi:

https://www.urlencoder.org/

try to encode 'Test 1+2' then you'll see what your issue is.

Compare with 'Test 1 2'

Exactly, as I've said WebRequest should encode the '+' characters as %2B.

Soewono Effendi 2019.08.30 09:12 #10

Javi_VR:

Exactly, as I've said WebRequest should encode the '+' characters as %2B.

Why/how do you come to the conclusion that WebRequest does encode ?

Where do you read/take that information ?

Or are you making your own assumption ?

1 2 3 4 5

New comment