WebRequest bug

To add comments, please log in or register
Javi_VR
54
Javi_VR  

Hello, 

It seems there is a bug in the URL encoding of the function WebRequest, specifically with the character '+'.

For example, if you send the string "Test 1+2" the server should receive "Test+1%2B2" but it receives "Test+1+2", so all ' +' characters are converted to spaces.

This problem persists even if I encode the URL with my own function:

string EncodeURL(string str)
{
    char   array[];
    string result = "";

    const int len = StringToCharArray(str, array, 0, -1, CP_UTF8) - 1;

    for (int i = 0; i < len; i++)
    {
        const unsigned char c = array[i];

        if (((c >= '0') && (c <= '9')) ||
            ((c >= 'A') && (c <= 'Z')) ||
            ((c >= 'a') && (c <= 'z')) ||
            (c == '-') ||
            (c == '_') ||
            (c == '.') ||
            (c == '~'))
        {
            result += CharToString(c);
        }
        else if (c == ' ')
        {
            result += CharToString('+');
        }
        else
        {
            result += StringFormat("%%%02X", c);
        }
    }

    return result;
}

I'm using latest version: MetaTrader 5 build 2128.

Cheers,

Javi


Alain Verleyen
40323
Alain Verleyen  
Javi_VR:

Hello, 

It seems there is a bug in the URL encoding of the function WebRequest, specifically with the character '+'.

For example, if you send the string "Test 1+2" the server should receive "Test+1%2B2" but it receives "Test+1+2", so all ' +' characters are converted to spaces.

This problem persists even if I encode the URL with my own function:

I'm using latest version: MetaTrader 5 build 2128.

Cheers,

Javi


You posted a question about WebRequest, you could at least post the relevant code.
Javi_VR
54
Javi_VR  
Alain Verleyen:
You posted a question about WebRequest, you could at least post the relevant code.

Read my post again. I didn't post any question, I reported a bug in a function with more than enough information to solve it.

What "relevant code" do you want me to post to show that the URL encoding of WebRequest is incorrect? This? WebRequest("GET", "http://example.com/script.php?text=Test 1+2", ...)

Soewono Effendi
2692
Soewono Effendi  
Alain Verleyen:
You posted a question about WebRequest, you could at least post the relevant code.
***
For example, if you send the string "Test 1+2" the server should receive "Test+1%2B2" but it receives "Test+1+2", so all ' +' characters are converted to spaces.
***

Do I understand you correctly, that you EXPECT WebRequest to do the URL encoding for you ?

It will send data verbatim, as is, whatever you throw in, that is sent to server. No url encoding conversion. Try it out.
Javi_VR
54
Javi_VR  
Soewono Effendi:
***
For example, if you send the string "Test 1+2" the server should receive "Test+1%2B2" but it receives "Test+1+2", so all ' +' characters are converted to spaces.
***

Do I understand you correctly, that you EXPECT WebRequest to do the URL encoding for you ?

It will send data verbatim, as is, whatever you throw in, that is sent to server. No url encoding conversion. Try it out.

No, you're wrong, WebRequest encodes the URL but incorrectly (at least with the GET method, I didn't test POST).

You can capture the TCP/IP packets or, if you have a server, inspect the raw request URI in PHP to check it.


Soewono Effendi
2692
Soewono Effendi  
Javi_VR:

No, you're wrong, WebRequest encodes the URL but incorrectly (at least with the GET method, I didn't test POST).

You can capture the TCP/IP packets or, if you have a server, inspect the raw request URI in PHP to check it.


when you say so. :)

Alain Verleyen
40323
Alain Verleyen  
Javi_VR:

Read my post again. I didn't post any question, I reported a bug in a function with more than enough information to solve it.

What "relevant code" do you want me to post to show that the URL encoding of WebRequest is incorrect? This? WebRequest("GET", "http://example.com/script.php?text=Test 1+2", ...)

At least, now we know you are using GET.

Javi_VR:

Hello, 

It seems there is a bug in the URL encoding of the function WebRequest, specifically with the character '+'.

For example, if you send the string "Test 1+2" the server should receive " Test+1%2B2" but it receives "Test+1+2", so all ' +' characters are converted to spaces.

Why is that ? It could but it should not. The replacement of a space by '+' is not a standard.

That's not exact.

  string url="http://www.example.com/mql5.php?value=Test 1+2";
  int res=WebRequest("GET",url,req_headers,10000,post,result,result_headers);

Wireshark capture :

281    8.385354    192.168.0.142    93.184.216.34    HTTP    346    GET /mql5.php?value= Test%201+2 HTTP/1.1

This problem persists even if I encode the URL with my own function:

Using a '+' to encode a space doesn't work. But %20 should be fine, and so a '+' just means a '+', no need to encode it.

Javi_VR:

No, you're wrong, WebRequest encodes the URL but incorrectly (at least with the GET method, I didn't test POST).

You can capture the TCP/IP packets or, if you have a server, inspect the raw request URI in PHP to check it.


You are wrong. The encoding is correct, it just doesn't use or allow encoding a space with a '+'.

If you absolutely want to use a '+' for space and your own encoding, use a POST, the data are untouched.

Reference.

Soewono Effendi
2692
Soewono Effendi  



try to encode 'Test 1+2' then you'll see what your issue is. 

Compare  with 'Test 1 2' 


Javi_VR
54
Javi_VR  
Alain Verleyen:

At least, now we know you are using GET.

Why is that ? It could but it should not. The replacement of a space by '+' is not a standard.

That's not exact.

Wireshark capture :

281    8.385354    192.168.0.142    93.184.216.34    HTTP    346    GET /mql5.php?value= Test%201+2 HTTP/1.1

Using a '+' to encode a space doesn't work. But %20 should be fine, and so a '+' just means a '+', no need to encode it.

You are wrong. The encoding is correct, it just doesn't use or allow encoding a space with a '+'.

If you absolutely want to use a '+' for space and your own encoding, use a POST, the data are untouched.

Reference.


The problem is not if WebRequest encodes the spaces as %20 or '+', the problem is that the character '+' should be encoded as %2B, if not it'll be interpreted as a space. Test it in any server with the example you wrote.

The last reference (RFC1866) says the query of a GET must be encoded as "application/x-www-form-urlencoded" then any '+' is interpreted for the servers as a space. And WebRequest specifies this encoding in the header as content-type, check the documentation or the TCP/IP packets.

I've implemented my own version of WebRequest with the sockets functions so for me there is no problem at all if this bug is solved or not. But I think it will be a good idea to specify this in the documentation, or even better, add a version of WebRequest without any encoding to let you use your own encoding instead of saying "use a POST".

Javi_VR
54
Javi_VR  
Soewono Effendi:
https://www.urlencoder.org/



try to encode 'Test 1+2' then you'll see what your issue is. 

Compare  with 'Test 1 2' 



Exactly, as I've said WebRequest should encode the '+' characters as %2B.

Soewono Effendi
2692
Soewono Effendi  
Javi_VR:


Exactly, as I've said WebRequest should encode the '+' characters as %2B.

Why/how do you come to the conclusion that WebRequest does encode ?
Where do you read/take that information ?
Or are you making your own assumption ?
To add comments, please log in or register