WebRequest bug - page 2

New comment
 
Soewono Effendi:
Why/how do you come to the conclusion that WebRequest does encode ?
Where do you read/take that information ?
Or are you making your own assumption ?

You are totally right. It doesn't encode at all, it "checks".

I would even say it decode :

  string url="http://www.example.com/encoded.php?value%3Dtest%20%281%2B2%29%2F3%3D1";           // EncodeURLFixed("value=test (1+2)/3=1")
  int res=WebRequest("GET",url,req_headers,10000,post,result,result_headers);

Wireshark capture :

22513    370.343579    192.168.0.142    93.184.216.34    HTTP    306    GET /webrequest.php?value=test%20(1+2)/3=1 HTTP/1.1

By the way I tried the same request with Postman, and it also "decode" but differently.

50601    900.011366    192.168.0.142    93.184.216.34    HTTP    393    GET /postman.php?value=test%20%281+2%29/3=1 HTTP/1.1

 
Javi_VR:


The problem is not if WebRequest encodes the spaces as %20 or '+', the problem is that the character '+' should be encoded as %2B, if not it'll be interpreted as a space. Test it in any server with the example you wrote.

The last reference (RFC1866) says the query of a GET must be encoded as "application/x-www-form-urlencoded" then any '+' is interpreted for the servers as a space. And WebRequest specifies this encoding in the header as content-type, check the documentation or the TCP/IP packets.

I've implemented my own version of WebRequest with the sockets functions so for me there is no problem at all if this bug is solved or not. But I think it will be a good idea to specify this in the documentation, or even better, add a version of WebRequest without any encoding to let you use your own encoding instead of saying "use a POST".

There is no bug, there is a coherent implementation.

I do agree the documentation should be improved and clearer.

 
Alain Verleyen:

You are totally right. It doesn't encode at all, it "checks".

I would even say it decode :

Wireshark capture :

22513    370.343579    192.168.0.142    93.184.216.34    HTTP    306    GET /webrequest.php?value=test%20(1+2)/3=1 HTTP/1.1

By the way I tried the same request with Postman, and it also "decode" but differently.

50601    900.011366    192.168.0.142    93.184.216.34    HTTP    393    GET /postman.php?value=test%20%281+2%29/3=1 HTTP/1.1


Alain Verleyen:

You are totally right. It doesn't encode at all, it "checks".


Again I'm sorry but, no, you're wrong. I think it would be much better if WebRequest does not make the encoding but that's exactly what's doing. It wouldn't be a problem if the encoding was correct but it's not.

I really don't know why you deny this (?), you made your own test capturing the TCP packet and you can see clearly that WebRequest encoded the space char in " ...value=Test 1+2" as %20: "...value=Test%201+2". 

It is encoding the URL string, here you have you own test: 


Alain Verleyen:

string url="http://www.example.com/mql5.php?value=Test 1+2";
int res=WebRequest("GET",url,req_headers,10000,post,result,result_headers);

Wireshark capture :

281    8.385354    192.168.0.142    93.184.216.34    HTTP    346    GET /mql5.php?value= Test%201+2 HTTP/1.1 


Please, could you tell who converted the space char in %20?


You can make all the tests you want capturing the packets or inspecting the raw URI requests in a server script and you'll see it over and over again.


If someday you need to send a complex GET query to a server script you'll understand where is the problem.

Well, a quick patch to solve this would be to encode the GET query parameters as base64 and decode them in the server script.


Alain Verleyen:

I would even say it decode :


Yes, that's true, WebRequest is also doing a decoding; it first decodes any posible encoding of the URL string and then applies it's own encoding.


Anyway, there's no point to keep this discussion. I reported the bug, if you both prefer to think that WebRequest is sending the GET queries as raw then perfect, no problem, go ahead.


 
Javi_VR:


Alain Verleyen:

You are totally right. It doesn't encode at all, it "checks".


Again I'm sorry but, no, you're wrong. I think it would be much better if WebRequest does not make the encoding but that's exactly what's doing. It wouldn't be a problem if the encoding was correct but it's not.

I really don't know why you deny this (?), you made your own test capturing the TCP packet and you can see clearly that WebRequest encoded the space char in " ...value=Test 1+2" as %20: "...value=Test%201+2". 

It is encoding the URL string, here you have you own test: 


Alain Verleyen:

Wireshark capture :

281    8.385354    192.168.0.142    93.184.216.34    HTTP    346    GET /mql5.php?value= Test%201+2 HTTP/1.1 


Please, could you tell who converted the space char in %20?


You can make all the tests you want capturing the packets or inspecting the raw URI requests in a server script and you'll see it over and over again.


If someday you need to send a complex GET query to a server script you'll understand where is the problem.

Well, a quick patch to solve this would be to encode the GET query parameters as base64 and decode them in the server script.


Alain Verleyen:

I would even say it decode :


Yes, that's true, WebRequest is also doing a decoding; it first decodes any posible encoding of the URL string and then applies it's own encoding.


Anyway, there's no point to keep this discussion. I reported the bug, if you both prefer to think that WebRequest is sending the GET queries as raw then perfect, no problem, go ahead.


You have difficulties to read and understand what other people write..

WebRequest is doing "something" while using a GET, I never said it send queries as raw. I agreed with you this should be documented, but that's NOT an "URL encoding" as I have demonstrated. That's not a bug and Metaquotes will never recognize it as a bug. I do have some experience dealing with them about bugs.

In general, I don't like your unpleasant and arrogant tone. Thanks to port this issue to our attention, anyway.

 

I'm facing the same issue, PHP and many others, decode plus sign (+) to a space,

Decodes any %## encoding in the given string. Plus symbols ('+') are decoded to a space character.

the behavior of WebRequest should be changed, or at least an option should be provided whether to encode the given URL or the coder should take care about it.

I am encoding my input to base64, and this encoding has ('+') sign, after facing the issue, I replaced ('+') with ('%2B'), but still no success since it is again changed to ('+') before sending, I check the network traffic by Fiddler, and I'm sure that the URL is changed.

 
Javi_VR:

Anyway, there's no point to keep this discussion. I reported the bug, if you both prefer to think that WebRequest is sending the GET queries as raw then perfect, no problem, go ahead.

Where did you report it, so I can up vote.

 
Mohammad Hossein Sadeghi:

Where did you report it, so I can up vote.

He reported it here. Good luck trying to port it to Metaquotes attention.

I am still not convinced there is any bug.

 
Alain Verleyen:

He reported it here. Good luck trying to port it to Metaquotes attention.

I am still not convinced there is any bug.

I agree it is not a bug, but it should be considered.

 
Mohammad Hossein Sadeghi:

I agree it is not a bug, but it should be considered.

I never encountered a problem, I also sometimes used Base64 data. Are you able to provide a use case with all relevant information to demonstrate the problem ?

It's always possible to workaround a problem using a Socket where you should have more control.

 
Alain Verleyen:

I never encountered a problem, I also sometimes used Base64 data. Are you able to provide a use case with all relevant information to demonstrate the problem ?

It's always possible to workaround a problem using a Socket where you should have more control.

void OnStart()
{
   const string encrypted1="Na6dIwFMr4weGnRNTYewxrpffSiYDhl0un02dGYEMv1lThKoYuUEk1Q+Dmfk477k";
   const string encrypted2="Na6dIwFMr4weGnRNTYewxrpffSiYDhl0un02dGYEMv1lThKoYuUEk1Q%2BDmfk477k";

   //--- send data
   char data[];
   char res_data[];
   string res_headers=NULL;
   int res1=WebRequest("GET","https://mql-crypt.herokuapp.com/index.php?param1="+encrypted1,"Content-Type: text/plain\r\n", 5000,data,res_data,res_headers);
   if(res1==200)
   {
      Alert(CharArrayToString(res_data));
   }

   int res2=WebRequest("GET","https://mql-crypt.herokuapp.com/index.php?param1="+encrypted2,"Content-Type: text/plain\r\n", 5000,data,res_data,res_headers);
   if(res2==200)
   {
      Alert(CharArrayToString(res_data));
   }
}

Both WebRequest calls will result in (from Fiddler):

GET https://mql-crypt.herokuapp.com/index.php?param1=Na6dIwFMr4weGnRNTYewxrpffSiYDhl0un02dGYEMv1lThKoYuUEk1Q+Dmfk477k HTTP/1.1
Cache-Control: no-cache
Connection: Keep-Alive
Proxy-Connection: Keep-Alive
Pragma: no-cache
Content-Type: text/plain
Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, */*
Accept-Charset: *,utf-8
Accept-Language: en
Host: mql-crypt.herokuapp.com
User-Agent: MetaTrader 4 Terminal/4.1260 (Windows NT 10.0; x64)

And web server will return the same response as:

Array
(
    [param1] => Na6dIwFMr4weGnRNTYewxrpffSiYDhl0un02dGYEMv1lThKoYuUEk1Q Dmfk477k
)
12345
New comment
Reason: