WebRequest(): how to handle HTTP 303 response downloading shared files from Google Drive?

[Dmitri Diall]

Hello all,

Wondering if someone could point me in the right direction vis-a-vis an issue I'm facing with WebRequest() when attempting to download shared files from Google Drive, e.g. sample file linked below:

https://drive.google.com/uc?id=1-PtADmc85Tws_RcRF9nGJ0LkkBvv3DWO&export=download 

Attached is the script I'm using to test this -- pretty much a straightforward "GET" request via WebRequest() call to the URL above... The issue seems to be that drive.google.com responds with HTTP 303 (See Other) which redirects to a server in a different domain (in this case doc-04-c8-docs.googleusercontent.com). So, basically, my questions are:

1. Is there a simple and stable way to work around this issue on Google Drive, which does not require the user to keep adding an arbitrary number of allowed URLs to the list under MT5 Options | Expert Advisors?
2. If not possible with Google Drive specifically, what other file hosting options I could look into that do not require complex HTTP request/response handling?

Here's a snapshot from the debugger after this piece of code:

int timeout = 5000;
string web_url = "https://drive.google.com/uc?id=1-PtADmc85Tws_RcRF9nGJ0LkkBvv3DWO&export=download";
string result_headers, headers = "";
char result_data[], post_data[];
int response = WebRequest("GET", web_url, headers, timeout, post_data, result_data, result_headers);

Here are the full contents of the `results_headers` string for reference just in case:

"Content-Type: application/binary
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Pragma: no-cache
Expires: Mon, 01 Jan 1990 00:00:00 GMT
Date: Wed, 23 Nov 2022 19:12:22 GMT
Location: https://doc-04-c8-docs.googleusercontent.com/docs/securesc/ha0ro937gcuc7l7deffksulhg5h7mbp1/29objud691gdskn9gh75bkb4h4s97lvk/1669230675000/14749911301360439218/*/1-PtADmc85Tws_RcRF9nGJ0LkkBvv3DWO?e=download&uuid=6e0efd01-78de-427a-9184-2b1b27cc4c88
Strict-Transport-Security: max-age=31536000
Content-Security-Policy: script-src 'report-sample' 'nonce-rqVzwkg7pZW9GfsVIwDGFA' 'unsafe-inline';object-src 'none';base-uri 'self';report-uri /_/DriveUntrustedContentHttp/cspreport;worker-src 'self'
Content-Security-Policy: require-trusted-types-for 'script';report-uri /_/DriveUntrustedContentHttp/cspreport
Accept-CH: Sec-CH-UA-Arch, Sec-CH-UA-Bitness, Sec-CH-UA-Full-Version, Sec-CH-UA-Full-Version-List, Sec-CH-UA-Model, Sec-CH-UA-WoW64, Sec-CH-UA-Platform, Sec-CH-UA-Platform-Version
Permissions-Policy: ch-ua-arch=*, ch-ua-bitness=*, ch-ua-full-version=*, ch-ua-full-version-list=*, ch-ua-model=*, ch-ua-wow64=*, ch-ua-platform=*, ch-ua-platform-version=*
Cross-Origin-Opener-Policy: same-origin
Server: ESF
Content-Length: 0
X-XSS-Protection: 0
X-Frame-Options: SAMEORIGIN
X-Content-Type-Options: nosniff
Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
" (length: 1494)

Here's a snapshot from wget fetching the same URL -- ideally, I'd like if WebRequest() could handle things in the same manner 😇

$ wget 'https://drive.google.com/uc?id=1-PtADmc85Tws_RcRF9nGJ0LkkBvv3DWO&export=download'
--2022-11-23 19:08:49--  https://drive.google.com/uc?id=1-PtADmc85Tws_RcRF9nGJ0LkkBvv3DWO&export=download
Resolving drive.google.com (drive.google.com)... 142.250.200.110, 2a00:1450:4003:808::200e
Connecting to drive.google.com (drive.google.com)|142.250.200.110|:443... connected.
HTTP request sent, awaiting response... 303 See Other
Location: https://doc-04-c8-docs.googleusercontent.com/docs/securesc/ha0ro937gcuc7l7deffksulhg5h7mbp1/l9ah82ffg2h3f91degchsc1c45bqc1lo/1669230525000/14749911301360439218/*/1-PtADmc85Tws_RcRF9nGJ0LkkBvv3DWO?e=download&uuid=e94e63c7-e275-4899-b9ff-e2282738ebfd [following]
Warning: wildcards not supported in HTTP.
--2022-11-23 19:08:49--  https://doc-04-c8-docs.googleusercontent.com/docs/securesc/ha0ro937gcuc7l7deffksulhg5h7mbp1/l9ah82ffg2h3f91degchsc1c45bqc1lo/1669230525000/14749911301360439218/*/1-PtADmc85Tws_RcRF9nGJ0LkkBvv3DWO?e=download&uuid=e94e63c7-e275-4899-b9ff-e2282738ebfd
Resolving doc-04-c8-docs.googleusercontent.com (doc-04-c8-docs.googleusercontent.com)... 142.250.201.65, 2a00:1450:4003:811::2001
Connecting to doc-04-c8-docs.googleusercontent.com (doc-04-c8-docs.googleusercontent.com)|142.250.201.65|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 54 [text/plain]
Saving to: ‘uc?id=1-PtADmc85Tws_RcRF9nGJ0LkkBvv3DWO&export=download’

uc?id=1-PtADmc85Tws_RcRF9nGJ0LkkBv 100%[================================================================>]      54  --.-KB/s    in 0s

2022-11-23 19:08:49 (1.21 MB/s) - ‘uc?id=1-PtADmc85Tws_RcRF9nGJ0LkkBvv3DWO&export=download’ saved [54/54]

[Fernando Carreiro]

1. Unfortunately, not! Only by using DLL calls to Windows API instead. or using some intermediate web-application.
   
2. I guess you will have to experiment and see! I've never considered doing such things with WebRequest().

[Carl Schreiber]

Have you allowed the URL in the Options (Ctrl+O):

[Fernando Carreiro]

Carl Schreiber #: Have you allowed the URL in the Options (Ctrl+O):

Yes he has and that is exactly what he is referring too. Google drive generates different server address depending on situation, thus requiring manual updating of that very list of accepted URLs.

[Dmitri Diall]

Fernando Carreiro #:

1. Unfortunately, not! Only by using DLL calls to Windows API instead. or using some intermediate web-application.
   
2. I guess you will have to experiment and see! I've never considered doing such things with WebRequest().

Hi Fernando -- thanks for the quick reply, as usual, albeit ultimately disappointing :-)

[Fernando Carreiro]

Dmitri Diall: Wondering if someone could point me in the right direction vis-a-vis an issue I'm facing with WebRequest() when attempting to download shared files from Google Drive, e.g. sample file linked below:

Just remembered a possibility — have you tried just setting "https://googleusercontent.com" in the URL list?

Maybe MetaTrader will automatically consider the "*.googleusercontent.com" sub-domains as valid.

I've never tried it before but maybe it works.

[Fernando Carreiro]

I just tried adding "https://*.googleusercontent.com" with a wildcard and it accepted it, but don't know if it actually works.

[Carl Schreiber]

What about tiny Url: https://tinyurl.com/app/

[Fernando Carreiro]

Carl Schreiber #: What about tiny Url: https://tinyurl.com/app/

That will not work. It is not a fixed URL. We are talking about Google Drive here that applies load balancing by directing to different servers.

EDIT: Besides, "tinyurl" just redirects the URL to the correct one. It will suffer the same issue.

[Dmitri Diall]

Fernando Carreiro #:

I just tried adding "https://*.googleusercontent.com" with a wildcard and it accepted it, but don't know if it actually works.

Good call on the wildcard... Just tried it, but doesn't seem to work -- it appears that the full domain name needs to be added to the allowed URL list.

int response   = WebRequest("GET", web_url, headers, timeout, post_data, result_data, result_headers);
int error_code = GetLastError();

The code above results in response == -1 and error_code == 4014 which, in MQL5, means "function is not allowed for call" (ERR_FUNCTION_NOT_ALLOWED), i.e. URL not found in allowed list.

Runtime Errors - Codes of Errors and Warnings - Constants, Enumerations and Structures - MQL5 Reference - Reference on algorithmic/automated trading language for MetaTrader 5

[Fernando Carreiro]

Dmitri Diall #: Good call on the wildcard... Just tried it, but doesn't seem to work -- it appears that the full domain name needs to be added to the allowed URL list. The code above results in response == -1 and error_code == 4014 which, in MQL5, means "function is not allowed for call" (ERR_FUNCTION_NOT_ALLOWED), i.e. URL not found in allowed list. Runtime Errors - Codes of Errors and Warnings - Constants, Enumerations and Structures - MQL5 Reference - Reference on algorithmic/automated trading language for MetaTrader 5

Did you try my suggestion of just using the parent domain name?

Forum on trading, automated trading systems and testing trading strategies

WebRequest(): how to handle HTTP 303 response downloading shared files from Google Drive?

Fernando Carreiro, 2022.11.23 22:24

Just remembered a possibility — have you tried just setting "https://googleusercontent.com" in the URL list?

Maybe MetaTrader will automatically consider the "*.googleusercontent.com" sub-domains as valid.

I've never tried it before but maybe it works.