How to run a test code and find out if you are blocked by Cloudflare - Expert Advisors and Automated Trading

Lorentzos Roussos 2023.03.21 15:06 #41

I run this test code again today out of curiosity

It works on mt4 (with reasonable gaps in between requests)

It returns error 1001 on mt5 (the mt5 error is 5203 but the address is there , i'll check again with my glasses)

(webreqeusts is mt4 mqh)

Files:

test.mq4 3 kb

WebRequests.mqh 11 kb

WebRequest.mqh 10 kb

test.mq5 3 kb

WebRequest broken in build Questions from Beginners MQL5 WebRequest returns -1 with

Sergey Golubev 2023.03.21 15:12 #42

Ugochukwu Mobi #:

Thank you so much but may you please explain the format of data more?? because to access date it is only few codes I don't see where I will change.

The code above that is what I was using and it was working great now to change the format what did I need to change in that code?
here the file is downloaded

but when I open it there is useless information

@Sergey Golubev how to change the format?

It may be something like that (Metatrader 4):

//-----   
string economicCalendar()
{
   int      handle_htm, handle_csv, _GetLastError;
   string   calendarName = "", fileName = "", strHTM = "", strCSV = "";
   
   datetime StartWeek = GetWeekStart(gmtime);// + tz*3600);
   
   string StartDay, strStartWeek = TimeToStr(StartWeek);
      
   StartYear  = (string)TimeYear (StartWeek);
   if(TimeMonth(StartWeek) > 9) StartMonth = (string)TimeMonth(StartWeek); else StartMonth = "0" + (string)TimeMonth(StartWeek);
   if(TimeDay  (StartWeek) > 9) StartDay   = (string)TimeDay  (StartWeek); else StartDay   = "0" + (string)TimeDay  (StartWeek);
   
   string StartTime = StartYear + "." + StartMonth + "." + StartDay; 
   string CalName   = CalendarName + " " + StartTime;
   
   if(!ReadFromFile)
   {
   int handle = FileOpen(CalendarDirectory + "\\" + CalName + ".csv",FILE_READ|FILE_CSV,";");
   FileClose(handle);
        
   string WebAdress = "http://ec.forexprostools.com/";
   
   strHTM = "";
        
        strHTM = httpGET(WebAdress);
   
      if(strHTM != "" || StringFind(strHTM,"Access Denied",0) < 0) 
      {
         if(SaveHTMFormat)
         {
         handle_htm = FileOpen(CalendarDirectory + "\\" + CalName + ".htm",FILE_WRITE|FILE_CSV);   
            
            if(handle_htm > 0)
            {
            FileWrite(handle_htm,strHTM);
            FileClose(handle_htm);
            }
            else 
                 {
                 _GetLastError = GetLastError();
                 Print("LoadWeek() - FileOpen() Error #",_GetLastError,"!");
                 //return("");
                 }
              }   
             
           strCSV = "";
           Print( "LoadWeek(", StartTime, "): writing csv..." );   
           handle_csv = FileOpen(CalendarDirectory + "\\" + CalName + ".csv",FILE_WRITE|FILE_CSV,";");   
           
              if(ConvertHTMtoCSV(StartWeek,strHTM,strCSV))
              {   
                 if(handle_csv > 0)
                 {
                      FileWrite(handle_csv,strCSV);
                      FileClose(handle_csv);
                 }
                 else
                 {
                      _GetLastError = GetLastError();
                      Print("LoadWeek() - FileOpen() Error #",_GetLastError,"!");
                      return("");
                 }
              }   
           }
        }

or I attach this News Trader EA here (yes, it is really public EA), and you can check by yourself
(it is version 6.3; I am having version 6.4 too - means - next version).

---------------------

I want to remind that this EA with the versions were created long time ago on the forum.

Files:

NewsTrader_v6.3_600r.mq4 115 kb

Martini ... PriceChannel Parabolic system Something interesting, old thread

Sergey Golubev 2023.03.21 15:16 #43

Ugochukwu Mobi #:

Thank you so much but may you please explain the format of data more?? because to access date it is only few codes I don't see where I will change.

The code above that is what I was using and it was working great now to change the format what did I need to change in that code?
here the file is downloaded

but when I open it there is useless information

@Sergey Golubev how to change the format?

I use -

string WebAdress = "http://ec.forexprostools.com/";

you use -

https://ec.forexprostools.com/

It is not used https in this EA ...

Anyway, EA was uploaded (it is not the latest/recent versions but it works).

Lorentzos Roussos 2023.03.21 15:35 #44

I set the referer header to investing.com/calendar and the user agent to apple iphones safari and it worked . I had the site open for a while though at the same time . maybe related

  char data[],result[];
  string headers="";
  int status=WebRequest("GET","https://sslecal2.forexprostools.com","Referer: https://www.investing.com/economic-calendar/\r\nUser-Agent: Mozilla/5.0 (iPhone; CPU iPhone OS 16_3_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.3 Mobile/15E148 Safari/604.1\r\n",15000,data,result,headers); 
  Print("Status "+status);
  Print("Data received "+IntegerToString(ArraySize(result)));
  int f=FileOpen("INVWR.html",FILE_WRITE|FILE_TXT);
  FileWriteString(f,headers);
  FileWriteString(f,CharArrayToString(result,0,WHOLE_ARRAY,CP_UTF8));
  FileClose(f);

Files:

simple.mq5 3 kb

Drazen Penic 2023.03.21 16:36 #45

Ugochukwu Mobi #:

Thank you so much but may you please explain the format of data more?? because to access date it is only few codes I don't see where I will change.

The code above that is what I was using and it was working great now to change the format what did I need to change in that code?
here the file is downloaded

but when I open it there is useless information

@Sergey Golubev how to change the format?

This useless information indicates that your request is blocked by Cloudflare, a service used to cache and protect websites from attacks.

Since you are scraping web page, site owner probably want's to ban that - they put Cloudflare in front so they can filter traffic.

Cloudflare can block single IP address or range of addresses, depending on the rules.

This is not Metatrader problem. Site owners are trying to prevent web scraping.

It can be helpfull to add referrer and user agent like Lorentzos did. This can imitate request from the web browser.

Getting Signal Trading history WebRequest is being cached I have been locked

Ugochukwu Mobi 2023.03.21 17:07 #46

Sergey Golubev #:

I use -

you use -

It is not used https in this EA ...

Anyway, EA was uploaded (it is not the latest/recent versions but it works).

Thank you so much for willing to assist the code works well but unfortunately I can't use it because it uses DLL. it has dependancy.

Ugochukwu Mobi 2023.03.21 17:09 #47

Lorentzos Roussos #:

I set the referer header to investing.com/calendar and the user agent to apple iphones safari and it worked . I had the site open for a while though at the same time . maybe related

Thank you so much @Lorentzos Roussos

Lorentzos Roussos 2023.03.21 17:23 #48

Ugochukwu Mobi #:

Thank you so much @Lorentzos Roussos

You are welcome

Also , guys , cache the results . You are receiving timestamps which informs you on when the last item is , in other words when you might need to check again .

Ugochukwu Mobi 2023.03.21 17:50 #49

Lorentzos Roussos #:

You are welcome

Also , guys , cache the results . You are receiving timestamps which informs you on when the last item is , in other words when you might need to check again .

It checks twice per day. You save me a lot of time but when I will have time, I will add the second source and it will be mql5 calendar.

Sugianto 2023.03.21 23:36 #50

Lorentzos Roussos #:

I set the referer header to investing.com/calendar and the user agent to apple iphones safari and it worked . I had the site open for a while though at the same time . maybe related

thank you lorentzos roussos, it worked like a charm. this really helped me.

News Filter Not Working. What is the new alternative??? - page 5