News Filter Not Working. What is the new alternative??? - page 7

[darkylt]

Lorentzos Roussos #:

I run this test code again today out of curiosity

It works on mt4 (with reasonable gaps in between requests)

It returns error 1001 on mt5 (the mt5 error is 5203 but the address is there , i'll check again with my glasses)

(webreqeusts is mt4 mqh)

Hello Sergey, it seems like with mt4 mq4 it cant bypass the cloudflare, please enlighten me 

[Sergey Golubev]

darkylt #:
Hello Sergey, it seems like with mt4 mq4 it cant bypass the cloudflare, please enlighten me 

I downloaded this old NewsTrader EA (for MT4) from this post #42 and placed it in experts folder and compile with MetaEditor.
After that - I attached it to the chart - and it works for me (the news are coming from investing . com calendar):

------------------------

So, you can take some ideas about something (in case you are coding newsfilter oe similar) from this EA (in case of MT4 for example).

[Lorentzos Roussos]

darkylt #:
Hello Sergey, it seems like with mt4 mq4 it cant bypass the cloudflare, please enlighten me 

Use Sergey's code if it works .

Guys you must respect the server you are stealing data from . That is scraping rule #1

Ideally switch to mt5 or forex factory .

[darkylt]

Sergey Golubev #:

I downloaded this old NewsTrader EA (for MT4) from this post #42 and placed it in experts folder and compile with MetaEditor.
After that - I attached it to the chart - and it works for me (the news are coming from investing . com calendar):

------------------------

So, you can take some ideas about something (in case you are coding newsfilter oe similar) from this EA (in case of MT4 for example).

Hi Sergey,

I myself is not a coder, not even close to an amateur c coder, but i studied your idea from #44 about the bypassing the cflare using the header and i GPT my way implemented the idea into my mqh and it works!
The not working one:

string NewsReader::ReadCBOE()
  {

   string cookie=NULL,headers;
   char post[],result[];
   string TXT="";
   int res;
//--- to work with the server, you must add the URL "https://www.google.com/finance"
//--- the list of allowed URL (Main menu-> Tools-> Settings tab "Advisors"):
   string google_url="https://ec.forexprostools.com/?columns=exc_currency,exc_importance&importance=1,2,3&calType=week&timeZone=15&lang=1";
//---
   ResetLastError();
//--- download html-pages
   int timeout=5000; //--- timeout less than 1,000 (1 sec.) is insufficient at a low speed of the Internet
   res=WebRequest("GET",google_url,cookie,NULL,timeout,post,0,result,headers);
//--- error checking
   if(res==-1)
     {
      Print("WebRequest error, err.code  =",GetLastError());
      MessageBox("You must add the address 'https://ec.forexprostools.com/' in the list of allowed URL tab 'Advisors' "," Error ",MB_ICONINFORMATION);
      //--- You must add the address ' "+ google url"' in the list of allowed URL tab 'Advisors' "," Error "
     }
   else
     {
      //--- successful download
      //PrintFormat("File successfully downloaded, the file size in bytes  =%d.",ArraySize(result));
      //--- save the data in the file
      int filehandle=FileOpen("news-log.html",FILE_WRITE|FILE_BIN);
      //--- проверка ошибки
      if(filehandle!=INVALID_HANDLE)
        {
         //---save the contents of the array result [] in file
         FileWriteArray(filehandle,result,0,ArraySize(result));
         //--- close file
         FileClose(filehandle);

         int filehandle2=FileOpen("news-log.html",FILE_READ|FILE_BIN);
         TXT=FileReadString(filehandle2,ArraySize(result));
         FileClose(filehandle2);
        }
      else
        {
         Print("Error in FileOpen. Error code =",GetLastError());
        }
     }

   return(TXT);
  }

---

The working one:

string NewsReader::ReadCBOE()
  {

   string cookie=NULL;
   char post[],result[];
   string TXT="";
   int res;
   
     // --- Construct the headers string
    string headers = "Referer: https://www.investing.com/economic-calendar/\r\n"
                     "User-Agent: Mozilla/5.0 (iPhone; CPU iPhone OS 16_3_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.3 Mobile/15E148 Safari/604.1\r\n";
//--- to work with the server, you must add the URL "https://www.google.com/finance"
//--- the list of allowed URL (Main menu-> Tools-> Settings tab "Advisors"):
   string google_url="https://ec.forexprostools.com/?columns=exc_currency,exc_importance&importance=1,2,3&calType=week&timeZone=15&lang=1";
//---
   ResetLastError();
//--- download html-pages
   int timeout=5000; //--- timeout less than 1,000 (1 sec.) is insufficient at a low speed of the Internet
   res = WebRequest("GET", google_url, cookie, headers, timeout, post, 0, result, headers);
//--- error checking
   if(res==-1)
     {
      Print("WebRequest error, err.code  =",GetLastError());
      MessageBox("You must add the address 'https://ec.forexprostools.com/' in the list of allowed URL tab 'Advisors' "," Error ",MB_ICONINFORMATION);
      //--- You must add the address ' "+ google url"' in the list of allowed URL tab 'Advisors' "," Error "
     }
   else
     {
      //--- successful download
      //PrintFormat("File successfully downloaded, the file size in bytes  =%d.",ArraySize(result));
      //--- save the data in the file
      int filehandle=FileOpen("news-log.html",FILE_WRITE|FILE_BIN);
      //--- проверка ошибки
      if(filehandle!=INVALID_HANDLE)
        {
         //---save the contents of the array result [] in file
         FileWriteArray(filehandle,result,0,ArraySize(result));
         //--- close file
         FileClose(filehandle);

         int filehandle2=FileOpen("news-log.html",FILE_READ|FILE_BIN);
         TXT=FileReadString(filehandle2,ArraySize(result));
         FileClose(filehandle2);
        }
      else
        {
         Print("Error in FileOpen. Error code =",GetLastError());
        }
     }

   return(TXT);
  }

I just have to know, is it really the referrer and the header is making this work?

[darkylt]

Lorentzos Roussos #:

Use Sergey's code if it works .

Guys you must respect the server you are stealing data from . That is scraping rule #1

Ideally switch to mt5 or forex factory .

Yes I've seen fx factory has no problem providing that, im just so scared that i messed up the whole thing. but ive learnt a lot here

[Sergey Golubev]

darkylt #:
and i GPT my way implemented the idea into my mqh and it works!
The not working one:

This NewsTrader Ea is not my EA. This EA was publicly coded for the forum and published by source codes with many versions (it was long time ago),
and I used this EA for real account to trade high impacted news events (and my trading results may be found some some threads as well).

So, if you want to improve or fix something - ask any code who will agree to do it, or use Freelance for example.
Because I do not trade the news for now (and I do not code or now).

[Lorentzos Roussos]

darkylt #:

Hi Sergey,

I just have to know, is it really the referrer and the header is making this work?

Yes , the trick Sergey used is exactly that , he is the best coder after all , although modest.

What the referrer is doing is "letting the server know" you came in from a website and you are not going straight to the source to scrap it.

Referrer means previous url in other words

[hamada90]

Lorentzos Roussos #:

I run this test code again today out of curiosity

It works on mt4 (with reasonable gaps in between requests)

It returns error 1001 on mt5 (the mt5 error is 5203 but the address is there , i'll check again with my glasses)

(webreqeusts is mt4 mqh)

bro i tested all files nothing work , please any Update  ?

[Lorentzos Roussos]

hamada90 #:

bro i tested all files nothing work , please any Update  ?

drop mt4 use mt5 . It has a built in calendar

[Ugochukwu Mobi]

hamada90 #:

bro i tested all files nothing work , please any Update  ?

At my side it is still working