(SOLVED) WebRequest - How to grab configured data webpage? (SOLVED by @Lorentzos Roussos)

New comment
 

Hi there..

I'm trying to work with grabbed data from webpage.



The webpage is: https://investing.com/technical/pivot-points-fibonacci

default Hourly


It shows Hourly data by default.
I wanna grab 30Minutes data.
Do you have any idea how to grab it using WebRequest function?

Thank you.

Pivot Point Fibonacci Forex - Investing.com
Pivot Point Fibonacci Forex - Investing.com
  • id.investing.com
Pivot Point Fibonacci Forex didasarkan pada kalkulasi level-level Fibonacci.
 
Do you really expect us to know how an external web site works? You want M30 data — ask them for the corresponding URL.
 
Hady Candra:

Hi there..

I'm trying to work with grabbed data from webpage.



The webpage is: https://investing.com/technical/pivot-points-fibonacci


It shows Hourly data by default.
I wanna grab 30Minutes data.
Do you have any idea how to grab it using WebRequest function?

Thank you.

You want some idea? well, I usually scrap the data with Python then pass the .csv to metatrader. It's simple and doesn't require much code (100 line at most).

Search for "python requests" on google.

Good luck.

 

There is a form in there that corresponds to this request .
So maybe you can submit a Post web request .
It appears the only parameter is in the select .


//i suspect something like 
void Pull(int minutes)
{
string params[1];
params[0]="period="+IntegerToString(minutes);//period assumed from select element on site
string server_response=WR_REQUEST(wr_post,"https://www.investing.com/technical/pivot-points-fibonacci",params,"Content-Type: application/x-www-form-urlencoded",NULL,NULL,"https://www.investing.com",5000);
ArrayFree(params);
}

//REQUEST CODE 
enum wr_type
{
wr_get=0,//GET
wr_post=1//POST
};
string WR_REQUEST(wr_type request_type,string url,string &params[],string type,string user_agent,string cookie,string referer,uint timeout)
{
string response="not sent";
string headers="";
/*
for headers , in type string include header descriptor (e.g. Content-Type:)
              in user agent string include user agent descriptor (e.g. user-agent:)
*/
if(type!=NULL) headers=type;
if(user_agent!=NULL) headers=headers+"\r\n"+user_agent;
char post[],result[];
int res;
string target_url=url;
string specs="";
//fill specs if they exist 
int params_total=ArraySize(params);
bool noparams=false;
if(params_total==1&&params[0]=="") noparams=true;
if(noparams==false)
{
for(int fp=0;fp<params_total;fp++)
{
specs=specs+params[fp];
if(fp<params_total-1) specs=specs+"&";
}
}
if(request_type==wr_get&&noparams==false) target_url=target_url+"?"+specs;
char data[];
int data_size=0;
int slen=StringLen(specs);
if(request_type==wr_post) data_size=StringToCharArray(specs,data,0,slen,CP_UTF8);
ResetLastError();
string req_string="GET";
if(request_type==wr_post) req_string="POST";
res=WebRequest(req_string,target_url,cookie,referer,timeout,data,data_size,result,headers);
if(res==-1)
{
Print("Error in WebRequest. Error code  =",GetLastError());
MessageBox("Add the address '"+url+"' in the list of allowed URLs on tab 'Expert Advisors'","Error",MB_ICONINFORMATION);
}
else
{
PrintFormat("The file has been successfully loaded, File size =%d bytes.",ArraySize(result));
int tit=ArraySize(result)-1;
string html="";
for(int xx=0;xx<=tit;xx++)
{
html=html+CharToStr(result[xx]);
}
response=html;
}  
return(response);
ArrayFree(result);
}
//REQUEST CODE ENDS HERE
 
William Roeder:
Do you really expect us to know how an external web site works? You want M30 data — ask them for the corresponding URL.

@William Roeder... thank you for your suggestion, but I think they will ignore what I'm asking for.



Ahmad Zuhairdi Noh:

You want some idea? well, I usually scrap the data with Python then pass the .csv to metatrader. It's simple and doesn't require much code (100 line at most).

Search for "python requests" on google.

Good luck.

@Ahmad Zuhairdi Noh thank you for your experienced idea... I'm not familiar with Python.. but will look in the future to enriched references. 


Lorentzos Roussos:

There is a form in there that corresponds to this request .
So maybe you can submit a Post web request .
It appears the only parameter is in the select .


@Lorentzos Roussos thank you so much, Sir. 
PERFECT.... That's what I'm looking for.
the way you did (inspecting elements from chrome browser?) was incredibly awesome.
I didn't expect you even created the PLUG&PLAY function, so I can just copy&paste them and try to run it..
I added some lines to save the RESULT ARRAY into *.html file.

here's the raw added version:

//i suspect something like 
void Pull(int minutes)
{
string params[1];
params[0]="period="+IntegerToString(minutes);//period assumed from select element on site
string server_response=WR_REQUEST(wr_post,"https://www.investing.com/technical/pivot-points-fibonacci",params,"Content-Type: application/x-www-form-urlencoded",NULL,NULL,"https://www.investing.com",5000);
ArrayFree(params);
}

//REQUEST CODE 
enum wr_type
{
wr_get=0,//GET
wr_post=1//POST
};
string WR_REQUEST(wr_type request_type,string url,string &params[],string type,string user_agent,string cookie,string referer,uint timeout)
{
string response="not sent";
string headers="";
/*
for headers , in type string include header descriptor (e.g. Content-Type:)
              in user agent string include user agent descriptor (e.g. user-agent:)
*/
if(type!=NULL) headers=type;
if(user_agent!=NULL) headers=headers+"\r\n"+user_agent;
char post[],result[];
int res;
string target_url=url;
string specs="";
//fill specs if they exist 
int params_total=ArraySize(params);
bool noparams=false;
if(params_total==1&&params[0]=="") noparams=true;
if(noparams==false)
{
for(int fp=0;fp<params_total;fp++)
{
specs=specs+params[fp];
if(fp<params_total-1) specs=specs+"&";
}
}
if(request_type==wr_get&&noparams==false) target_url=target_url+"?"+specs;
char data[];
int data_size=0;
int slen=StringLen(specs);
if(request_type==wr_post) data_size=StringToCharArray(specs,data,0,slen,CP_UTF8);
ResetLastError();
string req_string="GET";
if(request_type==wr_post) req_string="POST";
res=WebRequest(req_string,target_url,cookie,referer,timeout,data,data_size,result,headers);
if(res==-1)
{
Print("Error in WebRequest. Error code  =",GetLastError());
MessageBox("Add the address '"+url+"' in the list of allowed URLs on tab 'Expert Advisors'","Error",MB_ICONINFORMATION);
}
else
{
PrintFormat("The file has been successfully loaded, File size =%d bytes.",ArraySize(result));


//ADDED CODE TO SAVE SAVED.HTM FILE
//ADDED CODE TO SAVE SAVED.HTM FILE
         int filehandle=FileOpen("SAVED"+".htm",FILE_WRITE|FILE_BIN); 
         if(filehandle!=INVALID_HANDLE) 
           {
            FileWriteArray(filehandle,result,0,ArraySize(result)); 
            FileClose(filehandle); 
           } 
         else Print("Error FileOpen. code=",GetLastError()); 
//end of ADDED CODE TO SAVE SAVED.HTM FILE
//end of ADDED CODE TO SAVE SAVED.HTM FILE


int tit=ArraySize(result)-1;
string html="";
for(int xx=0;xx<=tit;xx++)
{
html=html+CharToStr(result[xx]);
}
response=html;
}  
return(response);
ArrayFree(result);
}
//REQUEST CODE ENDS HERE



Best regards,
Hady


(case closed - problem perfectly solved by @Lorentzos Roussos)

 
You are welcome . 
Yes the names of input fields inside form elements in html are usually related to post request parameters.
New comment
Reason: