Download MetaTrader 5
To add comments, please log in or register
Do you have questions to the application's developer - ask them in Comments section
Carl Schreiber
5930
Carl Schreiber 2014.02.15 21:01 

Well I managed to dld the csv-file for yesterday from the tds-calendar,

url: http://calendar.forex-tsd.com/calendar.php?csv=1&date=20140214&calendar[]=4&gmt=0

If I paste that into the address line of my FF, FF asks me to open this files with LibreOffice, which I can and where I see and recognize all Friday's News :)

OK, the address is correct!

Now I let mt4 b600+ grab this file:

GrabWeb(string strUrl, string& result, int debug) {
        int hInternet;
        int iResult;
        int lReturn[]   = {1};
        string sBuffer  = "                                                                                                                                                                                                                                                           ";        // 255 spaces
        int bytes;

        //strUrl = "http://calendar.forex-tsd.com/calendar.php?csv=1&date=20140214&calendar[]=4&gmt=0":
   int HttpOpen = InternetOpenW(" ", 0, " "," ",0 ); 
   int HttpConnect = InternetConnectW(HttpOpen, "", 80, "", "", 3, 0, 1); 
   hInternet = InternetOpenUrlW(HttpOpen,strUrl, NULL, 0, 0, 0);        
        
        if (debug>0) Print ("grab 1: Reading URL: " + strUrl, " hInternet: ",hInternet);           //added by MN        
                                                                
        if (hInternet == 0) 
                return (FALSE);

        iResult = InternetReadFile(hInternet, sBuffer, Buffer_LEN, lReturn);
        
        if (iResult == 0) 
                return (FALSE);
        bytes = lReturn[0];

        result = StringSubstr (sBuffer, 0, lReturn[0]);
        
        // If there's more data then keep reading it into the buffer
        while (lReturn[0] != 0) {
                iResult = InternetReadFile(hInternet, sBuffer, Buffer_LEN, lReturn);
                if (lReturn[0] == 0) 
                        break;
                bytes = bytes + lReturn[0];
                result = result + StringSubstr (sBuffer, 0, lReturn[0]);
        }
        if ( bytes < 1 ) { 
                strWebPage = "";
                if (debug>0) Print ("grab 2: Closing URL web connection to News-server loaded: ",strUrl," with ",bytes," bytes less than limt, dld is DELETED");  
        } else
                if (debug>0) Print ("grab 3: Closing URL web connection to News-server loaded: ",strUrl," with ",bytes," bytes ");  

        iResult = InternetCloseHandle(hInternet);
}

//In the expert tab I read:
grab 3: Closing URL web connection to News-server loaded: http://calendar.forex-tsd.com/calendar.php?csv=1&date=20140214&calendar[]=4&gmt=0 with 814 bytes 
It's exactly what I want - so far.

And the result is written to a local file herewith:

...
                if (result != "")               {         
                        handle = FileOpen (CalName, FILE_CSV | FILE_WRITE, ';');         
                        if (handle > 0) {
                                FileWrite (handle, result);
                                FileClose (handle);
...

Even this is done without any error, BUT:

from here on if I open this file and try to read it mt4 b600 cannot :(

...
int handle = FileOpen (fName, FILE_CSV | FILE_READ, ';');
...
while (!FileIsEnding(handle)) {
                sDate                                   = StringTrimLeft( StringTrimRight( FileReadString (handle)));          
                sTime                                   = StringTrimLeft( StringTrimRight( FileReadString (handle))); 
                News_Time                               = StrToTime(sDate + " " + sTime) + GmtSecShift; //GMTOffset * 3600; // GmtSecShift
...
}
FileClose(handle);
if (debug>0) Print("read 2: read TSD-File(",fName,", ",GmtSecShift,") read ",NewsTotal," News, from[0] ",TimeToStr(NewsTimes[0])," ",NewsCurrencies[0]," ",NewsDescriptions[0]," ",NewsRatings[0],"  until[",NewsTotal,"]  ",TimeToStr(NewsTimes[NewsTotal])," ",NewsCurrencies[NewsTotal]," ",NewsDescriptions[NewsTotal]," ",NewsRatings[NewsTotal]);
// I read in the expert-tab:
//read TSD-File(ForexTSD_2014-02-14.csv, 0) read 0 News, from[0] 2014.02.15 00:00   0  until[0]  2014.02.15 00:00   0
//And the debugger (within mt4 b600!!) shows for sDate and sTime just ???????????????

If I open the saved file with the Notepad++-editor I read only 1 line:

?????????????????-?????????????????????? ??-????????????????????????????????????? ....

The Encoding Option of Notepad++ tells me ANSI.

And even if I try to open this csv-file with Libre-Office I only see ??????????????????? hundreds of them, no date, no time, no news, nothing but ??

So do I have to do to get out of this mess?

Gooly


Adding a little joke look a the picture what mt4 wants to 'give' me as the fin. news from yesterday, Friday:

content of the csv-file

PS: I tried to go back to ..OpenA(..) and ..ConnectA(..) and ..OpenUrlA(..) but then I don't get a valid handle: hInternet == 0 :(

Tjipke de Vries
6275
Tjipke de Vries 2014.02.15 23:16  

GrabWeb not working on MT4 Build 600 ( 1 2 3 )

//+------------------------------------------------------------------+
//|                                                      GRABWEB.mq4 |
//|                               Copyright © 2014,  Tjipke de Vries |
//| https://forum.mql4.com/60608/page2#906517                 WDholic |
//+------------------------------------------------------------------+
#property copyright "Copyright © 2014,  Tjipke de Vries"
#property link      ""
#property version   "1.00"
#property strict

#import  "Wininet.dll"
   int InternetOpenW(string, int, string, string, int);
   int InternetConnectW(int, string, int, string, string, int, int, int); 
   int HttpOpenRequestW(int, string, string, int, string, int, string, int); 
   int InternetOpenUrlW(int, string, string, int, int, int);
   int InternetReadFile(int, string, int, int& OneInt[]);
   int InternetCloseHandle(int); 
   
#import

//+------------------------------------------------------------------+
//| Script program start function                                    |
//+------------------------------------------------------------------+
void OnStart()
  {
//---

   string URL="http://calendar.forex-tsd.com/calendar.php?csv=1&date=20140214&calendar[]=4&gmt=0";
  
    int HttpOpen = InternetOpenW(" ", 0, " "," ",0 ); 
    int HttpConnect = InternetConnectW(HttpOpen, "", 80, "", "", 3, 0, 1); 
    int HttpRequest = InternetOpenUrlW(HttpOpen,URL, NULL, 0, 0, 0);
   
   int read[1];
   int end;
   string Buffer = " ";
   string NEWS = "";


      string fileName = "DailyFX-News.csv";   
           string xmlFileName = fileName;
           int xmlHandle=FileOpen(xmlFileName, FILE_READ);
           //File exists if FileOpen return >=0. 
           if (xmlHandle >= 0) {FileClose(xmlHandle); FileDelete(xmlFileName);}    
   
   
                xmlHandle=FileOpen(xmlFileName, FILE_CSV|FILE_WRITE, ';');                 


   while (true)
   {
      InternetReadFile(HttpRequest, Buffer, StringLen(Buffer), read);
      if (read[0] > 0) NEWS = NEWS + StringSubstr(Buffer, 0, read[0]);
         else
           {
           FileWriteString(xmlHandle, NEWS);
           FileClose(xmlHandle);
           break;
           }          
   }
     
   if (HttpRequest > 0) InternetCloseHandle(HttpRequest); 
   if (HttpConnect > 0) InternetCloseHandle(HttpConnect); 
   if (HttpOpen > 0) InternetCloseHandle(HttpOpen);
   MessageBox(NEWS, "HTTP READ:" );   
  }
//+------------------------------------------------------------------+

@ Gooly Don't make your posting so wide

Carl Schreiber
5930
Carl Schreiber 2014.02.16 00:11  
deVries:

GrabWeb not working on MT4 Build 600 ( 1 2 3 )

@ Gooly Don't make your posting so wide


Thank you - but as I wanted to test your solution I started b600 /portable it updates to b610 and and can't compile any more:

2014.02.16 01:01:06.754 Compiler        cannot create interface
2014.02.16 01:01:00.703 Compiler        invalid version of MQL engine - 0 instead of 887
2014.02.16 01:01:00.703 Compiler        invalid MQL5 compiler engine version 883 instead of 887

sigh.

May be tomorrow when I get up I see it was just a nightmare.

gchrmt4
356
gchrmt4 2014.02.16 10:44  

I don't want to be picky, but your code is reading the HTTP response one byte at a time, and would start becoming very slow for large amounts of data. (It also relies on an Ansi->Unicode conversion which would not work properly if the server's response were something like UTF8, and relies on a particular layout of strings and other data in MT4's memory. If Buffer is held in memory exactly adjacent to other data then you could sometimes get random garbage appended into the string.)

The code at https://www.mql5.com/en/code/1998 (mentioned in GrabWeb not working on MT4 Build 600) uses the much more efficient technique of using InternetReadFile to read into a uchar[] buffer, and then doing conversion on the buffer.

VabRus
100
VabRus 2014.02.16 11:41  

A best way could be using

URLDownloadToFileW

Carl Schreiber
5930
Carl Schreiber 2014.02.16 12:04  
gchrmt4:

I don't want to be picky, but your code is reading the HTTP response one byte at a time, and would start becoming very slow for large amounts of data. (It also relies on an Ansi->Unicode conversion which would not work properly if the server's response were something like UTF8, and relies on a particular layout of strings and other data in MT4's memory. If Buffer is held in memory exactly adjacent to other data then you could sometimes get random garbage appended into the string.)

The code at https://www.mql5.com/en/code/1998 (mentioned in GrabWeb not working on MT4 Build 600) uses the much more efficient technique of using InternetReadFile to read into a uchar[] buffer, and then doing conversion on the buffer.


I was using a buffer with 255 bytes ( and got the advice not to make my lines so long;):

string sBuffer  = "    {...}          "; // 255 spaces

So I assume that this causes the char-marmelade I got?

Now it works - Thanks deVries!

Gooly

VaBus: I need the content immediately. Saving is only for security.

Thanks again to all!

gchrmt4
356
gchrmt4 2014.02.16 12:37  
gooly:

I was using a buffer with 255 bytes ( and got the advice not to make my lines so long;):

So I assume that this causes the char-marmelade I got?

The deVries code works, slowly, because it reads one byte at a time. Indirectly - and perhaps unintentionally - this does Ansi->Uncode conversion on the one byte. (The code in easymql.mqh at https://www.mql5.com/en/code/1998 is a much, much better way of using InternetReadFile.)

If you use a larger string buffer such as 255 bytes, then you are reading Ansi data into a buffer where MT4 is expecting Unicode, and you are almost certainly going to get garbage.

Carl Schreiber
5930
Carl Schreiber 2014.02.16 14:07  
gchrmt4:

The deVries code works, slowly, because it reads one byte at a time. Indirectly - and perhaps unintentionally - this does Ansi->Uncode conversion on the one byte. (The code in easymql.mqh at https://www.mql5.com/en/code/1998 is a much, much better way of using InternetReadFile.)

If you use a larger string buffer such as 255 bytes, then you are reading Ansi data into a buffer where MT4 is expecting Unicode, and you are almost certainly going to get garbage.


You are right, but it needs some changes - this works:

#import "wininet.dll"

   int InternetOpenW(string,int,string,string,int);
   int InternetConnectW(int,string,int,string,string,int,int,int);
   int InternetOpenUrlW(int,string,string,int,int,int);
   int InternetReadFile(int hFile,uchar &sBuffer[],int lNumBytesToRead,int &lNumberOfBytesRead[]);   
   int InternetCloseHandle(int);
#import

...

bool GrabWeb(string strUrl, string& result, int debug=0) {
// MUST HAVE THIS DEFINITION!!:    
// int InternetReadFile(int hFile,uchar &sBuffer[],int lNumBytesToRead,int &lNumberOfBytesRead[]);   

   int HttpOpen=InternetOpenW(" ",0," "," ",0);
   int HttpConnect = InternetConnectW(HttpOpen, "", 80, "", "", 3, 0, 1);
   int HttpRequest = InternetOpenUrlW(HttpOpen,strUrl, NULL, 0, 0, 0);
        
   int read[1];
   int sz;
   
   uchar buffer[1024];
   bool isOk;
   while(true) {
      isOk = InternetReadFile(HttpRequest,buffer,1024,read);
      sz += read[0];
      if(read[0]>0) result=result+CharArrayToString(buffer,0,read[0],CP_UTF8);
      else break;
   }

   if(HttpRequest>0) InternetCloseHandle(HttpRequest);
   if(HttpConnect>0) InternetCloseHandle(HttpConnect);
   if(HttpOpen>0) InternetCloseHandle(HttpOpen);
   if (sz>0) return(true);
   else return(false);

}

thanks!

VabRus
100
VabRus 2014.02.16 17:05  
#import "urlmon.dll"
  int URLDownloadToFileW(int pCaller,string szURL,string szFileName,int dwReserved,int Callback);
#import
int PeriodTimeWeek;
string urlNews = "http://calendar.forex-tsd.com/calendar.php?csv=1&calendar[]=4&";

         if (PeriodTimeWeek < iTime(NULL, PERIOD_W1, 0))
         {
            PeriodTimeWeek = iTime(NULL, PERIOD_W1, 0);
            date = StringConcatenate(getCorrectDate);
                                            
            string completeUrlNews = StringConcatenate(urlNews, "date=", 
                                                       date, "gmt=", 
                                                       IntegerToString(serverGMT()));
            string file = StringConcatenate(date, ".csv");
            if (!FileIsExist(file))
            {
               int res = URLDownloadToFileW(NULL, completeUrlNews, file, 0, NULL);
               if (res == 0)
                  Print("Download OK");
               else
                  Print("Download failed");
           }
        }

string getCorrectDate()
{
   string month = TimeMonth(TimeCurrent());
   string day = TimeDay(TimeCurrent());
   if (StringLen(month) == 1)
   {
      month = StringConcatenate("0", month);
   }
   if (StringLen(day) == 1)
   {
      day = StringConcatenate("0", day);
   }
   return(StringConcatenate(TimeYear(TimeCurrent()), month, day));
} 

It´s working... Once a week you get the news with only URLDownloadToFileW.

Do Hung
147
Do Hung 2014.04.23 04:30  
VabRus:

A best way could be using

URLDownloadToFileW


That's really good solution!

Be recommended!

/
To add comments, please log in or register