reading second row of a csv file - page 2

New comment
 

Thank you for your help! I can finally read that file in tester :) for reference here is the code I finished with:

fn = prefix + sym + ".csv";

int line2read=2;
int values=7;    
string Piv[2,7];                                 
                                                     
handle=FileOpen(fn,FILE_CSV|FILE_READ,',');     
if(handle>0){                                         
   int line=0;
   string myline;
   while(FileTell(handle)< FileSize(handle)){       
      for(int i=0;i<values;i++){                  
         Piv[line,i]=FileReadString(handle);
      }   
      line++;                                         
      if(line==line2read)break;      
   }
}   
if(handle<0){return(false);}
FileClose(handle);

for(int j=0;j<7;j++){

                                double fPiv = NormalizeDouble(StrToDouble(Piv[line2read-1,j]),Digits);
                                switch(j){
                                        case 0 : pivot[3] = fPiv; break;
                                        case 1 : pivot[2] = fPiv; break;
                                        case 2 : pivot[1] = fPiv; break;
                                        case 4 : pivot[4] = fPiv; break;
                                        case 5 : pivot[5] = fPiv; break;
                }  
                        
   }
   if(pivot[3]>0){return(true);}
}
 
1. i'll post here only the function declaration. the implementation has 115 lines, i guess that doesn't really make much sense in a post. just look into the attachement.

/**
 * Reads entire file into an array. Each element of the array corresponds to a line in the file,
 * with the newline separator removed.  Returns the number of resulting elements/lines.
 *
 * @param  string filename       - filename relative to .\experts\files
 * @param  string result[]       - array for resulting lines
 * @param  bool   skipEmptyLines - whether or not to skip empty lines
 *
 * @return int - number of resulting lines or -1 in case of any errors
 */
int FileReadLines(string filename, string result[], bool skipEmptyLines);

the function reads the full file. for normal files this is fine but if you process larger files (> 1MB) and only need to access line 2 the processing and memory overhead cannot be ignored, especially during trading. it would make sense to add an additional parameter "count" to stop processing after 2 lines, for example. until now i had no need for that, it's a simple change/breakout of the loop.


2. MQL is not made for reading "lines", all functions are designed for reading "values". that makes it a bit complicated to process lines in MQL. in fact it means whatever we do/want we can only read values, if we want lines we always need to reconstruct.
instead of avoiding one character in all files (and failing with that approach only 2 weeks later :-)) the opposite assumption is better. "no line must contain character XYZ" becomes "at least one line might contain character XYZ" and we can write the implementation around that. MQL gives us only values anyway, so it's not a problem if the separator character is mixed up with the file content as long as we know how to correctly detect line ends. if we need lines and we have to reconstruct them anyway, line ends are more important than value/field separators.

3. see the attachment. special care is taken for the last line. we never know if the last line is followed by another EOL or if this last EOL is missing (last line character + end_of_file). but it's a valid line, so we cannot lose it.

i don't like implementations long like that (for readability and maintaining) but sometimes there's no better way. the code also implements a proper way of error handling, small and very effective. to me the function is quite bullet proof, but who knows so i keep the error handling in. i want to know if/when it breaks...
Files:
filereadlines.zip  3 kb
 

Thanks a lot! Much appreciated.

 

for FileIsLineEnding() the documentation simply is plain wrong. it must read like this:

bool FileIsLineEnding( int handle)

For CSV file returns the current value of the internal "line ending" flag. The flag is set by the previous FileReadString() call and will be reset/set by the next FileReadString() call. FileIsLineEnding() will not check any characters at the current position of the file pointer by itself.

same is true according to FileIsEnding().

 

to complete your use case one more function and a bug fix for StringSubstr()

/**
 * Splits a string into substrings.
 *
 * @param  string input     - string to split
 * @param  string separator - separator string (can be longer then 1 character)
 * @param  string results[] - array containing the resulting substrings
 * @param  int    limit     - maximum number of substrings to extract (default: no limit)
 *
 * @return int - number of extracted substrings or -1 in case of any errors
 */
int Explode(string input, string separator, string &results[], int limit=NULL) {
   // parameter "input" *could* be an existing element of otherwise pre-populated array results[].  
   // to not overwrite it we work with a copy of it.
   string _input = StringConcatenate(input, "");

   int lenInput     = StringLen(input),
       lenSeparator = StringLen(separator);

   if (lenInput == 0) {                      // empty string
      ArrayResize(results, 1);
      results[0] = _input;
   }
   else if (StringLen(separator) == 0) {     // empty separator: explode into single characters
      if (limit==NULL || limit > lenInput)
         limit = lenInput;
      ArrayResize(results, limit);

      for (int i=0; i < limit; i++) {
         results[i] = StringSubstr(_input, i, 1);
      }
   }
   else {                                    // most common case: normal explode into substrings
      int size, pos;
      i = 0;

      while (i < lenInput) {
         ArrayResize(results, size+1);

         pos = StringFind(_input, separator, i);
         if (limit == size+1)
            pos = -1;
         if (pos == -1) {
            results[size] = StringSubstr(_input, i);
            break;
         }
         else if (pos == i) {
            results[size] = "";
         }
         else {
            results[size] = StringSubstrFix(_input, i, pos-i);
         }
         size++;
         i = pos + lenSeparator;
      }

      if (i == lenInput) {                   // input ends with separator -> last element is an empty string (the one after the separator)
         ArrayResize(results, size+1);
         results[size] = "";                 // TODO: check interaction of limit and separator at the end
      }
   }

   if (IsError(catch("Explode()")))
      return(-1);
   return(ArraySize(results));
}


/**
 * Bug fix/workaround for StringSubstr(string, start, length): MQL returns utter nonsense for length=0
 * Additional negative values for start and length.
 *
 * @param  string object
 * @param  int    start  - if negative, start index from the end of string
 * @param  int    length - if negative, number of characters to return left-side of start 
 *
 * @return string
 */
string StringSubstrFix(string object, int start, int length=EMPTY_VALUE) {
   if (length == 0)
      return("");

   if (start < 0)
      start = MathMax(0, start + StringLen(object));

   if (length < 0) {
      start += 1 + length;
      length = MathAbs(length);
   }
   return(StringSubstr(object, start, length));
}

with this your use case is solved like this (assumed, every line is expected to contain 6 values separated by tabs). btw., it's the separator used in FileReadLines() -> maximum mix-up :-)

string lines[], values[], filename="myfile.csv";
if (FileReadLines(filename, lines, false) < 2)
   return(catch("MyFunction()   file \""+ filename +"\" doesn't contain the expected number of lines", ERR_RUNTIME_ERROR));

if (Explode(lines[1], "\t", values) != 6) 
   return(catch("MyFunction()   line 2 of file \""+ filename +"\" doesn't contain the expected number of values", ERR_RUNTIME_ERROR));

string value1 = values[0];
...
string value6 = values[5];

regards


ps: some minor modifications in the last block since posting it the 1st time

 

Hi Paule
Thanks again.
According to the "TODO" in the code:

- Limit works as expected:
2012.12.02 21:53:19 test EURUSD,M5: Limit set to 2: Value1: 2079114; Value2: 1.3003;1354301865;EURUSD;0.01;1.29583;1.30385;0;0;

- Separater at the end of the line adds a new (empty) value:
2012.12.02 21:58:28 test EURUSD,M5: ArraySize without separator: 9
2012.12.02 21:59:51 test EURUSD,M5: ArraySize with separator: 10

I added the above code to the library and defined the functions in the header file. The default values in library function do not work or rather the function cannot be called without the optional parameter.  -.   " ')' - wrong parameters count C:\mt4\demo\acm\experts\test.mq4 (30, 33)"
It seems that others have had the same issues.... Any experience with that?

edit: it is not possible: https://docs.mql4.com/basis/variables/formal "MQL4-library functions imported within other modules cannot have parameters initialized by default values."

 
kronin:

Hi Paule
Thanks again.
According to the "TODO" in the code:

- Limit works as expected:
2012.12.02 21:53:19 test EURUSD,M5: Limit set to 2: Value1: 2079114; Value2: 1.3003;1354301865;EURUSD;0.01;1.29583;1.30385;0;0;

- Separater at the end of the line adds a new (empty) value:
2012.12.02 21:58:28 test EURUSD,M5: ArraySize without separator: 9
2012.12.02 21:59:51 test EURUSD,M5: ArraySize with separator: 10

I added the above code to the library and defined the functions in the header file. The default values in library function do not work or rather the function cannot be called without the optional parameter.  -.   " ')' - wrong parameters count C:\mt4\demo\acm\experts\test.mq4 (30, 33)"
It seems that others have had the same issues.... Any experience with that?

edit: it is not possible: https://docs.mql4.com/basis/variables/formal "MQL4-library functions imported within other modules cannot have parameters initialized by default values."

 

thanks for testing.

as for the default parameters you are right. FileReadLine is an excerpt from my real stdlibrary and that one is big, about 500 utility functions. default parameters inside a library are indeed supported, so there they are very useful. i just kept them in the code.

12
New comment