Download MetaTrader 5

File Operations via WinAPI

15 August 2008, 09:10
MetaQuotes Software Corp.
8
4 424

Introduction

MQL4 is designed in such a way that even incorrectly written programs are unable to mistakenly delete data from the hard disk. The functions used for file reading and writing operations can work in the following directories only (quote):

  • /HISTORY/<current broker> - especially for the FileOpenHistory function;
  • /EXPERTS/FILES - common case;
  • /TESTER/FILES - especially for testing.
Working with files from other directories is prohibited.

If you still need to work outside the directories (defined for safety reasons), you can call the functions of Windows OS. For this purpose, the functions of API represented in kernel32.dll library are widely used.


File Functions of kernel32.dll

It is based on the script found in the CodeBase under File Operations without Limitations. It is a good example of how functions can be imported into an MQL4 program.

// constants for function _lopen
#define OF_READ               0
#define OF_WRITE              1
#define OF_READWRITE          2
#define OF_SHARE_COMPAT       3
#define OF_SHARE_DENY_NONE    4
#define OF_SHARE_DENY_READ    5
#define OF_SHARE_DENY_WRITE   6
#define OF_SHARE_EXCLUSIVE    7
 
 
#import "kernel32.dll"
   int _lopen  (string path, int of);
   int _lcreat (string path, int attrib);
   int _llseek (int handle, int offset, int origin);
   int _lread  (int handle, string buffer, int bytes);
   int _lwrite (int handle, string buffer, int bytes);
   int _lclose (int handle);
#import

These functions are declared in msdn as obsolete, but they can still be used, see Obsolete Windows Programming Elements. I will give here the description of functions and parameters taken directly from that thread, as they are described by the script author, mandor:

// _lopen  : It opens the specified file. It returns: file descriptor.
// _lcreat : It creates the specified file. It returns: file descriptor.
// _llseek : It places the pointer in the open file. It returns: 
// the new shift of the pointer.
// _lread  : It reads the given number of bytes from the open file. 
// It returns: the number of the read bytes; 0 - if it is the end of the file.
// _lwrite : It writes the data from buffer into the specified file. It returns: 
// the number of written bytes.
// _lclose : It closes the specified file. It returns: 0.
// In case of unsuccessfully completion, all functions return the value of 
// HFILE_ERROR=-1.
 
// path   : String that defines the path and the filename.
// of     : The way of opening.
// attrib : 0 - reading or writing; 1 - only reading; 2 - invisible, or 
// 3 - system file.
// handle : File descriptor.
// offset : The number of bytes, by which the pointer shifts.
// origin : It indicates the initial point and the shifting direction: 0 - 
// forward from the beginning; 1 - from the current position; 2 - backward from the end of the file.
// buffer : Receiving/writing buffer.
// bytes  : The number of bytes to read.
 
// Methods of opening (parameter 'of'):
// int OF_READ            =0; // Open file for reading only
// int OF_WRITE           =1; // Open file for writing only
// int OF_READWRITE       =2; // Open file in the read/write mode
// int OF_SHARE_COMPAT    =3; // Open file in the mode of common 
// shared access. In this mode, any process can open this given 
// file any amount of times. At the attempt to open this file in any other
// mode, the function returns HFILE_ERROR.
// int OF_SHARE_DENY_NONE =4; // Open file in the mode of common access 
// without disabling the reading/writing by another process. At the attempt to open 
// this file in the mode of OF_SHARE_COMPAT, the function returns HFILE_ERROR.
// int OF_SHARE_DENY_READ =5; // Open file in the mode of common access with 
// disabling the reading by another process. At the attempt to open this file 
// with the flags of OF_SHARE_COMPAT and/or OF_READ, or OF_READWRITE, the function 
// returns HFILE_ERROR.
// int OF_SHARE_DENY_WRITE=6; // The same, but with disabling the writing.
// int OF_SHARE_EXCLUSIVE =7; // Disable for this current and for all other processes 
// to access to this file in the modes of reading/writing. The file in this mode can be 
// opened only once (with the current process). All other attempts 
// to open the file will fail.

Function "Reading from File"

Let us consider the function intended for reading from file. Its only parameter is a string variable that contains the file name. The imported function of _lopen(path,0) returns the pointer at an open file and, for its tasks, is very similar to function FileOpen() in MQL4.

//+------------------------------------------------------------------+
//|   read the file and return a string with its contents            |
//+------------------------------------------------------------------+
string ReadFile (string path) 
  {
    int handle=_lopen (path,OF_READ);           
    if(handle<0) 
      {
        Print("Error opening file ",path); 
        return ("");
      }
    int result=_llseek (handle,0,0);      
    if(result<0) 
      {
        Print("Error placing the pointer" ); 
        return ("");
      }
    string buffer="";
    string char1="x";
    int count=0;
    result=_lread (handle,char1,1);
    while(result>0) 
      {
        buffer=buffer+char1;
        char1="x";
        count++;
        result=_lread (handle,char1,1);
     }
    result=_lclose (handle);              
    if(result<0)  
      Print("Error closing file ",path);
    return (buffer);
  }

Function _lseek() has its analog in MQL4, too. It's FileSeek(). Function _lclose is used for closing files, like function FileClose(). The only new function is _lread(handle, buffer, bytes) that reads from the given file (the pointer at which must be preliminarily received by the function of _lopen()) into the variable 'buffer' the given number of bytes. You should use a string constant of the necessary length as the 'buffer ' variable. In this example, we can see:

    string char1="x";
    result=_lread (handle,char1,1);

- the string constant 'char' is given that has the length of one, i.e., it allows to read only one byte into it. At the same time, the value of this constant doesn't matter: it can be both "x", "Z" and even " " (the space character). You will not be able to read more bytes into it than it was initially defined for this constant. In this case, the attempts to read 2 or more bytes won't be successful. Besides, the result of function _lread() is the number of bytes really read. If the file is 20 bytes large and you try to read in a 30-byte long variable more than 20 bytes, the function will return 20. If we consecutively apply this function, we will move along the file reading one block of files by another. For example, a file is 22 bytes large. We start to read it by blocks of 10 bytes. Then, after two calls to function __lread(handle, buff, 10), to bytes at the end of the file will remain unread.


At the third call, __lread(handle, buff, 10) will return 2, i.e., the last 2 bytes will be read. At the fourth call, the function will return zero value - no byte is read, the pointer is at the end of the file. It is this fact that underlies the procedure of reading characters from a file in the cycle:

    while(result>0) 
      {
        buffer=buffer+char1;
        char1="x";
        count++;
        result=_lread (handle,char1,1);
     }

As long as the result (the number of bytes read) is more than zero, function _lread(handle, char1, 1) is called in cycles. As you can see, there is nothing complicated in these functions. The value of the read character is saved in the variable named char1. This character is written from the string variable 'buffer' at the next iteration. Upon completion of the operations, the user-defined function ReadFile() returns the contents of the file read in this variable. As you can see it doesn't create any difficulties.


Function "Writing to File"

Writing is, in a sense, even easier than reading. You should open a file and write into it a byte array using function _lwrite (int handle, string buffer, int bytes). Here, handle is a file pointer obtained by function _lopen(), parameter 'buffer' is a string variable, parameter 'bytes' shows how many bytes should be written. Upon writing, the file is closed with function _lclose(). Let's consider the author's function WriteFile():

//+------------------------------------------------------------------+
//|  write the buffer contents to the given path                     |
//+------------------------------------------------------------------+
void WriteFile (string path, string buffer) 
  {
    int count=StringLen (buffer); 
    int result;
    int handle=_lopen (path,OF_WRITE);
    if(handle<0) 
      {
        handle=_lcreat (path,0);
        if(handle<0) 
          {
            Print ("Error creating file ",path);
            return;
          }
        result=_lclose (handle);
     }
    handle=_lopen (path,OF_WRITE);               
    if(handle<0) 
      {
        Print("Error opening file ",path); 
        return;
      }
    result=_llseek (handle,0,0);          
    if(result<0) 
      {
        Print("Error placing pointer"); 
        return;
      }
    result=_lwrite (handle,buffer,count); 
    if(result<0)  
        Print("Error writing to file ",path," ",count," bytes");
    result=_lclose (handle);              
    if(result<0)  
        Print("Error closing file ",path);
  }

However, we should perform error checking. First of all, we try to open a file for writing:

    int handle=_lopen (path,OF_WRITE);

(function _lopen() is called with parameter OF_WRITE).

If the attempt fails (handle < 0), it tries to create a file with the specified name:

        handle=_lcreat (path,0);

If this function, too, returns a negative pointer, then function WriteFile() will be truncated. The resting code in this function is clear without further explanations. The simplest start() function allows you to check how script File_Read_Write.mq4 works.

//+------------------------------------------------------------------+
//| script program start function                                    |
//+------------------------------------------------------------------+
int start()
  {
//----
    string buffer=ReadFile("C:\\Text.txt");
    int count=StringLen(buffer);
    Print("Bytes counted:",count);
    WriteFile("C:\\Text2.txt",buffer);   
//----
   return(0);
  }
//+------------------------------------------------------------------+

Note that the back slash ("\") is written twice, though it must be once. The matter is that some special characters, like line feed character ("\n") or tabulation character ("\t") must be written using the back slash. If you forget this, then you may have problems specifying the path in the test variable during the program execution.

Please write two consecutive back slashes in string constant, not only one. In this case, the compiler will unambiguously accept it correctly.


Everything works, but there is one spoon of tar in this barrel of honey: script function ReadFile() operates very slowly for large files.


The reason for such slow reading of the file is in that we read the information one by one byte (character). You can see in the figure above that the file sized as 280 324 bytes required 103 seconds. This time is taken by reading 1 character 280 324 times. You can independently check the working time for script File Read Write.mq4 attached to the article. How can we accelerate reading from file? The answer is forced upon us - the function must read, say, 50 characters, not by one character at a time. Then the amount of _lread() function calls will be 50 times reduced. Therefore, the reading time must be reduced 50 times, too. Let's check it.

New Function of Reading a File in Blocks 50 Bytes Each

Change the code naming the new version as xFiles,mq4. Compile it and launch for execution. It will be recalled here that importing functions from DLLs must be enabled in the settings (Ctrl+O).



Thus, the execution time of the modified script xFiles.mq4 made 2047 milliseconds, which is approximately 2 seconds. Divide 103 seconds (the execution time of the initial script) by 2 seconds and obtain 103 / 2 = 51.5 times. Thus, the program execution time has really changed approximately 50 times, as it was expected to be. How was the code modified to achieve this?

The changes are small:

string ReadFile (string path) 
  {
    int handle=_lopen (path,OF_READ);
    int read_size = 50;           
    string char50="x                                                 ";
 
    if(handle<0) 
      {
        Print("Error opening file ",path); 
        return ("");
      }
    int result=_llseek (handle,0,0);      
    if(result<0) 
      {
        Print("Error placing the pointer" ); 
        return ("");
      }
    string buffer="";
    int count=0;
    int last;
    
    result=_lread (handle,char50,read_size);
    int readen;
    while(result>0 && result == read_size) 
      {
        buffer=buffer + char50;
        count++;
        result=_lread (handle,char50,read_size);
        last = result;
     }
    Print("The last read block has the size of, in bytes:", last);
    char50 = StringSubstr(char50,0,last);
    buffer = buffer + char50;    
    result=_lclose (handle);              
    if(result<0)  
      Print("Error closing file ",path);
    return (buffer);
  }

Please note that the string variable 'char50' is now initialized by the constant of 50 characters (character "x" and 49 spaces).


Now we can perform reading from file in such a manner that to read 50 bytes (characters) at a time into this variable:

result=_lread (handle,char50,read_size);

Here: read_size = 50. Of course, it is not likely that the size of the file to be read will always be a multiple of 50 bytes, which means that there will be a time when the value of this function execution will be other than 50. This is a signal to stop the cycle. The last block of characters, read to the variable, will be cut to the amount of bytes really read.


You can change the size of the buffer to be read to the size of N using function lread(), but don't forget to make two modifications:

  1. set the value of read_size for 'N';
  2. initialize string variable 'char50' of the constant length of N (N<256).

Well, we have accelerated the reading operation. The last task remained is processing the error of non-existing path when trying to write a file. In function WriteFile(), an attempt is made to create a file, but the situation is not processed, in which there is no folder containing the path to the file name. So we need another function -

Folder Creating Function

The folder creating function is also available in kernel32.dll - CreateDirectory Function. It should only be noted that this function tries to create only the bottommost folders, it does not create all in-between folders on the path, if they aren't present. For example, if we try use this function to create folder "C:\folder_A\folder_B", the attempt will succeed only if the path of "C:/folder_A" has already existed before the function call. Otherwise, folder_B will not be created. Let's add the new function to the import section:

#import "kernel32.dll"
   int _lopen  (string path, int of);
   int _lcreat (string path, int attrib);
   int _llseek (int handle, int offset, int origin);
   int _lread  (int handle, string buffer, int bytes);
   int _lwrite (int handle, string buffer, int bytes);
   int _lclose (int handle);
   int CreateDirectoryA(string path, int atrr[]);
#import

The first parameter contains the path to create a new folder, while the second parameter, atrr[], serves for specifying permissions for the folder to be created and must be of the _SECURITY_ATTRIBUTES type. We won't provide any information for the second parameter, but just pass the empty array named 'int'. In this case, the folder to be created will inherit all permissions from the parent folder. However, before trying to apply this function, we will have to perform such operation as -

Breaking Apart the Path

Indeed, let's have to create a 4th level folder, for example, like this:

"C:\folder_A\folder_B\folder_C\folder_D"

Here we will call the folder_D a 4th level folder, since there are three levels of folders above it. Disk 'C:' contains "folder_A", folder "C:\folder_A\" contains "folder_B", folder "C:\folder_A\folder_B\" contains "folder_C"; and so on. It means we have to break apart the entire path to the file into array of subfolders. Let's name the necessary function as ParsePath():

//+------------------------------------------------------------------+
//| break apart the path  into an array of subdfolders               |
//+------------------------------------------------------------------+
bool ParsePath(string & folder[], string path)
   {
   bool res = false;
   int k = StringLen(path);
   if (k==0) return(res);
   k--;
 
   Print("Parse path=>", path);
   int folderNumber = 0;
//----
   int i = 0;
   while ( k >= 0 )
      {
      int char = StringGetChar(path, k);
      if ( char == 92) //  back slash "\"
         {
         if (StringGetChar(path, k-1)!= 92)
            {
            folderNumber++;
            ArrayResize(folder,folderNumber);
            folder[folderNumber-1] = StringSubstr(path,0,k);
            Print(folderNumber,":",folder[folderNumber-1]);
            }
         else break;         
         }
      k--;   
      }
   if (folderNumber>0) res = true;   
//----
   return(res);   
   }   
//+------------------------------------------------------------------+

The delimiter between folders is the '\' character that has the value of 92 in the ANSI coding. The function gets 'path' as its argument and fills out the array named folder[], passed to the function, with the names of paths found, starting with the lowest one and closing with the highest one. In our example, the array will contain the following values:

folder[0] = "C:\folder_A\folder_B\folder_C\folder_D";
folder[1] = "C:\folder_A\folder_B\folder_C";
folder[2] = "C:\folder_A\folder_B";
folder[3] = "C:\folder_A";
folder[4] = "C:";

If we want to write a file named as "C:\folder_A\folder_B\folder_C\folder_D\test.txt", we can divide the specified path into file name test.txt and the structure of subfolders "C:\folder_A\folder_B\folder_C\folder_D" that contains this file. If the program fails creating a file on this path, first of all, it should be tried to create a folder of the lowest level, C:\folder_A\folder_B\folder_C\folder_D".

If the attempt to create this folder fails, too, most probably, the parent folder of "C:\folder_A\folder_B\folder_C" is absent. So we will create folders of higher and higher levels until we get a message about the successful completion of function CreateDirectoryA(). This is why we need a function that would fill out string array 'folder[]' with folder names in the ascending order. The very first zero index contains the folder of the lowest level, the root directory is in the latest array index.

Now we can assemble the function itself that creates all necessary in-between folders on the given path:

//+------------------------------------------------------------------+
//|  It creates all necessary folders and subfolders                 |
//+------------------------------------------------------------------+
bool CreateFullPath(string path)
   {
   bool res = false;
   if (StringLen(path)==0) return(false);
   Print("Create path=>",path);
//----
   string folders[];
   if (!ParsePath(folders, path)) return(false);
   Print("Total subfolders:", ArraySize(folders));
   
   int empty[];
   int i = 0;
   while (CreateDirectoryA(folders[i],empty)==0) i++;
   Print("Create folder:",folders[i]);
   i--;
   while (i>=0) 
      {
      CreateDirectoryA(folders[i],empty);
      Print("Created folder:",folders[i]);
      i--;
      }
   if (i<0) res = true;   
//----
   return(res);
   }

Now it remains only to make a small change in function WriteFile() considering the possibility to create a new folder.

//+------------------------------------------------------------------+
//|  write the buffer contents to the given path                     |
//+------------------------------------------------------------------+
void WriteFile (string path, string buffer) 
  {
    int count=StringLen (buffer); 
    int result;
    int handle=_lopen (path,OF_WRITE);
    if(handle<0) 
      {
        handle=_lcreat (path,0);
        if(handle<0) 
          {
            Print ("Error creating file ",path);
            if (!CreateFullPath(path))
               {
               Print("Failed creating folder:",path);
               return;
               }
            else handle=_lcreat (path,0);   
          }
        result=_lclose (handle);
        handle = -1;
     }
    if (handle < 0) handle=_lopen (path,OF_WRITE);               
    if(handle<0) 
      {
        Print("Error opening file ",path); 
        return;
      }
    result=_llseek (handle,0,0);          
    if(result<0) 
      {
        Print("Error placing the pointer"); 
        return;
      }
    result=_lwrite (handle,buffer,count); 
    if(result<0)  
        Print("Error writing to file ",path," ",count," bytes");
    result=_lclose (handle);              
    if(result<0)  
        Print("Error closing file ",path);
    return;        
  }

The logic of how the modified function works is given in figure below.


Please note that after the newly created file has been closed, we set the file descriptor variable 'handle' for a negative value.

        result=_lclose (handle);
        handle = -1;

This is done in order to check the value of 'handle' one line below and open the file for reading only if the first opening failed.

    if (handle < 0) handle=_lopen (path,OF_WRITE);

This will allow us to avoid situations where multiple files are opened by mistake and then left without being closed. In such cases, the operating system informs about exceeding the maximal permitted amount of open files and does n't allow us to open new files.

Let's modify function start() to check the new features:

//+------------------------------------------------------------------+
//| script program start function                                    |
//+------------------------------------------------------------------+
int start()
  {
//----
    int start = GetTickCount();
    string buffer=ReadFile("C:\\Text.txt");
 
    int middle = GetTickCount();
    int count=StringLen(buffer);
 
    Print("Bytes read:",count);
 
    WriteFile("C:\\folder_A\\folder_B\\folder_C\\folder_D\\Text2.txt",buffer);   
    int finish = GetTickCount();
    Print("File size is ",count," bytes. Reading:",(middle-start)," ms. Writing:",(finish-middle)," ms.");
//----
   return(0);
  }
//+------------------------------------------------------------------+

and launch script xFiles.mq4 for execution.



Conclusion

It is not very difficult to use the functions of WinAPI, but you should remember the reverse side of leaving the "sandbox":

Before launching an unknown executable application with the extension of ex4 (without MQL4 source code), which requires to import functions from external DLLs, please think of possible consequences.



Translated from Russian by MetaQuotes Software Corp.
Original article: https://www.mql5.com/ru/articles/1540

Attached files |
xFiles.mq4 (8.59 KB)
Last comments | Go to discussion (8)
MQL4 Comments
MQL4 Comments | 5 Jul 2009 at 11:27

Hello Rashid,

Your WinAPI file operations are very useful. Thank you so much. I incorporated your code into my Correlation indicator (which I will share when completed). However, your ReadFile() function seems to return a string from a text file dropping all the non-printable characters. In my application I NEED the non-printable characters "/r/n" to signal end of line. Because my indicator requires the user input (currency pair) be validated against the broker symbols. My set file does not use ";" delimiter; it is only one column of symbols in a plain text file. I actually tried to use the standard FileOpen() in FILE_CSV mode but it only allows the use of ';' as the delimiter (see attached).

Any help is much appreciated.

Scott

Lime
Lime | 26 Jan 2010 at 17:43

Hi Programmers!


How can I use the FileWrite function for update a whole file? So, I would like to delete the all characters of the previous content of the file.


There is one way - I think. I could delete the file before the write.

Do Hung
Do Hung | 11 Feb 2014 at 10:03

Hi,

int _lcreat (string path, int attrib);

This does not work on MT4 build 600, 604.

Please help! Thanks!

sythg
sythg | 14 Jul 2015 at 00:38

why using _lopen and friends that are still there for backward compatibility and msdn doesnt even document them anymore? you can use CreateFile from kernel32 or fopen from msvcrt..

also you should check if return value is -1 ( HFILE_ERROR ) which is the value if the function fails, checking if is less than 0 it's pretty bad since on error the value is -1 and not less than 0

31337 Investments Ltd.
Rafal Wieczorek | 17 Jun 2016 at 18:13
As mentioned before, using _lopen is broken since build 600, because DDL calls are now using Unicode strings, so either you have to use alternative calls (such as CreateFileW), or convert the ANSI string into Unicode.

Please update the article page.
Step on New Rails: Custom Indicators in MQL5 Step on New Rails: Custom Indicators in MQL5

I will not list all of the new possibilities and features of the new terminal and language. They are numerous, and some novelties are worth the discussion in a separate article. Also there is no code here, written with object-oriented programming, it is a too serous topic to be simply mentioned in a context as additional advantages for developers. In this article we will consider the indicators, their structure, drawing, types and their programming details, as compared to MQL4. I hope that this article will be useful both for beginners and experienced developers, maybe some of them will find something new.

Here Comes the New MetaTrader 5 and MQL5 Here Comes the New MetaTrader 5 and MQL5

This is just a brief review of MetaTrader 5. I can't describe all the system's new features for such a short time period - the testing started on 2009.09.09. This is a symbolical date, and I am sure it will be a lucky number. A few days have passed since I got the beta version of the MetaTrader 5 terminal and MQL5. I haven't managed to try all its features, but I am already impressed.

Using text files for storing input parameters of Expert Advisors, indicators and scripts Using text files for storing input parameters of Expert Advisors, indicators and scripts

The article describes the application of text files for storing dynamic objects, arrays and other variables used as properties of Expert Advisors, indicators and scripts. The files serve as a convenient addition to the functionality of standard tools offered by MQL languages.

How to create an indicator of non-standard charts for MetaTrader Market How to create an indicator of non-standard charts for MetaTrader Market

Through offline charts, programming in MQL4, and reasonable willingness, you can get a variety of chart types: "Point & Figure", "Renko", "Kagi", "Range bars", equivolume charts, etc. In this article, we will show how this can be achieved without using DLL, and therefore such "two-for-one" indicators can be published and purchased from the Market.