Grouped File Operations

MetaQuotes | 26 August, 2008

Introduction

It is not a problem to read or write one file. Even if you use WinAPI for this, as described in article File Operations via WinAPI. But what should we do if we don't know the exact name of the file, only its location in a certain folder and its extension? Of course, we can enter the necessary name manually, as a parameter, but what should we do if there are ten or more such files? We need a method of grouped processing the files of the same type in the given folder. This task can be efficiently solved using functions FindFirstFile(), FindNextFile() and FindClose() included in kernel32.dll.

Function FindFirstFile()

The function description is given in msdn at: http://msdn.microsoft.com/en-us/library/aa364418(VS.85).aspx.

HANDLE WINAPI FindFirstFile(
  __in   LPCTSTR lpFileName,
  __out  LPWIN32_FIND_DATA lpFindFileData
);

If follows from this description that the function returns the descriptor of the file found and meeting the search requirements. The search condition is specified in variable lpFileName that contains the path to search for the file and a possible name of the file. This function is convenient, since we can specify the search by mask, for example, find a file by mask "C:\folder\*.txt". The function will return the very first file found in the folder of "C:\folder" and having the extension of txt.

The returned result of the function is of the 'int' type in MQL4. To pass the input parameter, we can use the 'string' type. Now we have to sort out about what to pass to this function as the second parameter and how to process that parameter later. This function will be imported approximately like this:

#import "kernel32.dll"
int  FindFirstFileA(string path, .some second parameter);
#import

Here we can see the already known library of kernel32.dll. However, the function name is specified as FindFirstFileA(), not as FindFirstFile(). This comes from fact that many functions in this library have two versions: for working with strings in Unicode, the letter 'W' (FindFirstFileW) is added to the name, while for working with ANSI the letter 'A' (FindFirstFileA) is added.

Now we have to sort out the function's second parameter that is described as:

It means that it is the pointer to a certain structure of WIN32_FIND_DATA. In this case, structure is a certain area in the PC RAM. The pointer to this area (address) is passed to the function. We can allocate memory in MQL4 using a data array. The pointer is specified with the '&' character. We have only to get to know the size of the necessary memory in bytes to pass the pointer at. Below is the structure description.

typedef struct _WIN32_FIND_DATA {
  DWORD dwFileAttributes;
  FILETIME ftCreationTime;
  FILETIME ftLastAccessTime;
  FILETIME ftLastWriteTime;
  DWORD nFileSizeHigh;
  DWORD nFileSizeLow;
  DWORD dwReserved0;
  DWORD dwReserved1;
  TCHAR cFileName[MAX_PATH];
  TCHAR cAlternateFileName[14];
} WIN32_FIND_DATA, 

In MQL4, there are no types DWORD, TCHAR or FILETIME. DWORD is known to occupy 4 bytes, like int in MQL4, TCHAR has an internal representation of one byte. To calculate the total size of structure WIN32_FIND_DATA in bytes, we have only to sort out what FILETIME is.

typedef struct _FILETIME {
  DWORD dwLowDateTime;
  DWORD dwHighDateTime;
} FILETIME

FILETIME turns out to contain two DWORDs, which means that DWORD makes 8 bytes. Let's tabulate all this:

Now we can calculate the size of structure WIN32_FIND_DATA and visualize what and where can be found in it.

So the total size of the structure is: 4 + 8 + 8 + 8 + 4 + 4 + 4 +4 + 260 +14 = 318 bytes.

As you can see from the figure above, the file name starts with the 45th byte, the preceding 44 bytes containing various auxiliary information. It is necessary to pass to function FindFirstFile() some structure in MQL4 sized as 318 bytes, as the second parameter. It would be most convenient to use an array of the 'int' type, which would have a size not less than the required one. Divide 318 by 4 (since the internal representation of the 'int' type is 4 bytes), obtain 79.5, round it to the nearest larger integer, and see that we need an array of 80 elements.

The function import itself will now appear as follows:

#import "kernel32.dll"
int  FindFirstFileA(string path, int & answer[]);
#import

Here we are using the version of the function with the letter 'A' at the end of the name, FindFirstFileA(), for ANSI coding. The 'answer' array is passed by a link and serves for being filled with the structure of WIN32_FIND_DATA. An exemplary call:

   int win32_DATA[80];
   int handle = FindFirstFileA(TerminalPath() + "\experts\*.mq4",win32_DATA);

Functions FindNextFileA() and FindClose()

Function FindNextFileA() receives, as the first parameter, the 'handle' of the file preliminarily obtained by function FindFirstFileA() or by another, earlier call to function FindNextFileA(). The second parameter is the same. Function FindClose() just closes the search. This is why the full appearance of importing data of functions will be as follows:

#import "kernel32.dll"
int  FindFirstFileA(string path, int & answer[]);
bool FindNextFileA(int handle, int & answer[]);
bool FindClose(int handle);
#import

Now we have just to learn how to extract the name of a file written to array 'answer[]'.

Getting the File Name

The file name is contained by the array starting from the 45th through the 304th byte. The 'int' type contains 4 bytes, so each array element contains 4 characters, if we assume the array to be filled out with characters. So, in order to refer to the first character in the file name, we should skip 44/4 = 11 elements of the 'answer[]' array. The file name is inside the chain of 65 (260/4=65) array elements, starting from the 'answer[11]' (indexing starts with zero) and ending with 'answer[76]'.

Thus, we can get from array 'answer[]' the file name in blocks by 4 characters in each. The 'int' number represents a sequence of 32 bits that, in their turn, represent 4 blocks by 8 bits in each.

The youngest byte is to the right, the oldest byte is to the left. Bits are numbered in the ascending order, i.e., bits from the 1st one to the 8th one make the youngest byte. We can extract the necessary bytes using bitwise operations. To get the value of the youngest byte, we should fill with zero values all bits, from the 9th through the 32nd one. This is performed using logic operation AND.

int a = 234565;
int b = a & 0x000000FF;

Here 0x000000FF is the 32-bit integer that has zero values of all places, starting from the 9th one, while the places from 1 through 8 are equal to one. Thus, the obtained number b will contain only one (the youngest) byte of number a. We will turn the obtained byte (the character code) into a one-character string using function CharToStr().

Ok, we have obtained the first character. How should we obtain the next one? Very easily: We make a bitwise 8-bit shift to the right, and the second bit replaces the youngest bit. Then we will apply the already known operation of logic AND.

int a = 234565;
int b = (a >>8) & 0x000000FF;

As you can guess, the third byte will be obtained by shifting by 16 bits, while the oldest will be obtained by shifting by 24 bits. Thus, we can extract 4 characters from one element of the array of the 'int' type. Below is shown how the first 4 characters in the file name are obtained from the 'answer[]' array:

   string text="";
   
   int pos = 11;
   int curr = answer[pos];
      {
      text = text + CharToStr(curr & 0x000000FF)
         +CharToStr(curr >> 8 & 0x000000FF)
         +CharToStr(curr >> 16 & 0x000000FF)
         +CharToStr(curr >> 24 & 0x000000FF);
      }
    Print("text = ", text);

Let's create a separate function that returns a text string from the passed array named 'buffer'.

//+------------------------------------------------------------------+
//|  read text from buffer                                           |
//+------------------------------------------------------------------+ 
string bufferToString(int buffer[])
   {
   string text="";
   
   int pos = 10;
   for (int i=0; i<64; i++)
      {
      pos++;
      int curr = buffer[pos];
      text = text + CharToStr(curr & 0x000000FF)
         +CharToStr(curr >> 8 & 0x000000FF)
         +CharToStr(curr >> 16 & 0x000000FF)
         +CharToStr(curr >> 24 & 0x000000FF);
      }
   return (text);
   }

Thus, the problem of obtaining the file name from the structure is solved.

Getting the List of All Expert Advisors with Their Source Codes

A simple script demonstrates the features of the above functions:

//+------------------------------------------------------------------+
//|                                                CheckFindFile.mq4 |
//|                      Copyright © 2008, MetaQuotes Software Corp. |
//|                                       https://www.metaquotes.net/ |
//+------------------------------------------------------------------+
#property copyright "Copyright © 2008, MetaQuotes Software Corp."
#property link      "https://www.metaquotes.net/"
 
#property show_inputs
 
#import "kernel32.dll"
int  FindFirstFileA(string path, int& answer[]);
bool FindNextFileA(int handle, int& answer[]);
bool FindClose(int handle);
#import
 
 
//+------------------------------------------------------------------+
//| script program start function                                    |
//+------------------------------------------------------------------+
int start()
  {
//----
   int win32_DATA[79];
   
   int handle = FindFirstFileA(TerminalPath() + "\experts\*.mq4",win32_DATA);
   Print(bufferToString(win32_DATA));
   ArrayInitialize(win32_DATA,0);
 
   while (FindNextFileA(handle,win32_DATA))
      {
      Print(bufferToString(win32_DATA));
      ArrayInitialize(win32_DATA,0);
      }
 
   if (handle>0) FindClose(handle);
   
//----
   return(0);
  }
  
//+------------------------------------------------------------------+
//|  read text from buffer                                           |
//+------------------------------------------------------------------+ 
string bufferToString(int buffer[])
   {
   string text="";
   
   int pos = 10;
   for (int i=0; i<64; i++)
      {
      pos++;
      int curr = buffer[pos];
      text = text + CharToStr(curr & 0x000000FF)
         +CharToStr(curr >> 8 & 0x000000FF)
         +CharToStr(curr >> 16 & 0x000000FF)
         +CharToStr(curr >> 24 & 0x000000FF);
      }
   return (text);
   }  
//+------------------------------------------------------------------+

Here, after FindFirstFileA() and FindNextFileA() function calls, the array of win32_DATA (structure WIN32_FIND_DATA) is changed to an "empty" status, namely, all elements of the array are filled with zeros:

   ArrayInitialize(win32_DATA,0);

If we don't do this, it may happen that the "snatches" of the file name from the preceding call penetrate the process and we get some gobbledygook.

Example Realization: Creating Source Codes Backups

A simple example that demonstrates practical features of the above is the creation of source codes backups in a special directory. For this, we should combine the functions considered in this present article with those in article File Operations via WinAPI. We will then obtain simple script backup.mq4 given below. You can see the full code in the attached file. Here is only its start() function that uses all functions described in both articles:

//+------------------------------------------------------------------+
//| script program start function                                    |
//+------------------------------------------------------------------+
int start()
  {
//----
   string expert[1000]; // must be enough 
   string EAname="";    // EA name
   int EAcounter = 0;   // EAs counter   
   int win32_DATA[80];  
   
   int handle = FindFirstFileA(TerminalPath() + "\experts\*.mq4",win32_DATA);
   EAname = bufferToString(win32_DATA);
   expert[0] = EAname;
   ArrayInitialize(win32_DATA,0);
   
   int i=1;
   while (FindNextFileA(handle,win32_DATA))
      {
      EAname = bufferToString(win32_DATA);
      expert[i] = EAname;
      ArrayInitialize(win32_DATA,0);
      i++;
      if (i>=1000) ArrayResize(expert,2000); // now it will surely be enough
      }
 
   ArrayResize(expert, i);
   int size = i;
   
   if (handle>0) FindClose(handle);
 
   for (i = 0 ; i < size; i++) 
      {
      Print(i,":  ",expert[i]);   
      string backupPathName = backup_folder + "experts\\" + expert[i];
      string originalName = TerminalPath() + "\\experts\\" + expert[i];
      string buffer=ReadFile(originalName);
      WriteFile(backupPathName,buffer);   
      }
   if (size > 0 ) Print("There are ",size," files were copied to folder "+backup_folder+"experts\\");   
//----
   return(0);
  }
//+------------------------------------------------------------------+

Conclusion

It was shown how to perform operations with a group of files of the same type. You can read and process files using the above functions and realizing your own logic. You can make timely backups using an Expert Advisor or synchronize two sets of files in different folders, import quotes from some databases, and so on. It is recommended to control the use of DLLs and not to enable connecting libraries for third-parties' files having the extension of ex4. You must make sure that the program you are launching won't damage the data on your PC.