Download MetaTrader 5

How to Prepare MetaTrader 5 Quotes for Other Applications

2 January 2013, 10:01
Anatoli Kazharski
0
4 634

Contents

Introduction
1. Covered Topics
2. Data Format
3. Program's External Parameters
4. Checking Parameters Entered by a User
5. Global Variables
6. Information Panel
7. Application's Main Block
8. Creating Folders and Filing the Data
Conclusion


Introduction

Before I started studying MQL5, I tried many other applications for development of trading systems. I can't say that I wasted my time. Some of them contain a few useful tools allowing users to save time, deal with many issues, destroy some myths and quickly select some further direction for development without the knowledge of programming languages.

These applications need historical data. Due to the absence of some certain standard data format, they often had to be edited before they could be used (for example, in Excel) to comply with the format applicable to the necessary program. Even if you are able to figure out all necessary details, many things should still be done manually. Users can find different versions of scripts designed to copy the quotes from MetaTrader 4 to the necessary format. If there is such a demand, we can also develop the version of the script for MQL5.


1. Covered Topics

The article deals with the following topics:

  • Working with the symbol list in Market Watch window and the common symbol list on the server.
  • Checking available data depth and downloading missing amount if necessary with correct handling of various situations.
  • Displaying information about the requested data on the custom panel chart and the journal.
  • Preparing data for filing in a user-defined format.
  • Creating file directories.
  • Data filing.


2. Data Format

I will give an example of preparing the data meant to be used in NeuroShell DayTrader Professional (NSDT). I have tried NSDT versions 5 and 6 and found out that they have different requirements to the data format. NSDT version 5 date and time data should be in different columns. The first line in the file should have the following look:

"Date" "Time" "Open" "High" "Low" "Close" "Volume"

The header line in NSDT version 6 should have a different look to allow the application to accept a file. It means that date and time should be in the same column:

Date,Open,High,Low,Close,Volume

MetaTrader 5 allows users to save quotes in *.csv files. The data in a file look as follows:

Fig. 1. Data saved by MetaTrader 5 terminal

Fig. 1. Data saved by MetaTrader 5 terminal


However, we cannot just edit the header line as the date should have another format. For NSDT v.5:

dd.mm.yyyy,hh:mm,Open,High,Low,Close,Volume
For NSDT v.6:
dd/mm/yyyy hh:mm,Open,High,Low,Close,Volume

Drop-down lists will be used in the script external parameters where users are able to select the necessary format. Apart from selecting header and date formats, we will grant users the possibility to select the number of the symbols, data on which they want to write to files. To do this, we will prepare three versions:

  • Write the data on the current symbol only, on the chart of which (ONLY CURRENT SYMBOL) script has been launched.
  • Write the data on the symbols located in Market Watch window (MARKETWATCH SYMBOLS).
  • Write the data on all the symbols available on the server (ALL LIST SYMBOLS).

Let's enter the following code before the external parameters in the script code to create such lists:

//_________________________________
// HEADER_FORMATS_ENUMERATION
enum FORMAT_HEADERS
  {
   NSDT_5 = 0, // "Date" "Time" "Open" "High" "Low" "Close" "Volume"
   NSDT_6 = 1  // Date,Open,High,Low,Close,Volume
  };
//---
//___________________________
// ENUMERATION_OF_DATA_FORMATS
enum FORMAT_DATETIME
  {
   SEP_POINT1 = 0, // dd.mm.yyyy hh:mm
   SEP_POINT2 = 1, // dd.mm.yyyy, hh:mm
   SEP_SLASH1 = 2, // dd/mm/yyyy hh:mm
   SEP_SLASH2 = 3  // dd/mm/yyyy, hh:mm
  };
//---
//____________________________
// ENUMERATION_OF_FILING_MODES
enum CURRENT_MARKETWATCH
  {
   CURRENT          = 0, // ONLY CURRENT SYMBOLS
   MARKETWATCH      = 1, // MARKETWATCH SYMBOLS
   ALL_LIST_SYMBOLS = 2  // ALL LIST SYMBOLS
  };

More on enumerations can be found in MQL5 Reference.


3. Program's External Parameters

Now, we can create the entire list of all external parameters of the script:

//____________________________________________________________________
//+------------------------------------------------------------------+
//| EXTERNAL_PARAMETERS                                              |
//+------------------------------------------------------------------+
input datetime            start_date     = D'01.01.2011'; // Start Date
input datetime            end_date       = D'18.09.2012'; // End Date
input FORMAT_HEADERS      format_headers = NSDT_5;     // Format Headers
input FORMAT_DATETIME     format_date    = SEP_POINT2; // Format Datetime
input CURRENT_MARKETWATCH curr_mwatch    = CURRENT;    // Mode Write Symbols
input bool                clear_mwatch   = true;        // Clear Market Watch
input bool                show_progress  = true;        // Show Progress (%)

External parameters are used for the following purposes:

  • Users can specify a date interval using Start Date (start_date) and End Date (end_date) parameters.
  • Format Headers (format_headers) drop-down list allows users to choose a header format.
  • Format Datetime (format_date) drop-down list allows users to choose date and time formats.
  • Mode Write Symbols (curr_mwatch) drop-down list allows users to select the number of symbols for filing.
  • If Clear Market Watch (clear_mwatch) parameter is true, that allows users to delete all symbols from Market Watch window after the filing. This concerns only the symbols with the charts that are not active at the moment.
  • Show Progress (%) (show_progress) parameter displays the filing progress in data panel. The filing will be performed faster if this parameter is disabled.

Below is how the external parameters will look during the launch:

Fig. 2. Application's external parameters

Fig. 2. Application's external parameters


4. Checking Parameters Entered by a User

Let's create the function for checking the parameters entered by users before the base code. For example, the start date in Start Date parameter should be earlier than the one in End Date. Format of headers should match date and time formats. If a user has made some errors when setting the parameters, the following warning message is displayed and the program is stopped.

Sample warning message:

Fig. 3. Warning of incorrectly specified values

Fig. 3. Sample error message on incorrectly specified parameters


ValidationParameters() function:

//____________________________________________________________________
//+------------------------------------------------------------------+
//| CHECKING_CORRECTNESS_OF_PARAMETERS                               |
//+------------------------------------------------------------------+
bool ValidationParameters()
  {
   if(start_date>=end_date)
     {
      MessageBox("The start date should be earlier than the ending one!\n\n"
                 "Application cannot continue. Please retry.",
                 //---
                 "Parameter error!",MB_ICONERROR);
      //---
      return(true);
     }
//---
   if(format_headers==NSDT_5 && 
      (format_date==SEP_POINT1 || format_date==SEP_SLASH1))
     {
      MessageBox("For the headers of the following format:\n\n"
                 "\"Date\" ""\"Time\" ""\"Open\" ""\"High\" ""\"Low\" ""\"Close\" ""\"Volume\"\n\n"
                 "Date/time format can be selected out of two versions:\n\n"
                 "dd.mm.yyyy, hh:mm\n"
                 "dd/mm/yyyy, hh:mm\n\n"
                 "Application cannot continue. Please retry.",
                 //---
                 "Header and date/time formats do not match!",MB_ICONERROR);
      //---
      return(true);
     }
//---
   if(format_headers==NSDT_6 && 
      (format_date==SEP_POINT2 || format_date==SEP_SLASH2))
     {
      MessageBox("For the headers of the following format:\n\n"
                 "Date,Open,High,Low,Close,Volume\n\n"
                 "Date/time format can be selected out of two versions:\n\n"
                 "dd.mm.yyyy hh:mm\n"
                 "dd/mm/yyyy hh:mm\n\n"
                 "Application cannot continue. Please retry.",
                 //---
                 "Header and date/time formats do not match!",MB_ICONERROR);
      //---
      return(true);
     }
//---
   return(false);
  }


5. Global Variables

Next, we should determine all global variables and arrays that will be used in the script:

//____________________________________________________________________
//+------------------------------------------------------------------+
//| GLOBAL_VARIABLES_AND_ARRAYS                                      |
//+------------------------------------------------------------------+
MqlRates rates[]; // Array for copying data
//---
string symbols[]; // Symbol array
//---
// Array of graphic object names
string arr_nmobj[22]=
  {
   "fon","hd01",
   "nm01","nm02","nm03","nm04","nm05","nm06","nm07","nm08","nm09","nm10",
   "nm11","nm12","nm13","nm14","nm15","nm16","nm17","nm18","nm19","nm20"
  };
//---
// Array of displayed text containing graphic objects
string arr_txtobj[21];
//---
string path="";         // File path
int cnt_symb=0;         // Number of symbols
int sz_arr_symb=0;      // Symbol array size
int bars=0;             // Number of bars according to the specified TF
int copied_bars=0;      // Number of bars copied for writing
double pgs_pcnt=0;      // Writing progress
int hFl=INVALID_HANDLE;  // File handle
//---
string   // Variables for data formatting
sdt="",  // Date line
dd="",   // Day
mm="",   // Month
yyyy="", // Year
tm="",   // Time
sep="";  // Separator
//---
int max_bars=0; // Maximum number of bars in the terminal settings
//---
datetime
first_date=0,        // First available data in a specified period
first_termnl_date=0, // First available data in the terminal's database
first_server_date=0, // First available data in the server's database
check_start_date=0;  // Checked correct date value


6. Information Panel

Now, we should deal with the elements that are to be displayed at the information panel. Three types of graphic objects can be used as a background:

  • The most simple and obvious one – "Rectangle label" (OBJ_RECTANGLE_LABEL).
  • Those who want their interface to have a unique look can use "Bitmap" object (OBJ_BITMAP).
  • "Edit" object (OBJ_EDIT) can also be used as a background. Set "read only" property to remove the possibility to enter a text. There is also another advantage of using "Edit" object. If you have created an information panel in an Expert Advisor and you want it to have the same look during the tests in visualization mode, the last type is the only method to achieve that so far. Neither OBJ_RECTANGLE_LABEL, nor OBJ_BITMAP are displayed during the tests in visualization mode.

Though only a script instead of an Expert Advisor is developed in our case, the background with OBJ_EDIT object will be made as an example. The result is shown on the figure below:

Fig. 4. Information panel

Fig. 4. Information panel


Let's describe all the data shown on the panel:

  • Symbol (current/total) – symbol, the data on which is downloaded/copied/written at the moment. The left number in brackets shows the current symbol number. The right number shows common number of symbols the script will work with.
  • Path Symbol – symbol path or category it belongs to. If you bring up the context menu by right-clicking in Market Watch window and select "Symbols…", the window with the list of all symbols will appear. You can find more about that in the terminal's User Guide.
  • Timeframe – period (timeframe). The timeframe, at which the script will be launched, is to be used.
  • Input Start Date – data start date specified by a user in the script parameters.
  • First Date (H1) – the first available data date (bar) of the current timeframe.
  • First Terminal Date (M1) – the first available date of M1 timeframe in already existing terminal data.
  • First Server Date (M1) – the first available date of M1 timeframe on the server.
  • Max. Bars In Options Terminal – maximum number of bars to be displayed on the chart specified in the terminal settings.
  • Copied Bars – number of copied bars for writing.
  • Progress Value Current Symbol – percentage value of written data of the current symbol.

Below is the code of such information panel:

//____________________________________________________________________
//+------------------------------------------------------------------+
//| INFORMATION_PANEL                                                |
//|------------------------------------------------------------------+
void InfoTable(int s)
  {
   int fnt_sz=8;            // Font size
   string fnt="Calibri";     // Header font
   color clr=clrWhiteSmoke;  // Color
//---
   int xH=300;
   int height_pnl=0;
   int yV1=1,yV2=12,xV1=165,xV2=335,xV3=1;
//---
   string sf="",stf="",ssf="";
   bool flg_sf=false,flg_stf=false,flg_ssf=false;
//---
   if(show_progress) { height_pnl=138; } else { height_pnl=126; }
//---
   flg_sf=SeriesInfoInteger(symbols[s],_Period,SERIES_FIRSTDATE,first_date);
   flg_stf=SeriesInfoInteger(symbols[s],PERIOD_M1,SERIES_TERMINAL_FIRSTDATE,first_termnl_date);
   flg_ssf=SeriesInfoInteger(symbols[s],PERIOD_M1,SERIES_SERVER_FIRSTDATE,first_server_date);
//---
   if(flg_sf) { sf=TSdm(first_date); } else { sf="?"; }
   if(flg_stf) { stf=TSdm(first_termnl_date); } else { stf="?"; }
   if(flg_ssf) { ssf=TSdm(first_server_date); } else { ssf="?"; }
//---
   if(cnt_symb==0) { cnt_symb=1; }
//---
   int anchor1=ANCHOR_LEFT_UPPER,anchor2=ANCHOR_RIGHT_UPPER,corner=CORNER_LEFT_UPPER;
//---
   string path_symbol=SymbolInfoString(symbols[s],SYMBOL_PATH);
   path_symbol=StringSubstr(path_symbol,0,StringLen(path_symbol)-StringLen(symbols[s]));
//---
   arr_txtobj[0]="INFO TABLE";
   arr_txtobj[1]="Symbol (current / total) : ";
   arr_txtobj[2]=""+symbols[s]+" ("+IS(s+1)+"/"+IS(cnt_symb)+")";
   arr_txtobj[3]="Path Symbol : ";
   arr_txtobj[4]=path_symbol;
   arr_txtobj[5]="Timeframe : ";
   arr_txtobj[6]=gStrTF(_Period);
   arr_txtobj[7]="Input Start Date : ";
   arr_txtobj[8]=TSdm(start_date);
   arr_txtobj[9]="First Date (H1) : ";
   arr_txtobj[10]=sf;
   arr_txtobj[11]="First Terminal Date (M1) : ";
   arr_txtobj[12]=stf;
   arr_txtobj[13]="First Server Date (M1) : ";
   arr_txtobj[14]=ssf;
   arr_txtobj[15]="Max. Bars In Options Terminal : ";
   arr_txtobj[16]=IS(max_bars);
   arr_txtobj[17]="Copied Bars : ";
   arr_txtobj[18]=IS(copied_bars);
   arr_txtobj[19]="Progress Value Current Symbol : ";
   arr_txtobj[20]=DS(pgs_pcnt,2)+"%";
//---
   Create_Edit(0,0,arr_nmobj[0],"",corner,fnt,fnt_sz,clrDimGray,clrDimGray,345,height_pnl,xV3,yV1,2,C'15,15,15');
//---
   Create_Edit(0,0,arr_nmobj[1],arr_txtobj[0],corner,fnt,8,clrWhite,C'64,0,0',345,12,xV3,yV1,2,clrFireBrick);
//---
   Create_Label(0,arr_nmobj[2],arr_txtobj[1],anchor2,corner,fnt,fnt_sz,clr,xV1,yV1+yV2,0);
   Create_Label(0,arr_nmobj[3],arr_txtobj[2],anchor2,corner,fnt,fnt_sz,clr,xV2,yV1+yV2,0);
//---
   Create_Label(0,arr_nmobj[4],arr_txtobj[3],anchor2,corner,fnt,fnt_sz,clr,xV1,yV1+yV2*2,0);
   Create_Label(0,arr_nmobj[5],arr_txtobj[4],anchor2,corner,fnt,fnt_sz,clr,xV2,yV1+yV2*2,0);
//---
   Create_Label(0,arr_nmobj[6],arr_txtobj[5],anchor2,corner,fnt,fnt_sz,clr,xV1,yV1+yV2*3,0);
   Create_Label(0,arr_nmobj[7],arr_txtobj[6],anchor2,corner,fnt,fnt_sz,clr,xV2,yV1+yV2*3,0);
//---
   Create_Label(0,arr_nmobj[8],arr_txtobj[7],anchor2,corner,fnt,fnt_sz,clr,xV1,yV1+yV2*4,0);
   Create_Label(0,arr_nmobj[9],arr_txtobj[8],anchor2,corner,fnt,fnt_sz,clr,xV2,yV1+yV2*4,0);
//---
   Create_Label(0,arr_nmobj[10],arr_txtobj[9],anchor2,corner,fnt,fnt_sz,clr,xV1,yV1+yV2*5,0);
   Create_Label(0,arr_nmobj[11],arr_txtobj[10],anchor2,corner,fnt,fnt_sz,clr,xV2,yV1+yV2*5,0);
//---
   Create_Label(0,arr_nmobj[12],arr_txtobj[11],anchor2,corner,fnt,fnt_sz,clr,xV1,yV1+yV2*6,0);
   Create_Label(0,arr_nmobj[13],arr_txtobj[12],anchor2,corner,fnt,fnt_sz,clr,xV2,yV1+yV2*6,0);
//---
   Create_Label(0,arr_nmobj[14],arr_txtobj[13],anchor2,corner,fnt,fnt_sz,clr,xV1,yV1+yV2*7,0);
   Create_Label(0,arr_nmobj[15],arr_txtobj[14],anchor2,corner,fnt,fnt_sz,clr,xV2,yV1+yV2*7,0);
//---
   Create_Label(0,arr_nmobj[16],arr_txtobj[15],anchor2,corner,fnt,fnt_sz,clr,xV1,yV1+yV2*8,0);
   Create_Label(0,arr_nmobj[17],arr_txtobj[16],anchor2,corner,fnt,fnt_sz,clr,xV2,yV1+yV2*8,0);
//---
   Create_Label(0,arr_nmobj[18],arr_txtobj[17],anchor2,corner,fnt,fnt_sz,clr,xV1,yV1+yV2*9,0);
   Create_Label(0,arr_nmobj[19],arr_txtobj[18],anchor2,corner,fnt,fnt_sz,clr,xV2,yV1+yV2*9,0);
//---
   if(show_progress)
     {
      Create_Label(0,arr_nmobj[20],arr_txtobj[19],anchor2,corner,fnt,fnt_sz,clr,xV1,yV1+yV2*10,0);
      Create_Label(0,arr_nmobj[21],arr_txtobj[20],anchor2,corner,fnt,fnt_sz,clr,xV2,yV1+yV2*10,0);
     }
  }
//____________________________________________________________________
//+------------------------------------------------------------------+
//| CREATING_LABEL_OBJECT                                            |
//+------------------------------------------------------------------+
void Create_Label(long   chrt_id,   // chart id
                  string lable_nm,  // object name
                  string rename,    // displayed name
                  long   anchor,    // anchor point
                  long   corner,    // attachment corner
                  string font_bsc,  // font
                  int    font_size, // font size
                  color  font_clr,  // font color
                  int    x_dist,    // X scale coordinate
                  int    y_dist,    // Y scale coordinate
                  long   zorder)    // priority
  {
   if(ObjectCreate(chrt_id,lable_nm,OBJ_LABEL,0,0,0)) // creating object
     {
      ObjectSetString(chrt_id,lable_nm,OBJPROP_TEXT,rename);          // set name
      ObjectSetString(chrt_id,lable_nm,OBJPROP_FONT,font_bsc);        // set font
      ObjectSetInteger(chrt_id,lable_nm,OBJPROP_COLOR,font_clr);      // set font color
      ObjectSetInteger(chrt_id,lable_nm,OBJPROP_ANCHOR,anchor);       // set anchor point
      ObjectSetInteger(chrt_id,lable_nm,OBJPROP_CORNER,corner);       // set attachment corner
      ObjectSetInteger(chrt_id,lable_nm,OBJPROP_FONTSIZE,font_size);  // set font size
      ObjectSetInteger(chrt_id,lable_nm,OBJPROP_XDISTANCE,x_dist);    // set X coordinates
      ObjectSetInteger(chrt_id,lable_nm,OBJPROP_YDISTANCE,y_dist);    // set Y coordinates
      ObjectSetInteger(chrt_id,lable_nm,OBJPROP_SELECTABLE,false);     // unable to highlight the object, if FALSE
      ObjectSetInteger(chrt_id,lable_nm,OBJPROP_ZORDER,zorder);       // Higher/lower priority
      ObjectSetString(chrt_id,lable_nm,OBJPROP_TOOLTIP,"\n");         // no tooltip, if "\n"
     }
  }
//____________________________________________________________________
//+------------------------------------------------------------------+
//| CREATING_EDIT_OBJECT                                             |
//+------------------------------------------------------------------+
void Create_Edit(long   chrt_id,       // chart id
                 int    nmb_win,       // window (subwindow) index
                 string lable_nm,      // object name
                 string text,          // displayed text
                 long   corner,        // attachment corner
                 string font_bsc,      // font
                 int    font_size,     // font size
                 color  font_clr,      // font color
                 color  font_clr_brd,  // font color
                 int    xsize,         // width
                 int    ysize,         // height
                 int    x_dist,        // X scale coordinate
                 int    y_dist,        // Y scale coordinate
                 long   zorder,        // priority
                 color  clr)           // background color
  {
   if(ObjectCreate(chrt_id,lable_nm,OBJ_EDIT,nmb_win,0,0)) // creating object
     {
      ObjectSetString(chrt_id,lable_nm,OBJPROP_TEXT,text);                     // set name
      ObjectSetInteger(chrt_id,lable_nm,OBJPROP_CORNER,corner);                // set attachment corner
      ObjectSetString(chrt_id,lable_nm,OBJPROP_FONT,font_bsc);                 // set font
      ObjectSetInteger(chrt_id,lable_nm,OBJPROP_ALIGN,ALIGN_CENTER);         // center alignment
      ObjectSetInteger(chrt_id,lable_nm,OBJPROP_FONTSIZE,font_size);           // set font size
      ObjectSetInteger(chrt_id,lable_nm,OBJPROP_COLOR,font_clr);               // font color
      ObjectSetInteger(chrt_id,lable_nm,OBJPROP_BORDER_COLOR,font_clr_brd);    // background color
      ObjectSetInteger(chrt_id,lable_nm,OBJPROP_BGCOLOR,clr);                  // background color
      ObjectSetInteger(chrt_id,lable_nm,OBJPROP_XSIZE,xsize);                  // width
      ObjectSetInteger(chrt_id,lable_nm,OBJPROP_YSIZE,ysize);                  // height
      ObjectSetInteger(chrt_id,lable_nm,OBJPROP_XDISTANCE,x_dist);             // set X coordinate
      ObjectSetInteger(chrt_id,lable_nm,OBJPROP_YDISTANCE,y_dist);             // set Y coordinate
      ObjectSetInteger(chrt_id,lable_nm,OBJPROP_SELECTABLE,false);              // unable to highlight the object, if FALSE
      ObjectSetInteger(chrt_id,lable_nm,OBJPROP_ZORDER,zorder);                // Higher/lower priority
      ObjectSetInteger(chrt_id,lable_nm,OBJPROP_READONLY,true);                // Read only
      ObjectSetString(chrt_id,lable_nm,OBJPROP_TOOLTIP,"\n");                  // no tooltip if "\n"
     }
  }

After the script operation is complete or the script is deleted by a user ahead of time, all graphic objects created by the script should be deleted. The following functions will be used for that:

//____________________________________________________________________
//+------------------------------------------------------------------+
//| DELETE_ALL_GRAPHICAL_OBJECTS_CREATED_BY_THE_SCRIPT               |
//+------------------------------------------------------------------+
void DelAllScriptObjects()
  {
// Receive the size of graphical object names array
   int sz_arr1=ArraySize(arr_nmobj);
//---
// Delete all objects
   for(int i=0; i<sz_arr1; i++)
     { DelObjbyName(arr_nmobj[i]);  }
  }
//____________________________________________________________________
//+------------------------------------------------------------------+
//| DELETE_OBJECTS_BY_NAME                                           |
//+------------------------------------------------------------------+
int DelObjbyName(string Name)
  {
   int nm_obj=0;
   bool res=false;
//---
   nm_obj=ObjectFind(ChartID(),Name);
//---
   if(nm_obj>=0)
     {
      res=ObjectDelete(ChartID(),Name);
      //---
      if(!res) { Print("Object deletion error: - "+ErrorDesc(Error())+""); return(false); }
     }
//---
   return(res);
  }


7. Application's Main Block

The scripts' main function is OnStart(). This is the function used for calling all other functions for execution. The details of the program's operation are shown in the code:

//____________________________________________________________________
//+------------------------------------------------------------------+
//| SCRIPT >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> |
//+------------------------------------------------------------------+
void OnStart()
  {
// If user-defined parameters are incorrect,
// error message is shown and the program is closed
   if(ValidationParameters()) { return; }
//---
   max_bars=TerminalInfoInteger(TERMINAL_MAXBARS); // Receive available number of bars in the window
//---
   GetSymbolsToArray();           // Filling symbol array with names
   sz_arr_symb=ArraySize(symbols); // Receive symbol array size
//---
   SetSeparateForFormatDate();    // Set a separator for date format
//---
// Revise all symbols and write their data to file
   for(int s=0; s<=sz_arr_symb-1; s++)
     {
      copied_bars=0; // Reset copied bars variable to zero for writing
      pgs_pcnt=0.0;  // Reset variable of the symbol data writing progress 
      //---
      InfoTable(s); ChartRedraw();
      //---
      // Receive current symbol data
      int res=GetDataCurrentSymbol(s);
      //---
      if(res==0) { BC } // If zero, break the loop or start the next iteration
      //---
      if(res==2)        // Program operation interrupted by user
        {
         DelAllScriptObjects(); // Deleted objects created by the script from the chart
         //---
         Print("------\nUser deleted the script!"); break;
        }
      //---
      // Receive the path for creating the file and create directories for them
      // If the string is empty, break the loop or start the next iteration
      if((path=CheckCreateGetPath(s))=="") { BC }
      //---
      WriteDataToFile(s); // Write data to file
     }
//---
// Delete symbols from Market Watch window if necessary
   DelSymbolsFromMarketWatch();
//---
// Delete objects created by the script from the chart
   Sleep(1000); DelAllScriptObjects();
  }

Let's examine the functions where the key activities take place.

The symbol array (symbols[]) is filled with symbol names in GetSymbolsToArray() function. The array size, as well as the number of symbols in it depend on the variant chosen by a user in Mode Write Symbols (curr_mwatch) parameter.

If a user should have data only from one symbol, the array size is equal to 1.

ArrayResize(symbols,1); // Set the array size to be equal to 1
symbols[0]=_Symbol;     // Specify the current symbol's name

If a user wants to receive the data on all symbols from Market Watch window or all available symbols, the array size will be defined by the following function:

int SymbolsTotal(
   bool selected   // true – only MarketWatch symbols
);

To avoid creation of two blocks for two variants with almost identical code, we will make MWatchOrAllList() pointer function, which will return true or false. This value defines where the symbol list should be taken from - only from Market Watch window (true) or from the common list of available symbols (false).

//____________________________________________________________________
//+------------------------------------------------------------------+
//| POINTER_TO_MARKET_WATCH_WINDOW_OR_TO_COMMON_LIST                 |
//+------------------------------------------------------------------+
bool MWatchOrAllList()
  {
   if(curr_mwatch==MARKETWATCH) { return(true); }
   if(curr_mwatch==ALL_LIST_SYMBOLS) { return(false); }
//---
   return(true);
  }

After obtaining the number of symbols in the loop, we should go through the entire list and put the symbol name into the array at each iteration increasing the array size by one. The symbol name, in its turn, is obtained by the index number using SymbolName() function.

int SymbolName(
   int pos,        // list index number
   bool selected   // true – only MarketWatch symbols
);

MWatchOrAllList() pointer function is also used in SymbolName() function to select the symbol list. GetSymbolsToArray() function complete code:

//____________________________________________________________________
//+------------------------------------------------------------------+
//| FILLING_SYMBOL_ARRAY_WITH_NAMES                                  |
//+------------------------------------------------------------------+
void GetSymbolsToArray()
  {
// If only the current symbol data is required
   if(curr_mwatch==CURRENT)
     { ArrayResize(symbols,1); symbols[0]=_Symbol; }
//---
// If data on all symbols  from Market Watch window or
// or the entire symbol list is required
   if(curr_mwatch==MARKETWATCH || curr_mwatch==ALL_LIST_SYMBOLS)
     {
      // Receive the number of symbols in Market Watch window
      cnt_symb=SymbolsTotal(MWatchOrAllList());
      //---
      for(int i=0; i<=cnt_symb-1; i++)
        {
         string nm_symb="";
         //---
         ArrayResize(symbols,i+1); // Increase the array size by one once again
         //---
         // Receive a name of a symbol from Market Watch window
         nm_symb=SymbolName(i,MWatchOrAllList());
         symbols[i]=nm_symb; // Put the symbol name into the array
        }
     }
  }

SetSeparateForFormatDate() function is very simple. it is used to define what kind of separator will be used in the date depending on the user's choice in the drop-down list of Format Date (format_date) parameter.

//____________________________________________________________________
//+------------------------------------------------------------------+
//| DEFINING_SEPARATOR_FOR_DATE_FORMAT                               |
//+------------------------------------------------------------------+
void SetSeparateForFormatDate()
  {
   switch(format_date)
     {
      case SEP_POINT1 : case SEP_POINT2 : sep="."; break; // Full point as a separator
      case SEP_SLASH1 : case SEP_SLASH2 : sep="/"; break; // Slash as a separator
     }
  }

The basic loop with various checks comes next. If all the checks are successful, the data is written to the file. Otherwise, the loop is broken, all objects are deleted from the chart and the script is removed (in case of one symbol) or the next iteration starts (in case of more than one symbol). Each symbol of symbols[] array is consistently called in the loop. The index number is sent to each function of the loop. Thus, the accurate sequence in all functions is preserved.

The data on the current symbol in the loop is received at each iteration at the very beginning of the loop body. GetDataCurrentSymbol() function is used for that. Let's see what happens in this function.

Data availability is checked using CheckLoadHistory() function before copying symbol data to rate[] array. This function is provided by the developers as an example. Its initial version can be found in MQL5 Reference. I have only made slight corrections for using in this script. The Reference contains a detailed description (it would be a good idea to study it as well), therefore, I am not going to display my version here as it is almost the same. Besides, it can be found in the code with detailed comments.

The only thing that can be mentioned now is that CheckLoadHistory() function returns the error or successful execution code, according to which the appropriate message from switch operator block is saved in the journal. According to the received code, GetDataCurrentSymbol() function either goes on with its operation or returns its code.

If all goes well, history data is copied using CopyRates() function. The array size is saved in the global variable. Then, the exit from the function accompanied by returning of code 1 is performed. If something goes wrong, the function stops its operation in switch operator and returns code 0 or 2.

//____________________________________________________________________
//+------------------------------------------------------------------+
//| RECEIVE_SYMBOL_DATA                                              |
//+------------------------------------------------------------------+
int GetDataCurrentSymbol(int s)
  {
   Print("------\n№"+IS(s+1)+" >>>"); // Save a symbol number in the journal
//---
// Check and download the necessary amount of requested data
   int res=CheckLoadHistory(s,_Period);
//---
   InfoTable(s); ChartRedraw(); // Update the data in the data table
//---
   switch(res)
     {
      case -1 : Print("Unknown symbol "+symbols[s]+" (code: -1)!");                        return(0);
      case -2 :
         Print("Number of requested bars exceeds the maximum number that can be displayed on a chart (code: -2)!...\n"
               "...The available amount of data will be used for writing.");                break;
      //---
      case -3 : Print("Execution interrupted by user (code: -3)!");                         return(2);
      case -4 : Print("Download failed (code: -4)!");                                      return(0);
      case  0 : Print("All symbol data downloaded (code: 0).");                             break;
      case  1 : Print("Time series data is sufficient (code: 1).");                          break;
      case  2 : Print("Time series created based on existing terminal data (code: 2).");      break;
      //---
      default : Print("Execution result is not defined!");
     }
//---
// Copy data to the array
   if(CopyRates(symbols[s],_Period,check_start_date,end_date,rates)<=0)
     { Print("Error when copying symbol data "+symbols[s]+" - ",ErrorDesc(Error())+""); return(0); }
   else
     {
      copied_bars=ArraySize(rates); // Receive array size
      //---
      Print("Symbol: ",symbols[s],"; Timeframe: ",gStrTF(_Period),"; Copied bars: ",copied_bars);
     }
//---
   return(1); // Return 1, if all is well
  }

After that, the program is located in the main loop's body of OnStart() function again. The code is assigned by res local variable and the check is performed according to its value. Zero value stands for an error. It means that the data of the current symbol in the loop cannot be written. Error explanation has been saved in the journal and the decision is made if the loop should be broken (break) or the next iteration should be started (continue).

if(res==0) { BC } // If zero, the loop is interrupted or the next iteration starts

The code line above shows that this selection is performed by some mysterious BC characters. This is a macro expansion. More information about it can be found in MQL5 Reference. The only thing that should be mentioned here is that the entire expressions (typed in one line) can be pasted in a short entry as shown in the above example (BC). In some cases, this method can be even more convenient and compact than a function. In the current case, this looks as follows:

// Macro expansion with further action selection
#define BC if(curr_mwatch==CURRENT) { break; } if(curr_mwatch==MARKETWATCH || curr_mwatch==ALL_LIST_SYMBOLS) { continue; }

Below are other examples of macro expansions used in this script:

#define nmf __FUNCTION__+": " // Macro expansion of the function name before sending the message to the journal
//---
#define TRM_DP TerminalInfoString(TERMINAL_DATA_PATH) // Folder for storing the terminal data

If GetDataCurrentSymbol() returns 2, the program has been deleted by a user. MQL5 has IsStopped() function to identify this event. This function can be very helpful in loops to stop the program's operation correctly and in time. If the function returns true, then there are about three seconds to perform all actions before the program is forcibly deleted. In our case, all graphical objects are removed and the message is sent to the journal:

if(res==2) // Program execution interrupted by user
   {
    DelAllScriptObjects(); // Delete all objects created by the script from the chart
    //---
    Print("------\nUser deleted the script!"); break;
   }


8. Creating Folders and Filing the Data

CheckCreateGetPath() function checks the presence of the root data folder. Let's call it DATA_OHLC and place it to C:\Metatrader 5\MQL5\Files. It will contain folders with the symbol names. The files for writing the data will be created there.

If the root folder or the folder for the current symbol in the loop does not exist, the function creates it. If all goes well, the function returns a string containing the path for creating a file. The function returns an empty string in case of an error or an attempt to delete the program from the chart performed by a user.

The code below contains detailed comments making it easy to understand:

//____________________________________________________________________
//+------------------------------------------------------------------+
//| CHECK_DIRECTORY_AND_CREATE_NECESSARY_DATA_FOLDERS                |
//+------------------------------------------------------------------+
string CheckCreateGetPath(int s)
  {
   int i=1;
   long search=-1;
   string ffname="",lpath="";
   string file="*.csv",folder="*";
   string
   root="DATA_OHLC\\",         // Root data folder
   fSmb=symbols[s]+"\\",     // Symbol name
   fTF=gStrTF(_Period)+"\\"; // Symbol time frame
//---
   bool flgROOT=false,flgSYMBOL=false;
//---
//+------------------------------------------------------------------+
//| SEARCHING_FOR_DATA_OHLC_ROOT_FOLDER                              |
//+------------------------------------------------------------------+
   lpath=folder;
   search=FileFindFirst(lpath,ffname); // Set search handle in Metatrader 5\MQL5\Files
//---
   Print("Directory: ",TRM_DP+"\\MQL5\\Files\\");
//---
// Set the flag if the first folder is a root one
   if(ffname==root)
     { flgROOT=true; Print("Root folder "+root+" present"); }
//---
   if(search!=INVALID_HANDLE) // If search handle received
     {
      if(!flgROOT) // If the first folder is not a root one
        {
         // Sort out all files searching for the root folder
         while(FileFindNext(search,ffname))
           {
            if(IsStopped()) // Execution interrupted by user
              {
               // Delete objects created by the script from the chart
               DelAllScriptObjects();
               //---
               Print("------\nUser deleted the script!"); return("");
              }
            //---
            if(ffname==root) // Set the flag if found
              { flgROOT=true; Print("Root folder "+root+" present"); break; }
           }
        }
      //---
      FileFindClose(search); search=-1; // Close root folder search handle
     }
   else { Print("Error when receiving the search handle or directory "+TRM_DP+" is empty: ",ErrorDesc(Error())); }
//---
//+------------------------------------------------------------------+
//| SEARCHING_SYMBOL_FOLDER                                          |
//+------------------------------------------------------------------+
   lpath=root+folder;
//---
// Set search handle in the root folder ..\Files\DATA OHLC\
   search=FileFindFirst(lpath,ffname);
//---
// Set the flag if the first folder of the current symbol
   if(ffname==fSmb) { flgSYMBOL=true; Print("Symbol folder "+fSmb+" present"); }
//---
   if(search!=INVALID_HANDLE) // If search handle is received
     {
      if(!flgSYMBOL) // If the first folder is not of the current symbol
        {
         // Sort out all the files in the root folder searching the symbol folder
         while(FileFindNext(search,ffname))
           {
            if(IsStopped()) // Execution interrupted by user
              {
               // Delete objects created by the script from the chart
               DelAllScriptObjects();
               //---
               Print("------\nUser deleted the script!"); return("");
              }
            //---
            if(ffname==fSmb) // Set the flag if found
              { flgSYMBOL=true; Print("Symbol folder"+fSmb+" present"); break; }
           }
        }
      //---
      FileFindClose(search); search=-1; // Close symbol folder search handle
     }
   else { Print("Error when receiving search handle or the directory "+path+" is empty"); }
//---
//+------------------------------------------------------------------+
//| CREATE_NECESSARY_DIRECTORIES_ACCORDING_TO_CHECK_RESULTS          |
//+------------------------------------------------------------------+
   if(!flgROOT) // If there is no DATA_OHLC... root folder
     {
      if(FolderCreate("DATA_OHLC")) // ...we should create it
        { Print("..\DATA_OHLC\\ root folder created"); }
      else
        { Print("Error when creating DATA_OHLC: root folder",ErrorDesc(Error())); return(""); }
     }
//---
   if(!flgSYMBOL) // If there is no folder of the symbol, the values of which should be received...
     {
      if(FolderCreate(root+symbols[s])) // ...we should create it
        {
         Print("..\DATA_OHLC\\" symbol folder created+fSmb+"");
         //---
         return(root+symbols[s]+"\\"); // Return the path for creating the file for writing
        }
      else
        { Print("Error when creating ..\DATA_OHLC\\ symbol folder"+fSmb+"\: ",ErrorDesc(Error())); return(""); }
     }
//---
   if(flgROOT && flgSYMBOL)
     {
      return(root+symbols[s]+"\\"); // Return the path for creating the file for writing
     }
//---
   return("");
  }

If CheckCreateGetPath() function returns an empty line, the loop is interrupted or the next iteration starts using the already familiar macro expansion (BC):

// Receive the path for creating a file and create directories for them
// If the line is empty, the loop is interrupted or the next iteration starts
if((path=CheckCreateGetPath(s))=="") { BC }

If you have reached this stage, it means that the data has been successfully copied and path string variable contains the path for creating the file for writing the data of the current symbol in the loop.

Create WriteDataToFile() function to write data to the file. [Path]+[file name] is generated at the beginning of the function. A file name consists of a symbol name and the current time frame. For example, EURUSD_H1.csv. If the file with such a name is already present, it is just opened for writing. Previously written data will be deleted. New data will be written in it instead. If the file is created/opened successfully, FileOpen() function returns the handle that will be used for accessing the file.

Checking for the handle. If it is present, the header line is written. The appropriate line will be written depending on what headers have been selected by a user. The main loop of writing the history data begins afterwards.

Before writing the next line, it should be converted to the format specified by the user. To do that, we should receive the bar open time and sort out day, month, year and time separately by variables using StringSubstr() function. Then, we should define if the date and time will be located in a single or separate columns depending on the format specified by the user. Then all parts are joined together into one line using StringConcatenate() function. After all the lines are written, the file is closed by FileClose() function.

The entire WriteDataToFile() function code is shown below:

//____________________________________________________________________
//+------------------------------------------------------------------+
//| WRITE_DATA_TO_FILE                                               |
//+------------------------------------------------------------------+
void WriteDataToFile(int s)
  {
// Number of decimal places in the symbol price
   int dgt=(int)SymbolInfoInteger(symbols[s],SYMBOL_DIGITS);
//---
   string nm_fl=path+symbols[s]+"_"+gStrTF(_Period)+".csv"; // File name
//---
// Receive file handle for writing
   hFl=FileOpen(nm_fl,FILE_WRITE|FILE_CSV|FILE_ANSI,',');
//---
   if(hFl>0) // If the handle is received
     {
      // Write the headers
      if(format_headers==NSDT_5)
        { FileWrite(hFl,"\"Date\" ""\"Time\" ""\"Open\" ""\"High\" ""\"Low\" ""\"Close\" ""\"Volume\""); }
      //---
      if(format_headers==NSDT_6)
        { FileWrite(hFl,"Date","Open","High","Low","Close","Volume"); }
      //---
      // Write the data
      for(int i=0; i<=copied_bars-1; i++)
        {
         if(IsStopped()) // If program execution interrupted by a user
           {
            DelAllScriptObjects(); // Delete objects created by the script from the chart
            //---
            Print("------\nUser deleted the script!"); break;
           }
         //---
         sdt=TSdm(rates[i].time); // Bar open time
         //---
         // Divide the date by year, month and time
         yyyy=StringSubstr(sdt,0,4);
         mm=StringSubstr(sdt,5,2);
         dd=StringSubstr(sdt,8,2);
         tm=StringSubstr(sdt,11);
         //---
         string sep_dt_tm=""; // Separator of Date and Time columns
         //---
         // Join the data with the separator in the necessary order
         if(format_date==SEP_POINT1 || format_date==SEP_SLASH1) { sep_dt_tm=" "; }
         if(format_date==SEP_POINT2 || format_date==SEP_SLASH2) { sep_dt_tm=","; }
         //---
         // Join everything in one line
         StringConcatenate(sdt,dd,sep,mm,sep,yyyy,sep_dt_tm,tm);
         //---
         FileWrite(hFl,
                   sdt,// Date-time
                   DS_dgt(rates[i].open,dgt),      // Open price
                   DS_dgt(rates[i].high,dgt),      // High price
                   DS_dgt(rates[i].low,dgt),       // Low price
                   DS_dgt(rates[i].close,dgt),     // Close price
                   IS((int)rates[i].tick_volume)); // Tick volume price
         //---
         // Update writing progress value for the current symbol
         pgs_pcnt=((double)(i+1)/copied_bars)*100;
         //---
         // Update data in the table
         InfoTable(s); if(show_progress) { ChartRedraw(); }
        }
      //---
      FileClose(hFl); // Close the file
     }
   else { Print("Error when creating/opening file!"); }
  }

This was the last function in OnStart() function's basic loop. If it was not the last symbol, then everything is repeated for the next one. Otherwise, the loop is broken. If the user specified to clear the list of symbols in Market Watch window in the script parameters, the symbols with currently non-active charts will be deleted by DelSymbolsFromMarketWatch() function. After that, all graphical objects created by the script are deleted and the program stops. The data is ready for use.

The details on how to download data to NeuroShell DayTrader Professional can be found in my blog. Below is the video showing the script operation:



Conclusion

Whatever program I used for developing trading strategies, I always reached some limitations preventing me from further development of my ideas. Eventually, I realized that programming is essential here. MQL5 is the best solution for those who really want to succeed. However, other programs for data analysis and development of trading strategies can also be useful when searching for new ideas. It would have taken me much longer to find them, if I had used only one tool.

Good luck!

Translated from Russian by MetaQuotes Software Corp.
Original article: https://www.mql5.com/ru/articles/502

Attached files |
writedatatofile.mq5 (68.96 KB)
Step on New Rails: Custom Indicators in MQL5 Step on New Rails: Custom Indicators in MQL5

I will not list all of the new possibilities and features of the new terminal and language. They are numerous, and some novelties are worth the discussion in a separate article. Also there is no code here, written with object-oriented programming, it is a too serous topic to be simply mentioned in a context as additional advantages for developers. In this article we will consider the indicators, their structure, drawing, types and their programming details, as compared to MQL4. I hope that this article will be useful both for beginners and experienced developers, maybe some of them will find something new.

Here Comes the New MetaTrader 5 and MQL5 Here Comes the New MetaTrader 5 and MQL5

This is just a brief review of MetaTrader 5. I can't describe all the system's new features for such a short time period - the testing started on 2009.09.09. This is a symbolical date, and I am sure it will be a lucky number. A few days have passed since I got the beta version of the MetaTrader 5 terminal and MQL5. I haven't managed to try all its features, but I am already impressed.

Using text files for storing input parameters of Expert Advisors, indicators and scripts Using text files for storing input parameters of Expert Advisors, indicators and scripts

The article describes the application of text files for storing dynamic objects, arrays and other variables used as properties of Expert Advisors, indicators and scripts. The files serve as a convenient addition to the functionality of standard tools offered by MQL languages.

How to create an indicator of non-standard charts for MetaTrader Market How to create an indicator of non-standard charts for MetaTrader Market

Through offline charts, programming in MQL4, and reasonable willingness, you can get a variety of chart types: "Point & Figure", "Renko", "Kagi", "Range bars", equivolume charts, etc. In this article, we will show how this can be achieved without using DLL, and therefore such "two-for-one" indicators can be published and purchased from the Market.