Custom Strategy Tester based on fast mathematical calculations
32 690
3
Contents
- Introduction
- General information about the math calculations mode
- Saving the symbol history data for the tester based on mathematical calculations
- Loading MqlRates from a compressed byte array
- Prototype strategy based on a moving average
- Virtual position class
- Trading block class
- The first testing of the expert. Working in the optimizer
- Saving custom optimization results using the frame mechanism
- Getting a byte representation of the position history. Saving data in frames
- Creating a Strategy Analyzer
- Conclusion
Introduction
The Strategy Tester provided in MetaTrader 5 has powerful features for solving a variety of tasks. It can be used to test both complex strategies for trading instrument baskets, and single strategies with simple entry and exit rules. However, such a vast functionality does not always prove useful. Often, we just need to quickly check a simple trading idea or make approximate calculations, the accuracy of which will be compensated by their speed. The standard tester in MetaTrader 5 has an interesting but rarely used feature: it can perform computations in the math calculations mode. This is a limited mode for running the strategy tester, which nevertheless has all the advantages of full-fledged optimization: cloud computing is available, a genetic optimizer can be used and it is possible to collect custom data types.
Custom strategy tester may be necessary not only for those who require absolute speed. Testing in the math calculations mode opens the way for researchers as well. The standard strategy tester allows simulating trade operations as close to reality as possible. This requirement is not always useful in research. For example, sometimes it is necessary to obtain an estimate of the net efficiency of a trading system, without taking slippage, spread and commission into account. The math calculations tester, developed in this article, provides such ability.
Naturally, one cannot square the circle. This article is no exception. Writing a custom strategy tester requires serious and time-consuming work. The goal here is modest: to show that with the right libraries, creating a custom tester is not so difficult as it might seem at first.
If the topic proves interesting to my colleagues, this article will see a continuation that develops the proposed ideas.
General information about the math calculations mode
The math calculations mode is launched in the strategy tester window. To do this, select the menu item of the same name in the drop-down list:
Fig. 1. Selecting the math calculations mode in the strategy tester
This mode calls only a limited set of functions, and the trading environment (symbols, account information, trade server properties) is not available. OnTester() becomes the main call function, which can be utilized by users to set a special custom optimization criterion. It will be used alongside other standard optimization criteria and it can be displayed in the standard user strategy report. It is outlined in red in the screenshot below:
Fig. 2. Custom optimization criterion calculated in the OnTester function
The values returned by the OnTester function are selectable and optimizable. Let us demonstrate this in a simple expert:
//+------------------------------------------------------------------+ //| OnTesterCheck.mq5 | //| Copyright 2017, MetaQuotes Software Corp. | //| http://www.mql5.com | //+------------------------------------------------------------------+ #property copyright "Copyright 2017, MetaQuotes Software Corp." #property link "http://www.mql5.com" #property version "1.00" input double x = 0.01; //+------------------------------------------------------------------+ //| Tester function | //+------------------------------------------------------------------+ double OnTester() { double ret = MathSin(x); return(ret); } //+------------------------------------------------------------------+
Its code contains nothing but the input parameter x and the OnTester function, which calculates the sine value from the passed argument. In this case, x. Now try to optimize this function. To do this, select the "Slow complete algorithm" optimization mode in the strategy tester, and the previous simulation mode: "Math calculations".
Set the variation range of x in the optimization parameters: start — 0.01, step — 0.01, stop — 10. Once everything is ready, run the strategy tester. It will finish its work almost instantly. After that, open the optimization graph and select "1D graph" in the context menu. This will show a sine function in the graphical interpretation:
Fig. 3. Graphical representation of the sine function
A distinctive feature of this mode is the minimal consumption of resources. The read-write operations on the hard drive are minimized, tester agents do not download quotes of the requested symbols, no additional computations, all calculations are focused in the OnTester function.
Given the high-speed potential of OnTester, it is possible to create a self-sufficient computation module able to perform simple simulations. Here are the elements of this module:
- History of the symbol for testing
- Virtual position system
- Trading system for managing virtual positions
- Result analysis system
The self-sufficiency of the module means that a single expert will contain all the necessary data for testing and the testing system itself, which will use them. This expert can be easily passed to a distributed computing network in case a cloud optimization is required.
Let us move on to description of the first part of the system, namely, how to store the history for testing.
Saving the symbol history data for the tester based on mathematical calculations
The math calculations mode does not imply access to trading instruments. Calling functions like CopyRates(Symbol(),...) is meaningless here. However, historical data are necessary for the simulation. For this purpose, the quotes history of the required symbol can be stored in a precompressed array of the uchar[] type:
uchar symbol[128394] = {0x98,0x32,0xa6,0xf7,0x64,0xbc...};
Any type of data — sound, image, numbers and strings — can be represented as a simple set of bytes. A byte is a short block consisting of eight bits. Any information is stored as "batches" in a sequence consisting of these bytes. MQL5 has a special data type — uchar, where each value can represent exactly one byte. Thus, an array of uchar type with 100 elements can store 100 bytes.
Quotes of a symbol consist of many bars. Each bar includes the information about the bar opening time, its prices (High, Low, Open and Close) and volume. Each such value is stored in a variable of the appropriate length. Here is the table:
| Value | Data type | Size in bytes |
|---|---|---|
| Open Time | datetime | 8 |
| Open | double | 8 |
| High | double | 8 |
| Low | double | 8 |
| Close | double | 8 |
| Tick volume | long | 8 |
| Spread | int | 4 |
| Real volume | long | 8 |
It is easy to calculate that storing one bar will require 60 bytes, or a uchar array consisting of 60 elements. For a 24-hour Forex market, one trading day consists of 1,440 minute bars. Consequently, one-minute history of one year consists of approximately 391,680 bars. Multiplying this number by 60 bytes, we find out that one year of a minute history in uncompressed form takes up approximately 23 MB. Is this a lot or a little? It is not much by modern standards, but imagine what happens if we decide to test an expert on the data for 10 years. It will be necessary to store 230 MB of data, and maybe even distribute them over a network. This is very much even by modern standards.
Therefore, it is necessary to somehow compress this information. Fortunately, a special library has been written for working with Zip archives. In addition to various features, this library allows converting the compression result into an array of bytes, which greatly facilitates out work.
So, our algorithm will load the MqlRates array of bars, convert it into a byte representation, then compress it with the Zip archiver, and save the compressed data as a uchar array defined in the mqh file.
To convert quotes into a byte array, the system of conversion via the union type will be used. This system allows placing several data types in one storage field. Thus, it is possible to access the data of one type by addressing another. Such a union will store two types: the MqlRates structure and the uchar array, with the number of elements equal to the size of MqlRates. To understand how this system works, refer to the first version of the SaveRates.mq5 script, which converts the symbol history into a uchar byte array:
//+------------------------------------------------------------------+ //| SaveRates.mq5 | //| Copyright 2016, Vasiliy Sokolov. | //| http://www.mql5.com | //+------------------------------------------------------------------+ #property copyright "Copyright 2016, Vasiliy Sokolov." #property link "http://www.mql5.com" #property version "1.00" #include <Zip\Zip.mqh> #include <ResourceCreator.mqh> input ENUM_TIMEFRAMES MainPeriod; union URateToByte { MqlRates bar; uchar bar_array[sizeof(MqlRates)]; }RateToByte; //+------------------------------------------------------------------+ //| Script program start function | //+------------------------------------------------------------------+ void OnStart() { //-- Download quotes MqlRates rates[]; int total = CopyRates(Symbol(), Period(), 0, 20000, rates); uchar symbol_array[]; //-- Convert them to a byte representation ArrayResize(symbol_array, sizeof(MqlRates)*total); for(int i = 0, dst = 0; i < total; i++, dst +=sizeof(MqlRates)) { RateToByte.bar = rates[i]; ArrayCopy(symbol_array, RateToByte.bar_array, dst, 0, WHOLE_ARRAY); } //-- Compress them into a zip archive CZip Zip; CZipFile* file = new CZipFile(Symbol(), symbol_array); Zip.AddFile(file); uchar zip_symbol[]; //-- Get the byte representation of the compressed archive Zip.ToCharArray(zip_symbol); //-- Write it as a mqh include file CCreator creator; creator.ByteArrayToMqhArray(zip_symbol, "rates.mqh", "rates"); } //+------------------------------------------------------------------+
After this code is executed, the zip_symbol array will contain a compressed array of MqlRates structures — compressed history of quotes. Then the compressed array is stored as a mqh file on the computer hard drive. Details on how and why this is done are provided below.
Obtaining a byte representation of quotes and compressing them is not sufficient. It is necessary to record this representation as a uchar array. In this case, the array should be loaded in the form of a resource, i.e. it must be compiled with the program. For this purpose, create a special mqh header file containing this array as a simple set of ASCII characters. To do this, use the special CResourceCreator class:
//+------------------------------------------------------------------+ //| ResourceCreator.mqh | //| Copyright 2017, Vasiliy Sokolov. | //| http://www.mql5.com | //+------------------------------------------------------------------+ #property copyright "Copyright 2017, Vasiliy Sokolov." #property link "http://www.mql5.com" #include <Arrays\ArrayObj.mqh> //+------------------------------------------------------------------+ //| Contains the string identifiers of the created resource array | //+------------------------------------------------------------------+ class CResInfo : public CObject { public: string FileName; string MqhFileName; string ArrayName; }; //+------------------------------------------------------------------+ //| Creates a MQL resource as a byte array. | //+------------------------------------------------------------------+ class CCreator { private: int m_common; bool m_ch[256]; string ToMqhName(string name); void CreateInclude(CArrayObj* list_info, string file_name); public: CCreator(void); void SetCommonDirectory(bool common); bool FileToByteArray(string file_name, uchar& byte_array[]); bool ByteArrayToMqhArray(uchar& byte_array[], string file_name, string array_name); void DirectoryToMqhArray(string src_dir, string dst_dir, bool create_include = false); }; //+------------------------------------------------------------------+ //| Default constructor | //+------------------------------------------------------------------+ CCreator::CCreator(void) : m_common(FILE_COMMON) { ArrayInitialize(m_ch, false); for(uchar i = '0'; i < '9'; i++) m_ch[i] = true; for(uchar i = 'A'; i < 'Z'; i++) m_ch[i] = true; } //+------------------------------------------------------------------+ //| Sets the FILE_COMMON flag, or removes it | //+------------------------------------------------------------------+ CCreator::SetCommonDirectory(bool common) { m_common = common ? FILE_COMMON : 0; } //+------------------------------------------------------------------+ //| Converts all files in the src_dir directory to mqh files | //| containing the byte representation of these files | //+------------------------------------------------------------------+ void CCreator::DirectoryToMqhArray(string src_dir,string dst_dir, bool create_include = false) { string file_name; string file_mqh; CArrayObj list_info; long h = FileFindFirst(src_dir+"\\*", file_name, m_common); if(h == INVALID_HANDLE) { printf("Directory" + src_dir + " is not found, or it does not contain files"); return; } do { uchar array[]; if(FileToByteArray(src_dir+file_name, array)) { string norm_name = ToMqhName(file_name); file_mqh = dst_dir + norm_name + ".mqh"; ByteArrayToMqhArray(array, file_mqh, "m_"+norm_name); printf("Create resource: " + file_mqh); // Add information about the created resource CResInfo* info = new CResInfo(); list_info.Add(info); info.FileName = file_name; info.MqhFileName = norm_name + ".mqh"; info.ArrayName = "m_"+norm_name; } }while(FileFindNext(h, file_name)); if(create_include) CreateInclude(&list_info, dst_dir+"include.mqh"); } //+------------------------------------------------------------------+ /| Creates a mqh file with inclusions of all generated files | //+------------------------------------------------------------------+ void CCreator::CreateInclude(CArrayObj *list_info, string file_name) { int handle = FileOpen(file_name, FILE_WRITE|FILE_TXT|m_common); if(handle == INVALID_HANDLE) { printf("Failed to create the include file " + file_name); return; } // Create the include header for(int i = 0; i < list_info.Total(); i++) { CResInfo* info = list_info.At(i); string line = "#include \"" + info.MqhFileName + "\"\n"; FileWriteString(handle, line); } // Create a function for copying the resource array to the calling code FileWriteString(handle, "\n"); FileWriteString(handle, "void CopyResource(string file_name, uchar &array[])\n"); FileWriteString(handle, "{\n"); for(int i = 0; i < list_info.Total(); i++) { CResInfo* info = list_info.At(i); if(i == 0) FileWriteString(handle, " if(file_name == \"" + info.FileName + "\")\n"); else FileWriteString(handle, " else if(file_name == \"" + info.FileName + "\")\n"); FileWriteString(handle, " ArrayCopy(array, " + info.ArrayName + ");\n"); } FileWriteString(handle, "}\n"); FileClose(handle); } //+------------------------------------------------------------------+ //| converts the passed name into a correct name of the MQL variable | //+------------------------------------------------------------------+ string CCreator::ToMqhName(string name) { uchar in_array[]; uchar out_array[]; int total = StringToCharArray(name, in_array); ArrayResize(out_array, total); int t = 0; for(int i = 0; i < total; i++) { uchar ch = in_array[i]; if(m_ch[ch]) out_array[t++] = ch; else if(ch == ' ') out_array[t++] = '_'; uchar d = out_array[t-1]; int dbg = 4; } string line = CharArrayToString(out_array, 0, t); return line; } //+------------------------------------------------------------------+ //| Returns the byte representation of the passed file as the | //| byte_array array | //+------------------------------------------------------------------+ bool CCreator::FileToByteArray(string file_name, uchar& byte_array[]) { int handle = FileOpen(file_name, FILE_READ|FILE_BIN|m_common); if(handle == -1) { printf("Failed to open file " + file_name + ". Reason: " + (string)GetLastError()); return false; } FileReadArray(handle, byte_array, WHOLE_ARRAY); FileClose(handle); return true; } //+------------------------------------------------------------------+ //| Converts the passed byte_array byte array into a mqh file named | //| file_name which contains the description of an array named | //| array_name | //+------------------------------------------------------------------+ bool CCreator::ByteArrayToMqhArray(uchar& byte_array[], string file_name, string array_name) { int size = ArraySize(byte_array); if(size == 0) return false; int handle = FileOpen(file_name, FILE_WRITE|FILE_TXT|m_common, ""); if(handle == -1) return false; string strSize = (string)size; string strArray = "uchar " +array_name + "[" + strSize + "] = \n{\n"; FileWriteString(handle, strArray); string line = " "; int chaptersLine = 32; for(int i = 0; i < size; i++) { ushort ch = byte_array[i]; line += (string)ch; if(i == size - 1) line += "\n"; if(i>0 && i%chaptersLine == 0) { if(i < size-1) line += ",\n"; FileWriteString(handle, line); line = " "; } else if(i < size - 1) line += ","; } if(line != "") FileWriteString(handle, line); FileWriteString(handle, "};"); FileClose(handle); return true; }
Let us not dwell on its operation in detail, and only describe it in general terms, listing its features.
- Reads any arbitrary file on the hard drive and stores its byte representation as a uchar array in a mqh file.
- Reads any arbitrary directory on the hart drive and stores the byte representation of all files located in this directory. The byte representation for each such file is located in a separate mqh file containing a uchar array.
- It takes the uchar array of bytes as input and stores it as an array of characters in a separate mqh file.
- Creates a special header file that contains links to all mqh files created during the generation process. Also, a special function is created, which takes the array name as input and returns its byte representation. This algorithm uses dynamic code generation.
The described class is a powerful alternative to the regular resource allocation system in a MQL program.
By default, all file operations take place in the shared file directory (FILE_COMMON). If you run the script from the previous listing, the folder will have a new rates.mqh file (the file name is defined by the second parameter of the ByteArrayToMqhArray method). It will contain a giant rates[] array (the array name is defined by the third parameter of this method). Here is a snippet of the file contents:
Fig. 4. MqlRates quotes as a compressed byte array named rates
Data compression works fine. One year of uncompressed one-minute history for the EURUSD currency pair takes about 20 MB, after compression — only 5 MB. However, it is better not to open the rates.mqh file in MetaEditor: its size is much larger than this figure, and the editor may freeze. But do not worry. After compilation, the text is converted into bytes and the actual size of the program increases only by the real value of the stored information, in this case, by 5 MB.
By the way, this technique can be used in an ex5 program to store necessary information of any type, not just the quotes history.
Loading MqlRates from a compressed byte array
Now that the history is stored, it can be included in any MQL program by simply adding the include directive at its beginning:
... #include "rates.mqh" ...
Note that the rates.mqh file should be moved to the source directory of the program itself.
Including the data is not enough. It is also necessary to write a block of procedures to reverse the data into a normal MqlRates array. Let us implement a special function LoadRates to do this. It will take a reference to an empty MqlRates array as input. Once it is executed, the array will contain the conventional MqlRates quotes loaded from the compressed array. Here is the code for this function:
//+------------------------------------------------------------------+ //| Mtester.mqh | //| Copyright 2017, MetaQuotes Software Corp. | //| http://www.mql5.com | //+------------------------------------------------------------------+ #property copyright "Copyright 2017, MetaQuotes Software Corp." #property link "http://www.mql5.com" #include <Zip\Zip.mqh> #include "rates.mqh" //+------------------------------------------------------------------+ //| Casting MqlRates to uchar[] | //+------------------------------------------------------------------+ union URateToByte { MqlRates bar; uchar bar_array[sizeof(MqlRates)]; }; //+------------------------------------------------------------------+ //| Converts compressed data to a MqlRates array of quotes | //| Returns the number of received bars, returns -1 in the case of | //| failure | //+------------------------------------------------------------------+ int LoadRates(string symbol_name, MqlRates &mql_rates[]) { CZip Zip; Zip.CreateFromCharArray(rates); CZipFile* file = dynamic_cast<CZipFile*>(Zip.ElementByName(symbol_name)); if(file == NULL) return -1; uchar array_rates[]; file.GetUnpackFile(array_rates); URateToByte RateToBar; ArrayResize(mql_rates, ArraySize(array_rates)/sizeof(MqlRates)); for(int start = 0, i = 0; start < ArraySize(array_rates); start += sizeof(MqlRates), i++) { ArrayCopy(RateToBar.bar_array, array_rates, 0, start, sizeof(MqlRates)); mql_rates[i] = RateToBar.bar; } return ArraySize(mql_rates); } //+------------------------------------------------------------------+
The function is located in the Mtester.mqh file. This will be the first function for working in the math calculations mode. New features will be eventually added to the Mtester.mqh file, and it may turn into a full-fledged mathematical strategy testing engine.
Let us write a trivial strategy for the math calculations mode. It will perform only two functions: load quotes in the OnInit function and calculate the average value of all Close prices in the OnTester function. The calculation result will be returned to MetaTrader:
//+------------------------------------------------------------------+ //| MExpert.mq5 | //| Copyright 2017, MetaQuotes Software Corp. | //| http://www.mql5.com | //+------------------------------------------------------------------+ #property copyright "Copyright 2017, MetaQuotes Software Corp." #property link "http://www.mql5.com" #property version "1.00" #include "Mtester.mqh" //+------------------------------------------------------------------+ //| Quotes to be used in testing | //+------------------------------------------------------------------+ MqlRates Rates[]; //+------------------------------------------------------------------+ //| Expert initialization function | //+------------------------------------------------------------------+ int OnInit() { //-- Load the quotes for the specified symbol. if(LoadRates(Symbol(), Rates)==-1) { printf("Quotes for symbol " + Symbol() + " not found. Create the appropriate quotes resource."); return INIT_PARAMETERS_INCORRECT; } printf("Loaded " + (string)ArraySize(Rates) + " bars for symbol " + Symbol()); return(INIT_SUCCEEDED); } //+------------------------------------------------------------------+ //| Tester function | //+------------------------------------------------------------------+ double OnTester() { double mean = 0.0; for(int i = 0; i < ArraySize(Rates); i++) mean += Rates[i].close; mean /= ArraySize(Rates); return mean; } //+------------------------------------------------------------------+
After the expert is compiled, load it into the Strategy Tester and select the "Math calculations" mode. Run the testing and turn to the journal:
2017.12.13 15:12:25.127 Core 2 math calculations test of Experts\MTester\MExpert.ex5 started 2017.12.13 15:12:25.127 Core 2 Loaded 354159 bars for symbol EURUSD 2017.12.13 15:12:25.127 Core 2 OnTester result 1.126596405653942 2017.12.13 15:12:25.127 Core 2 EURUSD,M15: mathematical test passed in 0:00:00.733 (total tester working time 0:00:01.342) 2017.12.13 15:12:25.127 Core 2 217 Mb memory used
As you can see, the EA worked as expected. All quotes were correctly loaded, as indicated by the record about the number of loaded bars. Also, it correctly iterated over all bars to calculate the average value, which was returned to the calling thread. The average price of all EURUSD quotes for the last year turned out to be equal to 1.12660.
Prototype strategy based on a moving average
Results achieved are impressive: data were received and compressed, then stored as a static uchar array that was loaded into the expert, then the data were unzipped and converted back into an array of quotes. Now it is time to write the first useful strategy. Let us use the classic version based on a crossover of two moving averages. This strategy is easy to implement. Since the trading environment in the math calculations mode is not available, indicators like iMA cannot be called directly. Instead, the value of the moving average must be calculated manually. The main task in this testing mode is maximum acceleration. Therefore, all used algorithms must work fast. It is known that calculation of the moving average refers to the class of simple problems with a computational complexity of O(1). This means that the calculation speed of the average value should not depend on the moving average period. For these purposes, a premade library for the ring buffer will be used. The details of this algorithm have been discussed in a separate article.
First, create a template of the first expert:
//+------------------------------------------------------------------+ //| MExpert.mq5 | //| Copyright 2017, MetaQuotes Software Corp. | //| http://www.mql5.com | //+------------------------------------------------------------------+ #property copyright "Copyright 2017, MetaQuotes Software Corp." #property link "http://www.mql5.com" #property version "1.00" #include "Mtester.mqh" #include <RingBuffer\RiSma.mqh> input int PeriodFastMa = 9; input int PeriodSlowMa = 16; CRiSMA FastMA; // Ring buffer to calculate the fast moving average CRiSMA SlowMA; // Ring buffer to calculate the slow moving average //+------------------------------------------------------------------+ //| Quotes to be used in testing | //+------------------------------------------------------------------+ MqlRates Rates[]; //+------------------------------------------------------------------+ //| Expert initialization function | //+------------------------------------------------------------------+ int OnInit() { //-- Verify the correctness of the combination of parameters //-- The fast moving average cannot be larger than the slow moving average if(PeriodFastMa >= PeriodSlowMa) return INIT_PARAMETERS_INCORRECT; //-- Initialize the periods of ring buffers FastMA.SetMaxTotal(PeriodFastMa); SlowMA.SetMaxTotal(PeriodSlowMa); //-- Load the quotes for the specified symbol. if(LoadRates(Symbol(), Rates)==-1) { printf("Quotes for symbol " + Symbol() + " not found. Create the appropriate quotes resource."); return INIT_FAILED; } printf("Loaded " + (string)ArraySize(Rates) + " bars for symbol " + Symbol()); return(INIT_SUCCEEDED); } //+------------------------------------------------------------------+ //| Describe the strategy | //+------------------------------------------------------------------+ double OnTester() { for(int i = 1; i < ArraySize(Rates); i++) { FastMA.AddValue(Rates[i].close); SlowMA.AddValue(Rates[i].close); // The EA logic will be located here } return 0.0; } //+------------------------------------------------------------------+
It defines two parameters with averaging periods of the fast and slow MA. Then two ring buffers are declared for calculating these averages. The initialization block verifies the correctness of the entered parameters. Since the parameters will not be set by user, but rather selected automatically by the strategy tester in optimization mode, often the parameters will not be combined correctly. In this case, the fast MA period may turn out to be larger than that of the slow MA. To avoid this confusion and to save time on optimization, such passes will be finished even before they start. This is done by returning the INIT_PARAMETERS_INCORRECT constant in the OnInit block.
Once the buffers have been initialized, parameters checked and quotes loaded, comes the time to run the test itself: the OnTester function is started. In it, the main testing will be inside the 'for' block. The code shows that if the average value of the FastMA ring buffer is greater than the average value of SlowMA, it is necessary to open a long position, and vice versa. However, there is still no trading module for opening such long and short positions. It is yet to be written.
Virtual position class
As mentioned before, the math calculations mode is not fit for calculation of any strategies. Therefore, it has no trade functions. Also, the MetaTrader environment cannot be used as well. The term "position" is entirely meaningless, it simply does not exist. Therefore, it is necessary to create a simplified analog of the MetaTrader position. It will contain only the most necessary information. To do this, create a class with these fields:
- position opening time;
- position Open price;
- position closing time;
- position Close price;
- position volume;
- spread at the moment of position opening;
- direction of position.
Perhaps, it will be supplemented with additional information in the future, but these fields are sufficient for now.
//+------------------------------------------------------------------+ //| Mtester.mqh | //| Copyright 2017, MetaQuotes Software Corp. | //| http://www.mql5.com | //+------------------------------------------------------------------+ #property copyright "Copyright 2017, MetaQuotes Software Corp." #property link "http://www.mql5.com" #include <Object.mqh> #include "rates.mqh" #include "Type2Char.mqh" //+------------------------------------------------------------------+ //| Virtual position class for the tester based on math calculations | //+------------------------------------------------------------------+ class CMposition : public CObject { private: datetime m_time_open; datetime m_time_close; double m_price_open; double m_price_close; double m_volume; int m_spread; ENUM_POSITION_TYPE m_type; public: CMposition(void); static int Sizeof(void); bool IsActive(void); datetime TimeOpen(void); datetime TimeClose(void); double PriceOpen(void); double PriceClose(void); double Volume(void); double Profit(void); ENUM_POSITION_TYPE PositionType(void); static CMposition* CreateOnBarOpen(MqlRates& bar, ENUM_POSITION_TYPE pos_type, double vol); void CloseOnBarOpen(MqlRates& bar); }; //+------------------------------------------------------------------+ //| One CMposition position takes up 45 bytes of data | //+------------------------------------------------------------------+ int CMposition::Sizeof(void) { return 48; } CMposition::CMposition(void):m_time_open(0), m_time_close(0), m_price_open(0.0), m_price_close(0.0), m_volume(0.0) { } //+------------------------------------------------------------------+ //| True, if the position is closed | //+------------------------------------------------------------------+ bool CMposition::IsActive() { return m_time_close == 0; } //+------------------------------------------------------------------+ //| Position opening time | //+------------------------------------------------------------------+ datetime CMposition::TimeOpen(void) { return m_time_open; } //+------------------------------------------------------------------+ //| Position closing time | //+------------------------------------------------------------------+ datetime CMposition::TimeClose(void) { return m_time_close; } //+------------------------------------------------------------------+ //| Position Open price | //+------------------------------------------------------------------+ double CMposition::PriceOpen(void) { return m_price_open; } //+------------------------------------------------------------------+ //| Position Close price | //+------------------------------------------------------------------+ double CMposition::PriceClose(void) { return m_price_close; } //+------------------------------------------------------------------+ //| Position volume | //+------------------------------------------------------------------+ double CMposition::Volume(void) { return m_volume; } //+------------------------------------------------------------------+ //| Returns the type of trade position | //+------------------------------------------------------------------+ ENUM_POSITION_TYPE CMposition::PositionType(void) { return m_type; } //+------------------------------------------------------------------+ //| Position profit | //+------------------------------------------------------------------+ double CMposition::Profit(void) { if(IsActive()) return 0.0; int sign = m_type == POSITION_TYPE_BUY ? 1 : -1; double pips = (m_price_close - m_price_open)*sign; double profit = pips*m_volume; return profit; } //+------------------------------------------------------------------+ //| Create position based on the passed parameters | //+------------------------------------------------------------------+ static CMposition* CMposition::CreateOnBarOpen(MqlRates &bar, ENUM_POSITION_TYPE pos_type, double volume) { CMposition* position = new CMposition(); position.m_time_open = bar.time; position.m_price_open = bar.open; position.m_volume = volume; position.m_type = pos_type; return position; } //+------------------------------------------------------------------+ //| Closes the position at the Open price of the passed bar | //+------------------------------------------------------------------+ void CMposition::CloseOnBarOpen(MqlRates &bar) { m_price_close = bar.open; m_time_close = bar.time; } //+------------------------------------------------------------------+
Creation of a position is the most interesting point in this implementation. Its fields are protected from external modifications, but the static CreateOnBarOpen method returns the class object with properly set parameters. There is no way to create an object of this class without referring to this method. This protects the data against unintentional changes.
Trading block class
It is now necessary to create a class for managing these positions. It will be the analog of the MetaTrader functions. Obviously, the positions themselves should be stored in this module as well. Two CArrayObj collections are intended for this purpose: the first one — Active — is required for storing the active positions of the strategy, the other one — History — will contain the positions in history.
The class will also have special methods for opening and closing positions:
- EntryAtOpenBar — opens a position with the required direction and volume;
- CloseAtOpenBar — closes the position at the specified index.
The positions will be opened and closed at the prices of the bar passed. Unfortunately, this approach does not prevent "peeping into the future", but it is easy to implement and very fast.
The CMtrade class (let us name it so) turned out to be quite simple:
//+------------------------------------------------------------------+ //| Mtester.mqh | //| Copyright 2017, MetaQuotes Software Corp. | //| http://www.mql5.com | //+------------------------------------------------------------------+ #property copyright "Copyright 2017, MetaQuotes Software Corp." #property link "http://www.mql5.com" #include <Object.mqh> #include <Arrays\ArrayObj.mqh> #include "Mposition.mqh" //+------------------------------------------------------------------+ //| Trading module for opening virtual positions | //+------------------------------------------------------------------+ class CMtrade { public: CMtrade(void); ~CMtrade(); CArrayObj Active; CArrayObj History; void EntryAtOpenBar(MqlRates &bar, ENUM_POSITION_TYPE pos_type, double volume); void CloseAtOpenBar(MqlRates &bar, int pos_index); }; //+------------------------------------------------------------------+ //| Default constructor | //+------------------------------------------------------------------+ CMtrade::CMtrade(void) { Active.FreeMode(false); } //+------------------------------------------------------------------+ //| Deleting all remaining positions | //+------------------------------------------------------------------+ CMtrade::~CMtrade() { Active.FreeMode(true); Active.Clear(); } //+------------------------------------------------------------------+ //| Creates a new position and adds it to the list of active | //| positions. | //+------------------------------------------------------------------+ void CMtrade::EntryAtOpenBar(MqlRates &bar, ENUM_POSITION_TYPE pos_type, double volume) { CMposition* pos = CMposition::CreateOnBarOpen(bar, pos_type, volume); Active.Add(pos); } //+------------------------------------------------------------------+ //| Closes a virtual position at the pos_index index at the Open | //| price of the passed bar | //+------------------------------------------------------------------+ void CMtrade::CloseAtOpenBar(MqlRates &bar, int pos_index) { CMposition* pos = Active.At(pos_index); pos.CloseOnBarOpen(bar); Active.Delete(pos_index); History.Add(pos); } //+------------------------------------------------------------------+
In practice, all its functionality is reduced to two functions:
- Get the new position from the static CMposition::CreateOnBarOpen method and add it to the Active list (the EntryOnOpenBar method);
- Move the selected position from the list of active positions to the list of historical positions, while the moved position is closed by the static CMposition::CLoseOnBarOpen method.
The trading class has been created, and all components for testing the expert are now available.
The first testing of the expert. Working in the optimizer
Let us put all components together. Here is the source code of the strategy based on two moving averages for working in the mathematical optimizer.
//+------------------------------------------------------------------+ //| MExpert.mq5 | //| Copyright 2017, MetaQuotes Software Corp. | //| http://www.mql5.com | //+------------------------------------------------------------------+ #property copyright "Copyright 2017, MetaQuotes Software Corp." #property link "http://www.mql5.com" #property version "1.00" #include <RingBuffer\RiSma.mqh> #include "Mtester.mqh" input int PeriodFastMa = 9; input int PeriodSlowMa = 16; CRiSMA FastMA; // Ring buffer to calculate the fast moving average CRiSMA SlowMA; // Ring buffer to calculate the slow moving average CMtrade Trade; // Trading module for virtual calculations //+------------------------------------------------------------------+ //| Quotes to be used in testing | //+------------------------------------------------------------------+ MqlRates Rates[]; //+------------------------------------------------------------------+ //| Expert initialization function | //+------------------------------------------------------------------+ int OnInit() { //-- Verify the correctness of the combination of parameters //-- The fast moving average cannot be larger than the slow moving average //if(PeriodFastMa >= PeriodSlowMa) // return INIT_PARAMETERS_INCORRECT; //-- Initialize the periods of ring buffers FastMA.SetMaxTotal(PeriodFastMa); SlowMA.SetMaxTotal(PeriodSlowMa); //-- Load the quotes for the specified symbol. if(LoadRates(Symbol(), Rates)==-1) { printf("Quotes for symbol " + Symbol() + " not found. Create the appropriate quotes resource."); return INIT_FAILED; } printf("Loaded " + (string)ArraySize(Rates) + " bars for symbol " + Symbol()); return(INIT_SUCCEEDED); } //+------------------------------------------------------------------+ //| Describe the strategy | //+------------------------------------------------------------------+ double OnTester() { for(int i = 1; i < ArraySize(Rates)-1; i++) { MqlRates bar = Rates[i]; FastMA.AddValue(Rates[i].close); SlowMA.AddValue(Rates[i].close); ENUM_POSITION_TYPE pos_type = FastMA.SMA() > SlowMA.SMA() ? POSITION_TYPE_BUY : POSITION_TYPE_SELL; //-- Close the positions opposite to the current signal for(int k = Trade.Active.Total()-1; k >= 0 ; k--) { CMposition* pos = Trade.Active.At(k); if(pos.PositionType() != pos_type) Trade.CloseAtOpenBar(Rates[i+1], k); } //-- If there are no positions, open a new one in the specified direction. if(Trade.Active.Total() == 0) Trade.EntryAtOpenBar(Rates[i+1], pos_type, 1.0); } double profit = 0.0; for(int i = 0; i < Trade.History.Total(); i++) { CMposition* pos = Trade.History.At(i); profit += pos.Profit(); } return profit; } //+------------------------------------------------------------------+
The OnTester function is now complete. The code is very simple. Let us consider its operation step by step.
- A 'for' cycle iterates over all quotes.
- The current direction of the deal is determined inside the cycle: buy if the fast SMA is above the slow SMA, and sell otherwise.
- All active deals are iterated over, and if their direction does not match the current direction, they are closed.
- If there are no positions, a new position is opened in the specified direction.
- At the end of the search, all closed positions are iterated over again and their total profit is calculated, which is later returned to the strategy tester.
The expert is now ready for testing in the optimizer. Simply run it in math calculations mode. To make sure that the optimization works, let us make a full search of the moving average parameters, as shown in the figure below:
Fig. 5. Selecting the optimization field for parameters
The provided example shows 1000 optimization passes, each of which processes 1 year of one-minute history. Nevertheless, the computations do not take up much time in this mode. On a computer with an i7 processor, the entire optimization took approximately 1 minute, after which a graph was built:
Fig. 6. Graph of 1000 passes in the "slow complete algorithm" mode.
But so far, there are very few tools for analyzing the obtained results. In fact, all we currently have is a single number that reflects the virtual profit. To fix this situation, it is necessary to develop a custom optimization data format and come up with a mechanism for generating and loading it. We will dwell on this below.
Saving custom optimization results using the frame mechanism
MetaTrader 5 implements a very advanced technique for working with custom data. It is based on the mechanism for generation and retrieval of so-called frames. These are essentially ordinary binary data, placed either as separate values or as an array of these values. For example, during optimization, it is possible to generate an array of data of arbitrary size and send them to the MetaTrader 5 Strategy Tester. The data contained in this array can be read using the FrameNext function and processed further, for example, displayed on the screen. The work with frames is possible only in optimization mode and only within three functions: OnTesterInit(), OnTesterDeinit() and OnTesterPass(). They all have no parameters and no return values. But everything is easier than it may seem. To illustrate this, let us create a simple script showing the general algorithm for working with frames:
//+------------------------------------------------------------------+ //| OnTesterSample.mq5 | //| Copyright 2017, MetaQuotes Software Corp. | //| http://www.mql5.com | //+------------------------------------------------------------------+ #property copyright "Copyright 2017, MetaQuotes Software Corp." #property link "http://www.mql5.com" #property version "1.00" input int Param = 1; //+------------------------------------------------------------------+ //| OnTesterInit function | //+------------------------------------------------------------------+ void OnTesterInit() { printf("Optimization started"); } //+------------------------------------------------------------------+ //| A strategy pass takes place here | //+------------------------------------------------------------------+ double OnTester() { uchar data[5] = {1,2,3,4,5}; // Generate the data for the frame FrameAdd("sample", 1, Param, data); // Create a new frame with the provided data return 3.0; } //+------------------------------------------------------------------+ //| The last added optimization frame can be obtained here | //+------------------------------------------------------------------+ void OnTesterPass() { ulong pass = 0; string name = ""; ulong id = 0; double value = 0.0; uchar data[]; FrameNext(pass, name, id, value, data); //-- create a pass file and add it to the zip archive printf("Received new frame # " + (string)pass + ". Name: " + (string)name + " ID: " + (string)id + " Value of Param: " + DoubleToString(value, 0)); } //+------------------------------------------------------------------+ //| OnTesterDeinit function | //+------------------------------------------------------------------+ void OnTesterDeinit() { printf("Optimization complete"); } //+------------------------------------------------------------------+
Run this code in the strategy tester with the math calculations mode selected. Set "Slow complete algorithm" as the optimization mode. The only parameter Param will be changed in the range from 10 to 90 with a step of 10.
Messages about receiving new frames will start appearing immediately after the optimization starts. The start and end of optimization are also tracked through special events. Application log:
2017.12.19 16:58:08.101 OnTesterSample (EURUSD,M15) Optimization started 2017.12.19 16:58:08.389 OnTesterSample (EURUSD,M15) Received new frame # 1. Name: sample ID: 1 Value of Param: 20 2017.12.19 16:58:08.396 OnTesterSample (EURUSD,M15) Received new frame # 0. Name: sample ID: 1 Value of Param: 10 2017.12.19 16:58:08.408 OnTesterSample (EURUSD,M15) Received new frame # 4. Name: sample ID: 1 Value of Param: 50 2017.12.19 16:58:08.426 OnTesterSample (EURUSD,M15) Received new frame # 5. Name: sample ID: 1 Value of Param: 60 2017.12.19 16:58:08.426 OnTesterSample (EURUSD,M15) Received new frame # 2. Name: sample ID: 1 Value of Param: 30 2017.12.19 16:58:08.432 OnTesterSample (EURUSD,M15) Received new frame # 3. Name: sample ID: 1 Value of Param: 40 2017.12.19 16:58:08.443 OnTesterSample (EURUSD,M15) Received new frame # 6. Name: sample ID: 1 Value of Param: 70 2017.12.19 16:58:08.444 OnTesterSample (EURUSD,M15) Received new frame # 7. Name: sample ID: 1 Value of Param: 80 2017.12.19 16:58:08.450 OnTesterSample (EURUSD,M15) Received new frame # 8. Name: sample ID: 1 Value of Param: 90 2017.12.19 16:58:08.794 OnTesterSample (EURUSD,M15) Optimization complete
The most interesting are the messages that display the information on the frame number, its identifier and the value of the Param parameter. All this valuable information can be retrieved using the FrameNext function.
An interesting feature of this mode is the double start of the expert. An expert with event handlers in its code is started twice: first in the strategy optimizer, then on the chart in real time. While the expert in the optimizer generates new data, the expert running on the chart receives them. Thus, the source codes of the expert are processed by different instances of the expert, even if they are located in the same place.
Once the data are received in the OnTesterPass function, they can be processed in any way. In the test sample, these data are simply output to the console using the printf function. But the data processing that is to be implemented may turn out to be much more complicated. This will be considered in the next section.
Getting a byte representation of the position history. Saving data in frames
The frame mechanism provides a convenient way to save, process and distribute information. However, it is necessary to generate this information itself. In the example array, it was a simple static uchar array with the values 1, 2, 3, 4, 5. Such data is not of much use. But a byte array can have an arbitrary length and store any data. To do this, the custom data types should be converted into a byte array of the uchar type. Something similar has already been done to MqlRates, where the quotes were saved in a byte array. The same will be done to the custom data.
A custom strategy tester consists of two parts. The first part generates data, the second one analyzes the data and displays them in a user-friendly form. It is also obvious that the main information for the strategy analysis can be obtained by analyzing all deals in history. Therefore, at the end of each run, all deals in history will be converted into a byte array, which is later added to the new frame. After receiving such a frame in the OnTesterPass() function, it is possible to add it the previously received ones, creating a whole collection of frames.
It will be necessary to not only convert the data about the positions into a byte array, but also to extract its data. This will require two procedures for each data type:
- Procedure for converting a custom type into a byte array;
- Procedure for converting a byte array to a custom type.
We already have the CMtrade trade module with two collections of positions — active and historical. Let us focus on the historical positions. The procedures for converting virtual positions will be written as corresponding methods.
The method for converting a position into a byte array:
//+------------------------------------------------------------------+ //| Converts a position to byte representation in the form of array | //+------------------------------------------------------------------+ int CMposition::ToCharArray(int dst_start, uchar &array[]) { int offset = dst_start; //-- Copy time open position type2char.time_value = m_time_open; ArrayCopy(array, type2char.char_array, offset, 0, sizeof(datetime)); offset += sizeof(datetime); //-- Copy time close position type2char.time_value = m_time_close; ArrayCopy(array, type2char.char_array, offset, 0, sizeof(datetime)); offset += sizeof(datetime); //-- Copy price open position type2char.double_value = m_price_open; ArrayCopy(array, type2char.char_array, offset, 0, sizeof(double)); offset += sizeof(double); //-- Copy price close position type2char.double_value = m_price_close; ArrayCopy(array, type2char.char_array, offset, 0, sizeof(double)); offset += sizeof(double); //-- Copy volume position type2char.double_value = m_volume; ArrayCopy(array, type2char.char_array, offset, 0, sizeof(double)); offset += sizeof(double); //-- Copy spread symbol type2char.int_value = m_spread; ArrayCopy(array, type2char.char_array, offset, 0, sizeof(int)); offset += sizeof(int); //-- Copy type of position type2char.int_value = m_type; ArrayCopy(array, type2char.char_array, offset, 0, sizeof(char)); offset += sizeof(int); //-- return last offset return offset; }
The reverse procedure:
//+------------------------------------------------------------------+ //| Loads a position from a byte array | //+------------------------------------------------------------------+ int CMposition::FromCharArray(int dst_start, uchar &array[]) { int offset = dst_start; //-- Copy time open position ArrayCopy(type2char.char_array, array, 0, offset, sizeof(datetime)); m_time_open = type2char.time_value; offset += sizeof(datetime); //-- Copy time close position ArrayCopy(type2char.char_array, array, 0, offset, sizeof(datetime)); m_time_close = type2char.time_value; offset += sizeof(datetime); //-- Copy price open position ArrayCopy(type2char.char_array, array, 0, offset, sizeof(double)); m_price_open = type2char.double_value; offset += sizeof(double); //-- Copy price close position ArrayCopy(type2char.char_array, array, 0, offset, sizeof(double)); m_price_close = type2char.double_value; offset += sizeof(double); //-- Copy volume position ArrayCopy(type2char.char_array, array, 0, offset, sizeof(double)); m_volume = type2char.double_value; offset += sizeof(double); //-- Copy spread symbol ArrayCopy(type2char.char_array, array, 0, offset, sizeof(int)); m_spread = type2char.int_value; offset += sizeof(int); //-- Copy type of position ArrayCopy(type2char.char_array, array, 0, offset, sizeof(int)); m_type = (ENUM_POSITION_TYPE)type2char.int_value; offset += sizeof(int); //-- return last offset return offset; }
The TypeToChar union (using its instance type2char) is the core of both conversion procedures:
//+------------------------------------------------------------------+ //| Converting simple types to a byte array | //+------------------------------------------------------------------+ union TypeToChar { uchar char_array[128]; int int_value; double double_value; float float_value; long long_value; short short_value; bool bool_value; datetime time_value; char char_value; };
Everything is similar to the RateToByte union discussed in the section on conversion of quotes.
All procedures are designed in such a way that they allow loading data from the global array containing the data on all closed virtual positions. This allows organizing a highly efficient exhaustive algorithm that does not require additional copying of the memory.
The CMTrade class will loop through all positions in history. This is logical, considering that it is the one that stores the collection of historical positions. The class, similar to CMposition, works in two directions: converts a collection of history positions into a uchar array, and also performs the reverse procedure: loading the list of historical positions from a byte array.
The procedure for converting a collection into a byte array:
//+------------------------------------------------------------------+ //| Converts list of positions in history to compressed zip archive | //| in the form of byte array. Returns true if successful | //| and false, if otherwise. | //+------------------------------------------------------------------+ bool CMtrade::ToCharArray(uchar &array[]) { int total_size = CMposition::Sizeof()*History.Total(); if(total_size == 0) { printf(__FUNCTION__ + ": Received array is empty"); return false; } if(ArraySize(array) != total_size && ArrayResize(array, total_size) != total_size) { printf(__FUNCTION__ + ": failed resized received array"); return false; } //-- Store positions in a byte stream for(int offset = 0, i = 0; offset < total_size; i++) { CMposition* pos = History.At(i); offset = pos.ToCharArray(offset, array); } return true; }
The reverse procedure:
//+------------------------------------------------------------------+ //| Loads list of historical positions from compressed zip archive | //| passed in the form of a byte array. Returns true if successful | //| and false, if otherwise. | //+------------------------------------------------------------------+ bool CMtrade::FromCharArray(uchar &array[], bool erase_prev_pos = true) { if(ArraySize(array) == 0) { printf(__FUNCTION__ + ": Received array is empty"); return false; } //-- Size of a byte stream must exactly match the byte representation of the positions int pos_total = ArraySize(array)/CMposition::Sizeof(); if(ArraySize(array)%CMposition::Sizeof() != 0) { printf(__FUNCTION__ + ": Wrong size of received array"); return false; } if(erase_prev_pos) History.Clear(); //-- Restore all positions from the byte stream for(int offset = 0; offset < ArraySize(array);) { CMposition* pos = new CMposition(); offset = pos.FromCharArray(offset, array); History.Add(pos); } return History.Total() > 0; }
To put all elements together, it is sufficient to obtain the byte representation of the historical positions at the end of the pass and save them in the frame:
//+------------------------------------------------------------------+ //| Describe the strategy | //+------------------------------------------------------------------+ double OnTester() { for(int i = 1; i < ArraySize(Rates)-1; i++) { MqlRates bar = Rates[i]; FastMA.AddValue(Rates[i].close); SlowMA.AddValue(Rates[i].close); ENUM_POSITION_TYPE pos_type = FastMA.SMA() > SlowMA.SMA() ? POSITION_TYPE_BUY : POSITION_TYPE_SELL; //-- Close the positions opposite to the current signal for(int k = Trade.Active.Total()-1; k >= 0 ; k--) { CMposition* pos = Trade.Active.At(k); if(pos.PositionType() != pos_type) Trade.CloseAtOpenBar(Rates[i+1], k); } //-- If there are no positions, open a new one in the specified direction. if(Trade.Active.Total() == 0) Trade.EntryAtOpenBar(Rates[i+1], pos_type, 1.0); } uchar array[]; //-- Get the byte representation of historical positions Trade.ToCharArray(array); //-- Load the byte representation into the frame and send it for further processing FrameAdd(MTESTER_STR, MTESTER_ID, 0.0, array); return Trade.History.Total(); }
Once the frame is formed and sent to the OnTesterPass() procedure for processing, it is necessary to figure out what to do with the frame next. As mentioned before, this strategy tester consists of two parts: data generation block and collected information analysis block. Such analysis requires all the generated frames to be stored in a convenient and economical format, so that this format could also be later analyzed with ease. This will be done using a zip archive. First, it effectively compresses data, meaning that even information on a thousand deals would not take up much space. Second, it provides a convenient file system. Each pass can be stored as a separate file within a single zip archive.
So, let us create a procedure for converting the byte contents of a frame into a zip archive.
//+------------------------------------------------------------------+ //| Add each new pass to the zip archive | //+------------------------------------------------------------------+ void OnTesterPass() { ulong pass = 0; string name = ""; ulong id = 0; double value = 0.0; uchar data[]; FrameNext(pass, name, id, value, data); //-- create a pass file and add it to the zip archive printf("Received new frame of size " + (string)ArraySize(data)); string file_name = name + "_" + (string)id + "_" + (string)pass + "_" + DoubleToString(value, 5)+".hps"; CZipFile* zip_file = new CZipFile(file_name, data); Zip.AddFile(zip_file); }
Due to the fact that the class for working with zip archives is quite powerful and features universal methods, it is extremely easy to add a new pass to the archive as a separate file. In essence, OnTesterPass adds a new zip file to the Zip archive, which is declared on the global scope:
CZip Zip; // Zip archive to be filled with optimization passes This procedure is called in parallel at the end of each optimization pass and is not resource-intensive.
At the end of optimization, the generated zip archive should be simply saved as a corresponding zip file. This is also very simple. This is performed in the OnTesterDeinit() procedure:
//+------------------------------------------------------------------+ //| Save the zip archive of all passes to the computer hard drive | //+------------------------------------------------------------------+ void OnTesterDeinit() { Zip.SaveZipToFile(OptimizationFile, FILE_COMMON); string f_totals = (string)Zip.TotalElements(); printf("Optimization complete. Total optimization passes saved: " + f_totals); }
Here, OptimizationFile is a custom string parameter that sets the optimization name. By default, it is "Optimization.zip". Thus, after the optimization of the updated SmaSample strategy is complete, a corresponding zip archive will be created. It can be found in the Files folder and opened by standard means:
Fig. 7. Internal contents of the optimization file
As you can see, all saved passes are perfectly stored, showing a high compression ratio from 3 to 5 times.
Having collected and saved these data to the hard drive, it is necessary to load them in another program and analyze them. This will be considered in the next section.
Creating a Strategy Analyzer
In the previous section, a zip archive containing the information on all passes has been created. This information should be processed now. For this purpose, let us create a special program named M-Tester Analyzer. It will load the generated archive and display each pass as a convenient balance chart. M-Tester Analyzer will also calculate the summary statistics for the selected pass.
One of the key features of the entire testing complex is the ability to store information on all passes simultaneously. This means that it is sufficient to perform the optimization only once. All its passes will be saved in a single archive and transferred to the user. Any pass can be later loaded from this optimization and its statistics can be viewed without spending time on running the strategy tester again.
Sequence of the analyzer actions:
- Load the selected optimization archive
- Select one of the optimization passes in this archive
- Plot a dynamics chart of the virtual balance based on existing deals
- Calculate the basic statistics of the pass, including such parameters as the number of deals, total profit, total loss, profit factor, expected value, etc.
- Output the calculated statistics in the main program window in the form of a table.
It is necessary to equip the user with tools to select an arbitrary pass from the archive: let us provide a simple transition from the current pass to the next or previous one, as well as the ability to set a custom pass number.
The program will be based on the CPanel graphics engine. Currently, there is no article dedicated to this library, however, it is easy to learn, compact and has been repeatedly used in various projects and articles.
The main code of the analyzer is located in the CAnalizePanel class, derived from CElChart. The analyzer itself is implemented in the form of an expert. The main expert file starts the graphical window of the analyzer. Here is the main expert file:
//+------------------------------------------------------------------+ //| mAnalizer.mq5 | //| Copyright 2017, MetaQuotes Software Corp. | //| http://www.mql5.com | //+------------------------------------------------------------------+ #include "mAnalizerPanel.mqh" CAnalyzePanel Panel; input string FilePasses = "Optimization.zip"; //+------------------------------------------------------------------+ //| Expert initialization function | //+------------------------------------------------------------------+ int OnInit() { //--- Panel.Width(800); Panel.Height(630); Panel.XCoord(10); Panel.YCoord(20); Panel.LoadFilePasses(FilePasses); Panel.Show(); ChartRedraw(); //--- return(INIT_SUCCEEDED); } //+------------------------------------------------------------------+ //| Expert deinitialization function | //+------------------------------------------------------------------+ void OnDeinit(const int reason) { Panel.Hide(); } //+------------------------------------------------------------------+ //| ChartEvent function | //+------------------------------------------------------------------+ void OnChartEvent(const int id, const long &lparam, const double &dparam, const string &sparam) { switch(id) { case CHARTEVENT_OBJECT_ENDEDIT: { CEventChartEndEdit event(sparam); Panel.Event(&event); break; } case CHARTEVENT_OBJECT_CLICK: { CEventChartObjClick event(sparam); Panel.Event(&event); break; } } ChartRedraw(); } //+------------------------------------------------------------------+
As you can see, the code is extremely simple. An object of type CAnalyzePanel is created. Then its sizes are set in the OnInit method, after which it is displayed on the current chart (the Show method). Of all events coming from the chart, only two are of interest: the end of text input and the click on the graphical object. These events are converted into a special CEvent object and passed to the panel (Panel.Event(...)). The analyzer panel receives these events and processes them.
Let us now describe the analyzer panel itself. It consists of a large CAnalyzePanel class, so its entire contents will not be published. Its full code is attached at the end of the article for those interested. Only a brief description of its operation will be provided here, using the following class prototype:
//+------------------------------------------------------------------+ //| mAnalizer.mq5 | //| Copyright 2017, MetaQuotes Software Corp. | //| http://www.mql5.com | //+------------------------------------------------------------------+ #include <Panel\ElChart.mqh> #include <Panel\ElButton.mqh> #include <Graphics\Graphic.mqh> #include "ListPass.mqh" #include "TradeAnalyze.mqh" //+------------------------------------------------------------------+ //| Panel for analyzing the passes of the mathematical analyzer | //+------------------------------------------------------------------+ class CAnalizePanel : public CElChart { private: //-- Arrays of elements and their statistics CArrayObj m_stat_descr; // Description of statistics CArrayObj m_stat_all; // Value of statistics for all deals CArrayObj m_stat_long; // Value of statistics for long deals CArrayObj m_stat_short; // Value of statistics for short deals CTradeAnalize m_analize; // Statistics calculation module //-- Graphical controls CElChart m_name_analyze; // Name of the main window CElChart m_np; // "Pass #" text CElChart m_of_pass; // "of ### passes" text CElChart m_pass_index; // Pass number input box CElButton m_btn_next; // "Next pass" button CElButton m_btn_prev; // "Previous pass" button CGraphic m_graphic; // Balance dynamics chart //-- Infrastructure CListPass m_passes; // List of passes int m_curr_pass; // Index of the current pass CCurve* m_balance_hist; // Balance dynamics line on the chart bool IsEndEditPass(CEvent* event); bool IsClick(CEvent* event, CElChart* el); void NextPass(void); void PrevPass(void); int GetCorrectPass(string text); void RedrawGraphic(void); void RedrawCurrPass(void); void PlotStatistic(void); string TypeStatToString(ENUM_MSTAT_TYPE type); void CreateStatElements(void); string ValueToString(double value, ENUM_MSTAT_TYPE type); public: CAnalizePanel(void); bool LoadFilePasses(string file_name, int file_common = FILE_COMMON); virtual void OnShow(); virtual void OnHide(); virtual void Event(CEvent *event); };
As you can see, the main work of this class is hidden inside. Among the public methods, the main one is loading the zip file containing the optimization passes. All the work of the class can be divided into three parts:
- Creating a chart and adding a balance graph to it.
- Creating text labels in the form of CElChart controls, which display the testing statistics.
- Actual calculation of the pass statistics.
Let us briefly describe each of these sections.
It is necessary to create quite a lot of controls to display all the collected statistics for each pass. The analyzer displays ten basic statistical parameters. In addition, each parameter is calculated separately for all deals, for buy deals and for sell deals. 10 additional labels are required to show the names of the indicators. Thus, it is necessary to create 40 text labels. To avoid creating each control manually, let us create an automation procedure. To do this, each calculated statistical parameter will be assigned its own identifier in a special enumeration:
//+------------------------------------------------------------------+ //| Identifiers of the statistical value type | //+------------------------------------------------------------------+ enum ENUM_MSTAT_TYPE { MSTAT_PROFIT, MSTAT_ALL_WINS_MONEY, MSTAT_ALL_LOSS_MONEY, MSTAT_TRADERS_TOTAL, MSTAT_WIN_TRADERS, MSTAT_LOSS_TRADERS, MSTAT_MAX_PROFIT, MSTAT_MAX_LOSS, MSTAT_PROFIT_FACTOR, MSTAT_MATH_EXP, }; #define MSTAT_ELEMENTS_TOTAL 10
Also, define an identifier for the calculation direction:
//+------------------------------------------------------------------+ //| Statistics can be calculated for one of three directions | //+------------------------------------------------------------------+ enum ENUM_MSTATE_DIRECT { MSTATE_DIRECT_ALL, // For all deals MSTATE_DIRECT_LONG, // For buy deals only MSTATE_DIRECT_SHORT, // For sell deals only };
The panel contains four groups of controls, each of which is located in its own array:
- Controls displaying the statistic names (the m_stat_descr array)
- Controls displaying the values of statistics for all deals (the m_stat_all array)
- Controls displaying the values of statistics for long deals (the m_stat_long array)
- Controls displaying the values of statistics for short deals (the m_stat_short array)
All these controls are created at the first start in the CAnalyzePanel::CreateStatElements(void) method.
Once all controls are created, they should be filled with the correct values. Calculation of these values is delegated to the external CTradeAnalize class:
#include <Arrays\ArrayObj.mqh> #include <Dictionary.mqh> #include "..\MTester\Mposition.mqh" //+------------------------------------------------------------------+ //| Auxiliary control containing the necessary fields | //+------------------------------------------------------------------+ class CDiffValues : public CObject { public: double all; double sell; double buy; CDiffValues(void) : all(0), buy(0), sell(0) { } }; //+------------------------------------------------------------------+ //| Statistical analysis class | //+------------------------------------------------------------------+ class CTradeAnalize { private: CDictionary m_values; public: void CalculateValues(CArrayObj* history); double GetStatistic(ENUM_MSTAT_TYPE type, ENUM_MSTATE_DIRECT direct); }; //+------------------------------------------------------------------+ //| Calculates the value of statistic | //+------------------------------------------------------------------+ double CTradeAnalize::GetStatistic(ENUM_MSTAT_TYPE type, ENUM_MSTATE_DIRECT direct) { CDiffValues* value = m_values.GetObjectByKey(type); switch(direct) { case MSTATE_DIRECT_ALL: return value.all; case MSTATE_DIRECT_LONG: return value.buy; case MSTATE_DIRECT_SHORT: return value.sell; } return EMPTY_VALUE; } //+------------------------------------------------------------------+ //| Calculates the number of deals for each direction | //+------------------------------------------------------------------+ void CTradeAnalize::CalculateValues(CArrayObj *history) { m_values.Clear(); for(int i = 0; i < MSTAT_ELEMENTS_TOTAL; i++) m_values.AddObject(i, new CDiffValues()); CDiffValues* profit = m_values.GetObjectByKey(MSTAT_PROFIT); CDiffValues* wins_money = m_values.GetObjectByKey(MSTAT_ALL_WINS_MONEY); CDiffValues* loss_money = m_values.GetObjectByKey(MSTAT_ALL_LOSS_MONEY); CDiffValues* total_traders = m_values.GetObjectByKey(MSTAT_TRADERS_TOTAL); CDiffValues* win_traders = m_values.GetObjectByKey(MSTAT_WIN_TRADERS); CDiffValues* loss_traders = m_values.GetObjectByKey(MSTAT_LOSS_TRADERS); CDiffValues* max_profit = m_values.GetObjectByKey(MSTAT_MAX_PROFIT); CDiffValues* max_loss = m_values.GetObjectByKey(MSTAT_MAX_LOSS); CDiffValues* pf = m_values.GetObjectByKey(MSTAT_PROFIT_FACTOR); CDiffValues* mexp = m_values.GetObjectByKey(MSTAT_MATH_EXP); total_traders.all = history.Total(); for(int i = 0; i < history.Total(); i++) { CMposition* pos = history.At(i); profit.all += pos.Profit(); if(pos.PositionType() == POSITION_TYPE_BUY) { if(pos.Profit() > 0) { win_traders.buy++; wins_money.buy += pos.Profit(); } else { loss_traders.buy++; loss_money.buy += pos.Profit(); } total_traders.buy++; profit.buy += pos.Profit(); } else { if(pos.Profit() > 0) { win_traders.sell++; wins_money.sell += pos.Profit(); } else { loss_traders.sell++; loss_money.sell += pos.Profit(); } total_traders.sell++; profit.sell += pos.Profit(); } if(pos.Profit() > 0) { win_traders.all++; wins_money.all += pos.Profit(); } else { loss_traders.all++; loss_money.all += pos.Profit(); } if(pos.Profit() > 0 && max_profit.all < pos.Profit()) max_profit.all = pos.Profit(); if(pos.Profit() < 0 && max_loss.all > pos.Profit()) max_loss.all = pos.Profit(); } mexp.all = profit.all/total_traders.all; mexp.buy = profit.buy/total_traders.buy; mexp.sell = profit.sell/total_traders.sell; pf.all = wins_money.all/loss_money.all; pf.buy = wins_money.buy/loss_money.buy; pf.sell = wins_money.sell/loss_money.sell; }
The calculation itself is performed by the CalculateValues method. It should be passed a CArrayObj array, containing CMposition controls. But where does this array of virtual positions come from?
The fact is that the CAnalyzePanel class contains another class — CListPass. It is the one that loads the zip archive and creates a collection of passes. This class is very simple:
//+------------------------------------------------------------------+ //| Optimazer.mq5 | //| Copyright 2017, MetaQuotes Software Corp. | //| http://www.mql5.com | //+------------------------------------------------------------------+ #include <Zip\Zip.mqh> #include <Dictionary.mqh> #include "..\MTester\MTrade.mqh" //+------------------------------------------------------------------+ //| Stores the list of optimization passes | //+------------------------------------------------------------------+ class CListPass { private: CZip m_zip_passes; // Archive of all optimization passes CDictionary m_passes; // Loaded historical positions public: bool LoadOptimazeFile(string file_name, int file_common = FILE_COMMON); int PassTotal(void); CArrayObj* PassAt(int index); }; //+------------------------------------------------------------------+ //| Loads the list of optimization passes from a zip archive | //+------------------------------------------------------------------+ bool CListPass::LoadOptimazeFile(string file_name,int file_common=FILE_COMMON) { m_zip_passes.Clear(); if(!m_zip_passes.LoadZipFromFile(file_name, file_common)) { printf("Failed load optimization file. Last Error"); return false; } return true; } //+------------------------------------------------------------------+ //| The number of passes | //+------------------------------------------------------------------+ int CListPass::PassTotal(void) { return m_zip_passes.TotalElements(); } //+------------------------------------------------------------------+ //| Returns the list of deals of the pass with the specified index | //+------------------------------------------------------------------+ CArrayObj* CListPass::PassAt(int index) { if(!m_passes.ContainsKey(index)) { CZipFile* zip_file = m_zip_passes.ElementAt(index); uchar array[]; zip_file.GetUnpackFile(array); CMtrade* trade = new CMtrade(); trade.FromCharArray(array); m_passes.AddObject(index, trade); } CMtrade* trade = m_passes.GetObjectByKey(index); //printf("Total Traders: " + (string)trade.History.Total()); return &trade.History; }
As it can be seen, the CListPass class loads the optimization archive, but does not unpack it. This means that all data are stored in a compressed form even in the computer's memory, which saves the computer's RAM. The requested pass is unpacked and converted into a CMtrade object, after which it is saved in the internal storage in an unpacked form. The next time this control is called, unpacking will not be required.
Again, refer to the CAnalyzePanel class. We now know where the positions are loaded from (the CListPass class) and how their statistics are calculated (the CTradeAnalyze class). After creating the graphical controls, it remains to fill them with the correct values. This is done by the CAnalyzePanel::PlotStatistic(void) method:
//+------------------------------------------------------------------+ //| Displays the statistics | //+------------------------------------------------------------------+ void CAnalyzePanel::PlotStatistic(void) { if(m_stat_descr.Total() == 0) CreateStatElements(); CArrayObj* history = m_passes.PassAt(m_curr_pass-1); m_analize.CalculateValues(history); for(int i = 0; i < MSTAT_ELEMENTS_TOTAL; i++) { ENUM_MSTAT_TYPE stat_type = (ENUM_MSTAT_TYPE)i; //-- all trades CElChart* el = m_stat_all.At(i); string v = ValueToString(m_analize.GetStatistic(stat_type, MSTATE_DIRECT_ALL), stat_type); el.Text(v); //-- long trades el = m_stat_long.At(i); v = ValueToString(m_analize.GetStatistic(stat_type, MSTATE_DIRECT_LONG), stat_type); el.Text(v); //-- short trades el = m_stat_short.At(i); v = ValueToString(m_analize.GetStatistic(stat_type, MSTATE_DIRECT_SHORT), stat_type); el.Text(v); } }
All the basic controls required for the operation of the analyzer panel have been considered. The description turned out to be inconsistent, but such is the essence of programming: all elements are interconnected, and sometimes all of them have to be described simultaneously.
So, it is now time to start the analyzer on the chart. Before doing that, make sure the zip archive with the optimization results is present in the FILE_COMMON directory. By default, the analyzer loads the "Optimization.zip" file, which must be located in the common directory.
The greatest effect of the implemented features can be seen when switching the passes. The chart and statistics are automatically updated. The screenshot below demonstrates this moment:
Fig. 8. Switching passes in the analyzer of mathematical calculations
For a better understanding of the panel operation, here is its graphical scheme with tooltips: main controls are outlined with frames indicating the class and methods responsible for each group of controls:
Fig. 9. Main controls of the interface
In conclusion, let us describe the structure of the resulting project. All source codes are located in the MTester.zip archive. The project itself is located in the MQL5\Experts\MTester folder. However, like any other complex program, the project requires the inclusion of additional libraries. The ones unavailable in the standard MetaTrader 5 package are present in this archive in the MQL5\Include folder. First of all, it is the CPanel graphical library (location: MQL5\Include\Panel). Also, the library for working with zip archives (MQL5\Include\Zip) and the class for organizing an associative array are included (MQL5\Include\Dictionary). For the convenience of users, two MQL5 projects have been created. This is a new feature of MetaTrader 5, which was implemented recently. The first project is called MTester and contains the strategy tester and the strategy itself, which is based on the intersection of moving averages (SmaSample.mq5). The second project is called MAnalyzer and contains the source code of the analyzer panel.
Apart from the source codes, the archive contains the Optimization.zip file with optimization results, with about 160 passes of the strategy on the test data. This allows quickly checking the functionality of the passes analyzer without the need to perform a new optimization. The file is located in MQL5\Files.
Conclusion
In conclusion, here are the brief theses of the materials described in the article.
- The math calculations tester has a high speed due to the lack of the trading environment simulation. This makes it a good basis to create a custom high-performance algorithm for testing simple trading strategies. However, due to the lack of any control over the correctness of the performed trade operations, it is possible to inadvertently "peep into the future" — refer to quotes that have not arrived yet. Errors in such "Grails" are quite difficult to identify, but this is the price for high performance.
- It is not possible to access the trade environment from within the math calculations tester. Consequently, it is impossible to retrieve the quotes of the desired instrument. Therefore, in this mode, it is necessary to manually download the required data in advance, as well as to use custom libraries for calculating the indicators. The article showed how to prepare the data, how to compress them efficiently, and how to integrate them into the program execution module. This technique can also be useful to all those who want to distribute additional data with the program, which are required for its operation.
- In the math calculations mode, there is also no access to standard indicators. Therefore, it is necessary to calculate the necessary indicators manually. However, the speed is also very important. This makes the manual calculation of the indicators within the expert not just the only, but also the best solution in terms of speed. Fortunately, the ring buffer library is able to provide an efficient calculation of all necessary indicators in a constant time.
- The frame generation mode in MetaTrader 5 is a powerful, albeit a complicated mechanism, which provides users with great opportunities in writing custom analysis algorithms. For example, it is possible to create a custom strategy tester, which has been shown in this article. To utilize the full potential of the frame generation mode, you need to be able to work with binary data to the full extent. It is through the ability to work with this data type that it becomes possible to generate complex data — for example, a list of positions. The article showed: how to create a complex custom data type (the CMPosition class of positions); how to convert it into a byte representation and add it to the frame; how to get an array of bytes from the frame and convert them back into the custom list of positions.
- One of the most important parts of the strategy tester is the data storage system. Obviously, the amount of data obtained during testing is huge: each testing can include hundreds or even thousands of passes, and each pass includes a lot of deals, that can easily reach up to several tens of thousands. The success of the whole project depends on how effectively this information is stored and distributed. Therefore, a zip archive was selected. Due to the fact that MQL5 has a powerful and fast library for working with this type of files, it becomes easy to organize a custom file storage for optimizations passes. Each optimization is a single zip file containing all passes. Each pass is represented by a single compressed file. Archiving leads to a high data compression, which make even a large-scale optimization have a modest size.
- Creating a custom strategy tester is not sufficient. A separate subsystem is required for analyzing these optimization results. Such a subsystem is implemented in the form of the M-Tester Analyzer program. This is a separate program module, which loads the optimization results as a zip archive and outputs them to the chart, displaying the basic statistics for each pass. M-Tester Analyzer is based on several classes and the CPanel graphical library. It is a simple and convenient library, which can be used to quickly build a powerful graphical interface. Using the system library CGraphic, the analyzer displays an informative graph of the balance dynamics.
Despite the fact that an impressive result was obtained, and the tester based on mathematical calculations actually gained profit, it is still missing many necessary things. Here are some of the priority components that need to be added to the next versions.
- Information on symbol (name, tick value, symbol, spread, etc.). This information is necessary to calculate the possible commissions, spread and swap. It will also be required for calculation of profit in the deposit currency (currently, profit is calculated in points).
- Information on the strategy and its parameters for each pass. It is necessary to know not only the result of the strategy, but also all its parameter values. To do this, the generated report should also have an additional data type integrated into it.
- Controlling the correctness of the performed actions. At this stage, it is quite easy to "peep into the future", which will result in a "Grail", which has nothing to do with reality. At least a minimal control mechanism is necessary in the future versions. However, it is still difficult to determine how it should look like.
- Integration of the report generation mechanism with the real strategy tester. Nothing prevents us from converting the result obtained in the standard MetaTrader 5 strategy tester into the developed report format. This gives the ability to analyze reliable testing results using M-Trade Analyzer. Thus, there will be several testing systems and one analysis system.
- Further development of M-Trade Analyzer. The program currently has only the basic features. Those are clearly not sufficient to fully process the data. It is necessary to add additional statistics and separate balance graphs both for sell deals and buy deals. It would also be reasonable to learn how to save the history of deals in a text file, and then load it in Excel, for instance.
Translated from Russian by MetaQuotes Ltd.
Original article: https://www.mql5.com/ru/articles/4226
LifeHack for traders: Fast food made of indicators
Automatic construction of support and resistance lines
Controlled optimization: Simulated annealing
Testing patterns that arise when trading currency pair baskets. Part III
- Free trading apps
- Over 8,000 signals for copying
- Economic news for exploring financial markets
You agree to website policy and terms of use
congratulations!
your article is really great! really for professionals ;-))
why don't think to enable standard indicators to work on custom symbold imported as array?
it's really too time-expensive and difficult to rewrite indicatos to work on array
i also developed by myself the code to do virtual trading, and i'd like to use it using virtual indicators too, with similar syntax to original mql5 one
thanks