Developing a Multi-Currency Expert Advisor (Part 29): Improving the Conveyor
1 420
0
Introduction
In the previous three parts, we deviated slightly from the main line of development, devoting our efforts to creating auxiliary tools that will be useful to us in one way or another in the future.
For example, in Part 28, we have expanded the capital management features of our multi-currency EA by developing and implementing a new software module — closing manager. It allowed tracking total profit or loss relative to a dynamically updated base balance and restarting all trading strategies when the set values were reached. Future plans included expanding its functionality by adding the ability to trail total profits and move the entire set of open positions to breakeven.
In Part 27, we created another component — a dialog that occupies the entire terminal chart area, capable of displaying multi-line text with flexible font settings and scrolling support. This tool made information visualization more convenient and clear. We added this component to the Adwizard library as a tool for displaying various types of EA runtime information.
In general, we have consolidated the previously adopted approach, which consists of a clear division of all program code into a library part (Adwizard repository) and the project part or parts (SimpleCandles and SymbolsInformer repositories).
Now let's go back to the results obtained in Part 25. There we reached a major milestone in the development of our multi-currency EA, having completed the creation of a universal automatic optimization system and successfully integrated a new SimpleCandles trading strategy into it. The entire process — from creating an optimization project in the database and launching multi-stage optimization on a cluster of agents to generating a ready-to-use final EA — was fully automated.
However, since the publication of this article, we have encountered a number of difficulties or inconveniences when using this system. This is not surprising, as this was only the first iteration of the development of the system as a whole. So let's look at some improvements that will allow us to use automatic optimization more effectively.
Mapping out the path
Let's recall the essence of the proposed conveyor for building the final trading EA. In the first stage, we want to conduct multiple optimization processes of one trading strategy for different symbols, timeframes, and other parameters in the MetaTrader 5 strategy tester. From the obtained optimization results, we will select quite a few good ones for each symbol (if there are any, of course). Let's call them single instances of a trading strategy.
In the second stage, we will conduct optimization, identifying the best groups from a small number of single instances of trading strategies. That is, from thousands of copies, we will leave a group of only 8-16 pieces for each symbol that showed the best results when working together. During the third stage, we will combine these best groups for loading and use in the final EA.
After a long development process, all the mentioned actions have been automated as much as possible. Now we need to manually specify the parameters for generating an optimization project, that is, essentially, a general scenario according to which automatic optimization will proceed. After its completion, you will need to perform several manipulations to launch the final EA on the trading account. However, the time currently spent on manual operations (from a few minutes) is nothing compared to the time during which an automatic optimization conveyor can operate without the need for intervention in the process (hours, days, or even weeks). This is the tool we started working with.
The first inconvenience appeared when after we finished testing the integration of one new strategy and decided to move on to another new strategy. It turned out that despite the declared division of the code into two independent parts (library and project), some connections between them still remained. Let's see where they occur and try to eliminate it.
Another difficulty was that the duration of individual optimization tasks performed within the optimization project pipeline could be quite long. It directly depends on the size of the time interval over which all optimizations are carried out. If we want to carry out optimization over a period of, for example, 5 years, then this process will take much more time compared to optimization over a period of 3 months. But as practice has shown, during genetic optimization, good combinations of strategy parameters can be found much earlier than the planned end of this process. Therefore, the overall conveyor time can be reduced by stopping the optimization processes after a selected period of time. Let's add to our system the ability to specify a time limit for completing each optimization task.
Finally, let's add a little convenience to monitoring the automatic optimization process by implementing the output of more detailed information about the current task rather than just its ID.
For clarity, let us walk through the entire process step by step creating the final EA, while stopping to make any desired corrections.
Creating a database
First you need to clone the following two repositories to MQL5/Shared Projects: Adwizard library repository and project repository. Find the details on cloning repositories from the MQL5 Algo Forge repository to your computer in this article.
You can also create your own repository as a project one, using the SimpleCandles repository as a template. We discussed how to do this in the articles dedicated to the transition to a new strategy. This will be necessary if you want to implement any of your own trading strategies. We will continue to use the SimpleCandles strategy and the project repository of the same name. It contains a project for creating a final multi-currency EA using this trading strategy.
When both repositories are in the terminal working folder at MQL5/Shared Projects, compile the EA file for creating an automatic optimization SimpleCandles/Optimization/CreateProject.mq5 and drag the compiled version that appears in the Terminal Navigator onto any chart. In the input parameters dialog, switch to the Inputs tab and see the following:
In the optimization database file parameter, we have specified the database name that we used in previous articles. Since we are planning to make some changes, including those affecting the structure of the optimization database, we cannot continue to use the old database. It is necessary to create a new one.
This is done very simply: just specify another desired database name in this input. If a file with this name does not exist, the Project Creation EA will automatically create an empty database with this name and the required table structure. To avoid having to change the default name value when running this EA again, let's change it to a new one in the EA's source file:
//+------------------------------------------------------------------+ //| Inputs | //+------------------------------------------------------------------+ sinput group "::: Database" sinput string fileName_ = "article.17607.db.sqlite"; // - Optimization database file sinput group "::: Project parameters - Basic" sinput string projectName_ = "SimpleCandles"; // - Name sinput string projectVersion_ = "1.00"; // - Version sinput string symbols_ = "GBPUSD,EURUSD,EURGBP"; // - Symbols sinput string timeframes_ = "H1,M30"; // - Timeframes ...
It is also more convenient to change the values of other parameters in the source code of this EA than in the parameter entry dialog when launching the EA. This is a crucial stage where we should think through the overall scenario of the automatic optimization conveyor, record it in the inputs of the project creation EA, and then run it once. In practice, we ran it more than once, since it is very difficult to fully determine all the parameters at once. Therefore, we move in iterations: we fix the parameter values in the source code, create a project, run automatic optimization, and if we see that something needs to be changed, we stop the process and return to the beginning.
What could go wrong? For example, we chose too long a time frame for testing, and rough estimates allow us to predict that the entire process will take several weeks. In this case, if we do not want to wait that long, we can try reducing the duration of the test time interval or, for example, reducing the number of different symbols for which the optimization passes will be repeated. To do this, we make changes to the inputs and create the project again. You can even delete the database file beforehand so that the newly created project is the only one in the new database, and not added to an existing one.
In general, changing the default values of these parameters in the project part is acceptable and quite justified: each project requires individual configuration, which is carried out within its framework.
The result of this step is a created database file with the specified name in the terminal's shared data folder. The created database contains information about the auto optimization project configuration (in the 'projects' table) for automatic optimization, which is a set of stages ('stages' table). Stages consist of jobs ('jobs' table), and within one job there can be one or more optimization tasks ('tasks' table). These relationships are shown in the figure by arrows:
All project tasks are in the Queued status, meaning they are queued for execution. Now you can move on to the next step — launching the automatic optimization conveyor, within which all created tasks will be converted into separate runs of optimization of stage EAs in the MetaTrader 5 strategy tester.
Launching the optimization conveyor
To do this, we will compile the auto optimization EA file SimpleCandles/Optimization/Optimization.mq5 and drag the compiled version that appears in the Terminal Navigator onto any chart. In the input parameters dialog, on the Inputs tab, we will see the following:
Here we see again that the default parameter values are the name of the previous database file. We can, of course, manually replace it now with a new name article.17607.db.sqlite, but practice has shown that this is inconvenient. The reason is that while setting up the conveyor we will most likely have to do several test runs first. And on each of them we will have to manually change the name of the database file to the current one.
However, if we try to do as described above, that is, change the name value in the code, we will find that these changes need to be made not in the project part, but in the library part. In the project part, the optimization EA file (SimpleCandles/Optimization/Optimization.mq5) simply includes the library file so that when compiling we get an executable file in the project folder:
#include "../Include/Adwizard/Experts/Optimization.mqh"
There is nothing else in it, so the value that is substituted by default is in the included library file. Let's fix this as follows: in the project file, before including the library file, we declare constants with the required default parameter values and at the same time correct the path to the included file by removing the Include folder name:
// Constants with default parameters for the project: // - File with the main database #define OPT_FILEMNAME "article.17607.db.sqlite" // - Path to the Python interpreter #define OPT_PYTHONPATH "C:\\Python\\Python312\\python.exe" #include "../../Adwizard/Experts/Optimization.mqh"
In the library section (Adwizard/Experts/Optimization.mqh), we will assume that these constants may not exist. In this case, we declare them with empty values assigned. Next, we use their values to substitute into the inputs:
// Create constants for default parameters, // if they are not defined in the project part #ifndef OPT_FILEMNAME #define OPT_FILEMNAME "" #endif #ifndef OPT_PYTHONPATH #define OPT_PYTHONPATH "" #endif sinput string fileName_ = OPT_FILEMNAME; // - File with the main database sinput string pythonPath_ = OPT_PYTHONPATH; // - Path to the Python interpreter
Now we can specify these parameters in the project part code without touching the library part. That is, it will be taken the same for different projects.
Limiting the time for completing tasks
As already mentioned, another difficulty was that the duration of individual optimization tasks performed within the optimization project pipeline could be quite long. To add a time limit on the execution of a single optimization task, we will need to make changes in several places.
First, we need to add inputs to the project creation EA file, through which we could specify the desired maximum execution time of tasks. These changes concern the project part. Secondly, in the 'tasks' table of the database, we need to add a field to store this value. Thirdly, the optimization EA code should provide support for using the maximum execution time value. These changes will be made in the library section.
Let's add two parameters to the SimpleCandles/Optimization/CreateProject.mq5 file to specify the maximum execution time of optimization tasks at the first and second stages.
//+------------------------------------------------------------------+ //| Inputs | //+------------------------------------------------------------------+ sinput group "::: Database" sinput string fileName_ = "article.17607.db.sqlite"; // - Optimization database file sinput group "::: Project parameters - Basic" sinput string projectName_ = "SimpleCandles"; // - Name sinput string projectVersion_ = "1.00"; // - Version sinput string symbols_ = "GBPUSD,EURUSD,EURGBP"; // - Symbols sinput string timeframes_ = "H1,M30"; // - Timeframes sinput group "::: Project parameters - Optimization interval" sinput datetime fromDate_ = D'2023-09-01'; // - Start date sinput datetime toDate_ = D'2024-01-01'; // - End date sinput group "::: Project parameters - Account" sinput string mainSymbol_ = "GBPUSD"; // - Main symbol sinput int deposit_ = 10000; // - Initial deposit sinput group "::: Stage 1. Search" sinput string stage1ExpertName_ = "Stage1.ex5"; // - Stage EA sinput string stage1Criterions_ = "6,6,6"; // - Optimization criteria for tasks sinput long stage1MaxDuration_ = 20; // - Max duration of tasks (с) sinput group "::: Stage 2. Grouping" sinput string stage2ExpertName_ = "Stage2.ex5"; // - Stage EA sinput string stage2Criterion_ = "6"; // - Optimization criterion for tasks sinput long stage2MaxDuration_ = 20; // - Max duration of tasks (с) //sinput bool stage2UseClusters_= false; // - Use clustering? sinput double stage2MinCustomOntester_ = 500; // - Min norm. profit sinput uint stage2MinTrades_ = 20; // - Min number of trades sinput double stage2MinSharpeRatio_ = 0.7; // - Min Sharpe coeff. sinput uint stage2Count_ = 8; // - Number of strategies in the group (1 - 16) sinput group "::: Stage 3. Final" sinput string stage3ExpertName_ = "Stage3.ex5"; // - Stage EA sinput ulong stage3Magic_ = 27183; // - Magic sinput bool stage3Tester_ = true; // - For the tester?
We cannot specify a maximum time for the third stage, since it uses a single pass of the strategy tester rather than optimization. Only upon its completion are the standardized volumes of opened positions calculated and the initialization line of the final EA is generated. If we interrupt this pass early, we will not be able to obtain its results. For the first and second stages, stopping the optimization process earlier will only result in a reduction in the total number of passes performed. If there are still quite a lot of them, then there is nothing to worry about.
To add a new field to the tasks table structure, we only need to add one SQL query to the optimization database db.opt.schema.sql structure:
-- Table: tasks DROP TABLE IF EXISTS tasks; CREATE TABLE tasks ( id_task INTEGER PRIMARY KEY AUTOINCREMENT, id_job INTEGER NOT NULL REFERENCES jobs (id_job) ON DELETE CASCADE ON UPDATE CASCADE, optimization_criterion INTEGER DEFAULT (7) NOT NULL, start_date DATETIME, finish_date DATETIME, max_duration INTEGER NOT NULL DEFAULT 0, status TEXT NOT NULL DEFAULT Queued CHECK (status IN ('Queued', 'Process', 'Done') ) );
Now let's move on to editing the library section. Basically, we will need to make changes to two files. First, in the optimization task class file Adwizard/Optimization/OptimizerTask.mqh add another field for the maximum duration to the structure for reading task parameters from the database:
//+------------------------------------------------------------------+ //| Optimization task class | //+------------------------------------------------------------------+ class COptimizerTask { protected: // ... public: // Data structure for reading a single row of a query result struct params { string expert; int optimization; string from_date; string to_date; int forward_mode; string forward_date; double deposit; string symbol; string period; string tester_inputs; ulong id_task; int optimization_criterion; long max_duration; } m_params; // ... };
Add it to the request to retrieve task information from the database:
//+------------------------------------------------------------------+ //| Get the next optimization task from the queue | //+------------------------------------------------------------------+ void COptimizerTask::Load(ulong p_id) { // Save task ID m_id = p_id; // Request to get optimization task from queue by ID string query = StringFormat( "SELECT s.expert," " s.optimization," " s.from_date," " s.to_date," " s.forward_mode," " s.forward_date," " s.deposit," " j.symbol," " j.period," " j.tester_inputs," " t.id_task," " t.optimization_criterion," " t.max_duration" " FROM tasks t" " JOIN" " jobs j ON t.id_job = j.id_job" " JOIN" " stages s ON j.id_stage = s.id_stage" " WHERE t.id_task=%I64u;", m_id); // Open the database if(DB::Connect(m_fileName)) { // Execute the request int request = DatabasePrepare(DB::Id(), query); // ... } }
In the method for checking that the task is complete, we will add a block of code that, if a non-zero value is specified for the maximum duration of the task, will get the elapsed time from the database since the start of the task and compare it with the specified maximum. If the current duration has already exceeded the maximum, the task's forced stop method is called:
//+------------------------------------------------------------------+ //| Task completed? | //+------------------------------------------------------------------+ bool COptimizerTask::IsDone() { // If there is no current task, then everything is done if(m_id == 0) { return true; } // Result bool res = false; // If this is the EA optimization task if(m_type == TASK_TYPE_EX5) { // Check if the strategy tester has finished its work res |= MTTESTER::IsReady(); // If the tester is running and the maximum duration is specified, then if(!res && m_params.max_duration > 0) { // Request to get the elapsed execution time of the current task string query = StringFormat("SELECT unixepoch(datetime()) - unixepoch(start_date) AS duration" " FROM tasks" " WHERE id_task=%I64u;", m_id); // Get the execution time in seconds DB::Connect(m_fileName); long duration = StringToInteger(DB::GetValue(query)); DB::Close(); // If the execution time is greater than the maximum allowed, if(duration > m_params.max_duration) { // Stop the task Stop(); } } // If this is a task to run a Python program, then } else if(m_type == TASK_TYPE_PY) { // ... } } else { res = true; } return res; }
In Adwizard/Optimization/OptimizationProject.mqh, add passing the maximum duration parameter to the task creation methods:
//+------------------------------------------------------------------+ //| Optimization project class | //+------------------------------------------------------------------+ class COptimizationProject { public: // ... // Add new tasks to the database for the specified optimization criteria COptimizationProject* AddTasks(string p_criterions, long p_maxDuration = 0); COptimizationProject* AddTasks(string &p_criterions[], long p_maxDuration = 0); // ... }; //+------------------------------------------------------------------+ //| Add new tasks to the database for the specified | //| optimization criteria in one string | //+------------------------------------------------------------------+ COptimizationProject* COptimizationProject::AddTasks(string p_criterions, long p_maxDuration) { // Array for optimization criteria string criterions[]; StringReplace(p_criterions, ";", ","); StringSplit(p_criterions, ',', criterions); return AddTasks(criterions, p_maxDuration); } //+------------------------------------------------------------------+ //| Add new tasks to the database for the specified | //| optimization criteria in the array | //+------------------------------------------------------------------+ COptimizationProject* COptimizationProject::AddTasks(string &p_criterions[], long p_maxDuration) { // For each job of the current stage FOREACH_AS(m_stage.jobs, m_job) { // For each optimization criterion FOREACH(p_criterions) { // Create a new task object for the job m_task = new COptimizationTask(0, m_job, (int) p_criterions[i], p_maxDuration); // Insert it into the optimization database m_task.Insert(); // Add it to the array of all tasks APPEND(m_tasks, m_task); // Add it to the current job's tasks array APPEND(m_job.tasks, m_task); } } return &this; }
In the Adwizard/Optimization/OptimizationTask.mqh file, all we have to do is add an additional field to the COptimizationTask class and ensure its initialization in the constructor and in the method for inserting data into the 'tasks' table in the database:
//+------------------------------------------------------------------+ //| Optimization task class | //+------------------------------------------------------------------+ class COptimizationTask { public: ulong id_task; // task ID ulong id_job; // job ID int optimization; // Optimization criterion long maxDuration; // Max duration string status; // Task status COptimizationJob* job; // The job for the task will be launched for // Constructor COptimizationTask(ulong p_taskId = 0, COptimizationJob* p_job = NULL, int p_optimization = 6, long p_maxDuration = 0, string p_status = "Done"); // Create a task in the database void Insert(); }; //+------------------------------------------------------------------+ //| Constructor | //+------------------------------------------------------------------+ COptimizationTask::COptimizationTask(ulong p_taskId = 0, COptimizationJob* p_job = NULL, int p_optimization = 6, long p_maxDuration = 0, string p_status = "Done") : id_task(p_taskId), job(p_job), id_job(!!p_job ? p_job.id_job : 0), optimization(p_optimization), maxDuration(p_maxDuration), status(p_status) {} //+------------------------------------------------------------------+ //| Create a task in the database | //+------------------------------------------------------------------+ void COptimizationTask::Insert() { string query = StringFormat("INSERT INTO tasks " " VALUES (NULL,%I64u,%d,NULL,NULL,%I64d,'%s');", id_job, optimization, maxDuration, status); id_task = DB::Insert(query); PrintFormat(__FUNCTION__" | %s -> %I64u", query, id_task); } //+------------------------------------------------------------------+
Now we can compile two advisors that can be used to create optimization projects in the database, specifying the maximum possible execution time of optimization tasks in the first two stages of the conveyor (CreateProject.ex5) and launch the conveyor for execution (Optimization.ex5).
Display data during optimization
Since it was previously more important for us to ensure the correct operation of the automatic optimization conveyor EA, the issue of displaying the process info was secondary. We used the standard Comment() function, which allows displaying small text in the upper left corner of the chart with the running EA. The only thing displayed was the ID of the current optimization task. Now that the optimization EA's main work is more or less established, we can move on to less important things. In addition, we now have a ready-made component for more flexible text output on the EA chart — CConsoleDialog class. As you might remember, the object of this class allows us to create a dialog box that expands to cover the entire chart, with collapse and close buttons, and displays scrollable and scalable multi-line text. Let's use it.
To do this, we need to add the following. In the included library file of the optimization EA (Adwizard/Experts/Optimization.mqh), we need to create a global pointer to an object of this class, and in the initialization function, create the dialog object itself and call its launch method:
CConsoleDialog *dialog; // Dialog for displaying text with data //+------------------------------------------------------------------+ //| Expert initialization function | //+------------------------------------------------------------------+ int OnInit() { // If the database file is not specified, then exit if(fileName_ == "") { PrintFormat(__FUNCTION__" | ERROR: Set const OPT_FILEMNAME with filename of DB in project", 0); return INIT_FAILED; } // Create an optimizer optimizer = new COptimizer(fileName_, pythonPath_); // Create and launch a dialog to display information dialog = new CConsoleDialog(); dialog.Create(__FILE__); dialog.Run(); // Create the timer and start its handler EventSetTimer(2); OnTimer(); return(INIT_SUCCEEDED); }
In the timer handler, we add the transfer of new text to the dialog object received from the optimizer object:
//+------------------------------------------------------------------+ //| Expert timer function | //+------------------------------------------------------------------+ void OnTimer() { // Start the optimizer handling optimizer.Process(); dialog.Text(optimizer.Text()); }
Let's add a chart event handling function, OnChartEvent(), that passes events to the dialog object to ensure user interaction:
//+------------------------------------------------------------------+ //| Event handling | //+------------------------------------------------------------------+ void OnChartEvent(const int id, // event ID const long & lparam, // event parameter of the long type const double & dparam, // event parameter of the double type const string & sparam) { // event parameter of the string type if(!!dialog && !IsStopped()) { dialog.ChartEvent(id, lparam, dparam, sparam); } }
Make sure to delete the created dialog object, when the EA finishes working, and redraw the chart:
//+------------------------------------------------------------------+ //| Expert deinitialization function | //+------------------------------------------------------------------+ void OnDeinit(const int reason) { PrintFormat(__FUNCTION__" | Reason: %d", reason); EventKillTimer(); // Remove the optimizer if(!!optimizer) { delete optimizer; } // Remove the dialog if(!!dialog) { dialog.Destroy(); delete dialog; ChartRedraw(); } }
Now we need to generate the desired text that will be displayed on the screen during the optimization. Implement this in the Text() method of the optimizer object in Adwizard/Optimization/Optimizer.mqh:
//+------------------------------------------------------------------+ //| Information about the current optimization state | //+------------------------------------------------------------------+ string COptimizer::Text(void) { string text = ""; // Get the number of projects with different statuses DB::Connect(m_fileName); int process_projects_count = (int) DB::GetValue("SELECT count(status) FROM projects WHERE status = 'Process'"); int queued_projects_count = (int) DB::GetValue("SELECT count(status) FROM projects WHERE status = 'Queued'"); int done_projects_count = (int) DB::GetValue("SELECT count(status) FROM projects WHERE status = 'Done'"); int total_projects_count = process_projects_count + queued_projects_count + done_projects_count; DB::Close(); // Add this to the message text text += StringFormat("DB: %s | %d Projects (Process: %d, Queued: %d, Done: %d)\n", m_fileName, total_projects_count, process_projects_count, queued_projects_count, done_projects_count ); // If there is an active project if(process_projects_count > 0) { // Add information about the current task to the message text text += m_task.Text(); // And the total number of tasks in the queue if(m_totalTasks) text += StringFormat( "Total tasks in queue: %d\n", m_totalTasks); } return text; }
We will get information about the current task by calling the Text() method of the COptimizerTask class. Get information about the project, stage and job the current optimization task belongs to in the Adwizard/Optimization/OptimizerTask.mqh file. We will also calculate the time elapsed since the task was started and the remaining time until completion if the maximum allowable time for the task to complete is specified:
//+------------------------------------------------------------------+ //| Current task info | //+------------------------------------------------------------------+ string COptimizerTask::Text() { string text = ""; // If there is an active task if(m_params.id_task) { DB::Connect(m_fileName); // Add information about the project text += StringFormat("═════════════════════════════════════════════════════════════════════════\n" "PROJECT: %s v. %s\n%s\n\n", DB::GetValue("SELECT name FROM projects WHERE status = 'Process' LIMIT 1"), DB::GetValue("SELECT version FROM projects WHERE status = 'Process' LIMIT 1"), DB::GetValue("SELECT description FROM projects WHERE status = 'Process' LIMIT 1") ); // Request to get all information about the task string query = "SELECT s.name, s.expert, s.from_date, s.to_date, " " j.symbol, j.period, t.optimization_criterion, t.start_date, " " time(max_duration, 'unixepoch') AS max_duration," " time(unixepoch('now') - unixepoch(t.start_date), 'unixepoch') AS elapsed_time," " time(MAX(0, max_duration - (unixepoch('now') - unixepoch(t.start_date))), 'unixepoch') AS remaining_time" " FROM stages s" " JOIN" " projects p ON s.id_project = p.id_project AND" " p.status = 'Process' AND" " s.expert IS NOT NULL" " JOIN jobs j ON j.id_stage = s.id_stage" " JOIN tasks t ON t.id_job = j.id_job AND t.status = 'Process';"; // Execute the request int request = DatabasePrepare(DB::Id(), query); struct Row { string stage_name; string expert_name; string from_date; string to_date; string symbol; string timeframe; int optimization_criterion; string start_date; string max_duration; string elapsed_time; string remainig_time; } row; // If there is no error if(request != INVALID_HANDLE) { // Read data from the first line of the result and add it to the text if(DatabaseReadBind(request, row)) { text += StringFormat("TASK #%I64u:\n" " %10.10s │ %14.14s │ %-23s │ %6.6s │ %-3.3s │ %15.15s │ %-10.10s │ %-10.10s\n" "────────────┼────────────────┼─────────────────────────┼────────┼─────┼─────────────────┼────────────┼─────────────\n" " %10.10s │ %14.14s │ %s - %s │ %6.6s │ %-3.3s │ %15.15s │ %-10.10s │ %-10.10s \n\n" "═════════════════════════════════════════════════════════════════════════\n", m_id, "Stage", "Expert", "Testing period", "Symbol", "TF", "Criterion", "Max Durat.", "Remaining", row.stage_name, row.expert_name, row.from_date, row.to_date, row.symbol, row.timeframe, s_criterionNames[row.optimization_criterion], m_params.max_duration ? row.max_duration : "Unlimited", row.remainig_time ); } } DatabaseFinalize(request); DB::Close(); } return text; }
We will arrange the obtained data about the task in the form of a text table. At this point, the edits to ensure this functionality are complete, and we can move on to testing.
Test
Let's launch the SimpleCandles/Optimization/Optimization.ex5 optimization EA. We will see something like this:
For this screenshot, we set the maximum allowed execution time for a single optimization task to 3 minutes (180 seconds). We will be performing optimizations on three symbols and two timeframes over the 4-month period.
In the first stage, for each symbol timeframe combination, we will run the optimization 3 times. Therefore, in total we will need to complete 2 symbols * 3 timeframes * 3 optimizations = 18 tasks at the first stage.
In the second stage, we will perform one optimization run to select a good group for each symbol-timeframe combination. Thus, in the second stage you will need to complete 2 symbols * 3 timeframes = 6 tasks.
In the third stage, one final task is performed, so the total number of tasks is 18 + 6 + 1 = 25. We set a maximum time for all but the last one. If we roughly estimate the execution time of the last task as 3 minutes, then we can estimate how long the entire automatic optimization conveyor process will take: 25 * 3 = 75 minutes = 1 hour 15 minutes. This is, of course, much less than days or weeks. But let's try an even more extreme option.
we will limit the maximum execution time to just 20 seconds. In this case, the total time of the automatic optimization process will decrease further and will be about 25 * 20 seconds = 500 seconds = 8 minutes 20 seconds.
After this short time, all optimization tasks are completed. In the MetaTrader 5 terminal, where we ran this process, we can see the results of the last third stage task, which was a single pass of the Expert Advisor combining 2 symbols * 3 timeframes * 8 instances in a group = 48 single instances of the SimpleCandles trading strategy:
These results have not yet been standardized, meaning the drawdown over these four months was less than USD 500 with a maximum expected value of USD 1000 (10%). Therefore, we can increase the size of opened positions by approximately 2 times and not go beyond 10% drawdown during this period. This operation has already been completed at the end of the third stage, and information on position sizes, taking this standardization into account, has already been saved in the separate database of the final EA.
Also, in the third stage, neither a closing risk manager nor a closure manager was used.
According to the previously adopted convention, the name of the final EA database file is formed according to a strictly defined algorithm from the project name, the magic number specified in the project parameters, and the "test" suffix. In our particular case, after the automatic optimization process was completed, a file named SimpleCandles-27183.test.db.sqlite was created in the terminal common data folder.
Let's look at the contents of the final EA's database. It contains four tables, but we are only interested in the last two now:
The strategy_groups table features one entry for the formed strategy group with the id_group=1 ID. If we re-run the automatic optimization conveyor for the same project, new rows with different IDs will be added to this table. We need the value of the strategy group ID to specify it in the parameters of the final EA. In the strategies table, we have added single instances of trading strategies that belong to a specific group of strategies. The remaining tables will be used by the final EA when working on the trading account.
Let's compile the final EA SimpleCandles/SimpleCandles.mq5 file and run its testing with the existing strategy group ID, without auto update, disabled risk manager, closing manager, and magic number 27183:
We will get the following results:
Now the drawdown is already approaching the maximum expected value specified in the final EA settings, and the profit, accordingly, has increased proportionally to the increase in position sizes by approximately two times.
Conclusion
The work done in this article has significantly improved and made the process of using our automatic optimization conveyor more convenient. We did not add new functionality to the trading logic, but focused on solving practical problems we encountered while using the system.
We continued to separate the library and project parts. Now, to create a new project based on a different trading strategy, it is sufficient to describe its parameters in the project section, without making changes to the overall core. This process is not yet complete and will continue in the future.
The implementation of a mechanism for limiting the execution time of optimization tasks gave us a powerful tool for controlling the duration of the entire conveyor. We can now flexibly balance the depth of optimization and time spent by stopping processes at the first two stages once a reasonable limit is reached, which is critical for rapid testing and iterative development.
Integrating the CConsoleDialog component to display detailed information about the optimization transformed process monitoring from tracking task IDs into convenient and visual monitoring. We can now see in real time what stage, symbol, and timeframe the conveyor is running at, how much time remains until the current task is completed, and what the overall progress is.
Thus, the entire cycle — from creating a project database and launching the conveyor to obtaining a final EA with standardized parameters ready for testing — was successfully demonstrated in practice. An extreme run with very short time limits clearly demonstrated the viability of this approach: even with severely limited durations for individual tasks, the system is capable of completing a full optimization cycle and delivering a working result. This opens up opportunities for rapid prototyping and testing new trading strategies.
Thank you for your attention! See you soon!
Important warning
All results presented in this article and all previous articles in the series are based only on historical testing data and are not a guarantee of any profit in the future. The work within this project is of a research nature. All published results can be used by anyone at their own risk.
| # | Name | Version | Description | Recent changes |
|---|---|---|---|---|
| SimpleCandles | Project working folder (inside MQL5/Shared Projects) | |||
| 1 | SimpleCandles.mq5 | 1.03 | Final EA for parallel operation of several groups of model strategies. The parameters will be taken from the built-in group library. | Part 29 |
| └ Optimization | Project optimization EAs folder | |||
| 2 | CreateProject.mq5 | 1.05 | EA script for creating a project with stages, jobs and optimization tasks. | Part 29 |
| 3 | Optimization.mq5 | 1.03 | EA for projects auto optimization | Part 29 |
| 4 | Stage1.mq5 | 1.02 | Trading strategy single instance optimization EA (stage 1) | Part 25 |
| 5 | Stage2.mq5 | 1.01 | Trading strategies instances group optimization EA (stage 2) | Part 25 |
| 6 | Stage3.mq5 | 1.01 | The EA that saves a generated standardized group of strategies to an EA database with a given name. | Part 25 |
| └ Strategies | Project strategies folder | Part 25 | ||
| 7 | SimpleCandlesStrategy.mqh | 1.03 | SimpleCandles trading strategy class | Part 29 |
| Adwizard | Adwizard library folder (inside MQL5/Shared Projects) | |||
| └ Base | Base classes other project classes inherit from | |||
| 8 | Advisor.mqh | 1.04 | EA base class | Part 10 |
| 9 | Factorable.mqh | 1.06 | Base class of objects created from a string | Part 28 |
| 10 | FactorableCreator.mqh | 1.00 | Class of creators that bind names and static constructors of CFactorable descendant classes | Part 24 |
| 11 | Interface.mqh | 1.01 | Basic class for visualizing various objects | Part 4 |
| 12 | Receiver.mqh | 1.04 | Base class for converting open volumes into market positions | Part 12 |
| 13 | Strategy.mqh | 1.04 | Trading strategy base class | Part 10 |
| └ Database | Files for handling all types of databases used by project EAs | |||
| 14 | Database.mqh | 1.13 | Class for handling the database | Part 29 |
| 15 | db.adv.schema.sql | 1.00 | Final EA's database structure | Part 22 |
| 16 | db.cut.schema.sql | 1.00 | Structure of the truncated optimization database | Part 22 |
| 17 | db.opt.schema.sql | 1.06 | Optimization database structure | Part 29 |
| 18 | Storage.mqh | 1.01 | Class for handling the Key-Value storage for the final EA in the EA database | Part 23 |
| └ Experts | Files with common parts of used EAs of different type | |||
| 19 | Expert.mqh | 1.24 | The library file for the final EA. Group parameters can be taken from the EA database | Part 28 |
| 20 | Optimization.mqh | 1.06 | Library file for the EA that manages the launch of optimization tasks | Part 29 |
| 21 | Stage1.mqh | 1.19 | Library file for the single instance trading strategy optimization EA (Stage 1) | Part 23 |
| 22 | Stage2.mqh | 1.04 | Library file for the EA optimizing a group of trading strategy instances (Stage 2) | Part 23 |
| 23 | Stage3.mqh | 1.04 | Library file for the EA saving a generated standardized group of strategies to an EA database with a given name. | Part 23 |
| └ Optimization | Classes responsible for auto optimization | |||
| 24 | OptimizationJob.mqh | 1.00 | Optimization project stage job class | Part 25 |
| 25 | OptimizationProject.mqh | 1.00 | Optimization project class | Part 25 |
| 26 | OptimizationStage.mqh | 1.00 | Optimization project stage class | Part 25 |
| 27 | OptimizationTask.mqh | 1.01 | Optimization task class (creation) | Part 29 |
| 28 | Optimizer.mqh | 1.04 | Class for the project auto optimization manager | Part 29 |
| 29 | OptimizerTask.mqh | 1.06 | Optimization task class (conveyor) | Part 29 |
| └ Strategies | Examples of trading strategies used to demonstrate how the project works | |||
| 24 | HistoryStrategy.mqh | 1.00 | Class of the trading strategy for replaying the history of deals | Part 16 |
| 25 | SimpleVolumesStrategy.mqh | 1.11 | Class of trading strategy using tick volumes | Part 22 |
| └ Utils | Auxiliary utilities, macros for code reduction | |||
| 26 | ConsoleDialog.mqh | 1.01 | Class for displaying text data on a chart | Part 28 |
| 26 | ExpertHistory.mqh | 1.00 | Class for exporting trade history to file | Part 16 |
| 27 | Macros.mqh | 1.07 | Useful macros for array operations | Part 26 |
| 28 | MTTester.mqh | — | File for working with the strategy tester from the MultiTester library | Part 28 |
| 29 | NewBarEvent.mqh | 1.00 | Class for defining a new bar for a specific symbol | Part 8 |
| 30 | SymbolsMonitor.mqh | 1.01 | Class for obtaining information about trading instruments (symbols) | Part 28 |
| └ Virtual | Classes for creating various objects united by the use of a system of virtual trading orders and positions | |||
| 31 | Money.mqh | 1.01 | Basic money management class | Part 12 |
| 32 | TesterHandler.mqh | 1.07 | Optimization event handling class | Part 23 |
| 33 | VirtualAdvisor.mqh | 1.12 | Class of the EA handling virtual positions (orders) | Part 28 |
| 34 | VirtualChartOrder.mqh | 1.02 | Graphical virtual position class | Part 28 |
| 35 | VirtualCloseManager.mqh | 1.00 | Closing manager class | Part 28 |
| 36 | VirtualHistoryAdvisor.mqh | 1.00 | Trade history replay EA class | Part 16 |
| 37 | VirtualInterface.mqh | 1.00 | EA GUI class | Part 4 |
| 38 | VirtualOrder.mqh | 1.09 | Class of virtual orders and positions | Part 22 |
| 39 | VirtualReceiver.mqh | 1.04 | Class for converting open volumes to market positions (receiver) | Part 23 |
| 40 | VirtualRiskManager.mqh | 1.06 | Risk management class (risk manager) | Part 28 |
| 41 | VirtualStrategy.mqh | 1.09 | Class of a trading strategy with virtual positions | Part 23 |
| 42 | VirtualStrategyGroup.mqh | 1.04 | Class of trading strategies group(s) | Part 28 |
| 43 | VirtualSymbolReceiver.mqh | 1.00 | Symbol receiver class | Part 3 |
| Common/Files | Shared data folder of MetaTrader 5 terminals | |||
| 44 | article.17607.db.sqlite | — | Optimization database | Part 29 |
| 45 | SimpleCandles-27183.test.db.sqlite | — | Final EA database | Part 29 |
The source code is also available in SimpleCandles and Adwizard
Translated from Russian by MetaQuotes Ltd.
Original article: https://www.mql5.com/ru/articles/17607
Warning: All rights to these materials are reserved by MetaQuotes Ltd. Copying or reprinting of these materials in whole or in part is prohibited.
This article was written by a user of the site and reflects their personal views. MetaQuotes Ltd is not responsible for the accuracy of the information presented, nor for any consequences resulting from the use of the solutions, strategies or recommendations described.
Automating Classic Market Methods in MQL5 (Part 4): Mark Minervini's Trend Template
How to Research a Trading Idea: A Range Breakout Strategy Case Study
Features of Experts Advisors
Exporting MetaTrader 5 Open Positions to a Live-Refreshing HTML Dashboard
- Free trading apps
- Over 8,000 signals for copying
- Economic news for exploring financial markets
You agree to website policy and terms of use