Speed Up Calculations with the MQL5 Cloud Network [ru]
MetaQuotes | 7 February, 2012 | Views: 2320
Multithreaded Testing in MetaTrader 5
You can long enumerate all the advantages of the new MetaTrader 5 trading platform and argue that it is better than other programs for technical analysis and trading in financial markets. There is one more indisputable argument in favor of the platform. And this last argument is the Strategy Tester in the MetaTrader 5 Client Terminal. In this article we describe its great features and explain why MetaQuotes Software Corp. developers are so proud of it.
The 5th generation client terminal has got not only a new powerful and fast MQL5 language for programming trading strategies, but also an absolutely new Strategy Tester that has been designed from scratch. The Tester is used not only for receiving the results of trading strategies tested on historical data, but also allows to optimize it, i.e. to find the optimal parameters.
Strategy optimization is a multiple run of a trading strategy on the same period of history with different sets of parameters on which it depends. This is a standard task of mass calculations, which can be parallelized, and as you might have guessed - the tester in MetaTrader 5 is multithreaded! What this actually means, we will now see at an example of optimization of an Expert Advisor from the standard distribution pack.
Test Conditions
For the purposes stated above we use a computer with Intel(R) Quad Q9300 (4 cores, 2.5 GHz) and 4 GB of memory with the Operating System Windows Vista 64 bit and MetaTrader 5 build 574.
The Expert Advisor Moving Average.mq5 from the standard delivery pack with the following settings is tested:
- Symbol: EURUSD H1
- Testing interval: from 2011.01.01 to 2011.10.01
- Price simulation mode: 1 minute OHLC (Open, High, Low and Close prices on 1-minute bars are used)
- Optimization type: slow complete algorithm, totally 14,040 passes
Optimization on Local Agents
First, run the optimization on local agents. We have four testing agents - the optimal number by the number of cores. Disable the use of remote agents from the local network and the agents of the MQL5 Cloud Network:
After the end of optimization, go to the Journal: 14,040 passes on 4 local agents took two hours and 44 seconds.
2012.01.31 18:11:20 Tester file cache E:\ProgramFiles\MetaTrader 5\tester\cache\Moving Average.EURUSD.H1.1.xml written 2012.01.31 18:11:18 Core 4 connection closed 2012.01.31 18:11:18 Core 3 connection closed 2012.01.31 18:11:18 Core 2 connection closed 2012.01.31 18:11:18 Core 1 connection closed 2012.01.31 18:11:18 Statistics locals 14040 tasks (100%), remote 0 tasks (0%), cloud 0 tasks (0%) 2012.01.31 18:11:18 Statistics optimization passed in 2 hours 00 minutes 44 seconds 2012.01.31 18:11:18 Tester optimization finished, total passes 14040
Now let's see how long it takes the agents of the MQL5 Cloud Network to solve the same task.
Optimization Using the MQL5 Cloud Network
This time we do not use local agents, instead we use only MQL5 Cloud Network agents. Click the "Start" button and watch the progress of the optimization. The video shows the process in real time.
During the optimization, each node of the MQL5 Cloud Network distributes tasks (single runs) to available agents. The optimization took only 79 seconds, giving the acceleration in the 90 (!) times. Traders may need to run hundreds of thousands of optimization passes in a reasonable time. With the MetaTrader 5 tester, you need only an hour for calculations in the MQL5 Cloud Network, while without the network you would spend a few days. Now with one click you can involve thousands of cores to solve a task. And it's available to everyone! But how does it work?
MQL5 Cloud Network Includes Thousands of Computers
The MQL5 Cloud Network consists of nodes - dedicated servers, to which testing agents connect to perform tasks. These nodes are managers (poolers), as they combine agents around the world into larger pools based on their geographical location. Being in the idle mode, each agent sends a message notifying that it is ready to perform a task. The interval between such messages depends on the current load of the MQL5 Cloud Network.
Each node of the network is at the same time treated by MetaTrader 5 as a point to access the MQL5 Cloud Network; a terminal connects to them using the MQL5.com account details. The list of servers of the MQL5 Cloud Network and the number of cloud agents available through them can be found in the terminal, the Tester window, tab "Agents".
An agent is free, that is in the idle mode, in case it is not busy performing its own local tasks received from a local computer or local network. While an agent is busy, it makes no attempt to take tasks from the MQL5 Cloud Network. Within several minutes after completing local calculations, the agent gets in touch with the nearest MQL5 Cloud Network node and offers its services. Thus, your testing agents are working on the network only if you do not need them. And, of course, the agents work on the network in accordance with the set schedule.
Thanks to the ease of installation and the minimum necessary settings of the MetaTrader 5 Agents Manager, thousands of testing agents are available in the network at any given time. The general statistics of MQL5 Cloud Network agents and completed tasks is available on the main page of the project at http://cloud.mql5.com.
Running Distributed Computing Using the MQL5 Cloud Network Agents
Like with conventional optimization, you need to set all the testing options and Expert Advisor's input parameters. Before that, do not forget to specify your MQL5.community login in the terminal settings and allow the use of the MQL5 Cloud Network. The four required steps are shown in the below figure.
Click the "Start" button and the optimization process starts. The terminal prepares a task for the testing agents, which includes:
- a compiled Expert Advisor file with the EX5 extension
- indicators and EX5 libraries that are enabled using the directives #property tester_indicator and #property tester_library (DLL's are definitely not allowed in the cloud)
- data files needed for the test, enabled using the directive #property tester_file
- testing/optimization conditions (the name of the financial instrument, testing interval, simulation mod, etc.)
- trading environment (symbol properties, trading conditions, etc.)
- the set of Expert Advisor parameters that form the entire set of required passes, i.e. tasks
The MetaTrader 5 terminal communicates with the nodes of the MQL5 Cloud Network and gives each node a separate package of tasks to perform specific passes. Each node is actually a proxy server, since it receives a task and a package of passes, and then begins to distribute these tasks to agents connected to it. In this case the files of Expert Advisors, indicators, libraries and data files are not stored on the hard drives of the MQL5 Cloud Network servers.
Also, EX5 files are not stored on hard disks of cloud agents for reasons of confidentiality. Data files are saved on a disk, but after optimization data files are deleted.
This is the whole procedure of communication between your client terminal and the MQL5 Cloud Network - actually, it sends packets of tasks to the network and waits for the results.
Synchronization in the Cloud and Distribution of History to Agents
Each node of the MQL5 Cloud Network keeps the history of the required symbols and sends it to the agents connected to it on demand. If it has no history of symbol XYZ from broker ABC, then the node automatically downloads the necessary history data from your terminal. Therefore, your terminal should be ready to provide such a story.
We recommend you to run a preliminary single test of a strategy on your computer before you send it to the MQL5 Cloud Network. This approach automatically provides downloading and synchronization of all the required history from a trading server.
As a rule, 4 to 8 agents are installed on a modern computer, but history data are stored in a single folder in the MQL5 Strategy Tester Agent installation directory. All cloud agents installed by one MQL5 Strategy Tester Agent manager, receive the history from this folder. If 8 agents are installed, and they are all available for the MQL5 Cloud Network, the required history is downloaded only once. This allows you to save traffic and hard disk space. Also it is convenient to carry out synchronization between cloud agents and nodes of the distributed computing network.
Thus, all the agents that perform optimization of a trading strategy in a given time interval and on a given symbol are automatically provided with the same synchronized history and market environment.
Warming up
How does optimization run on a local computer optimization? If you have 8 cores, usually 8 default local agents are available to you. When you click "Start", tasks are distributed to local agents, the required is downloaded (if necessary) and the process begins. In this case optimization start almost instantaneously. But if you distribute tasks to the MQL5 Cloud Network, the procedure changes a little.
Cloud agents are not permanently connected to the network managers, it is technically unjustified and costly for all reasons. Instead, the agents periodically ask MQL5 Cloud Network servers about whether there are any new tasks for them. This happens often enough to ensure the rapid mobilization of the required number of agents, and rare enough, so as not to overload the network traffic with such messages. So when you run optimization, you can see the growth in the number of agents that connect to the fulfillment of your tasks. This is the real-time process of how cloud agents access the MQL5 Cloud Network and receive tasks for certain passes.
If there are no tasks, agents contact managers quite rarely. But if an order to calculate thousands (tens of thousands) of tasks comes, the picture changes. We can say that the activity of the MQL5 Cloud Network increases, and after completing the task the number of applications of agents for new tasks reduces. And if after completing a task, for example, from Europe, an order for other tasks comes from Asia, the network will be ready for a quick start. You can call this behavior of the network "warming up".
Thus, the MQL5 Cloud Network is ready again to accept new tasks to perform them in the shortest possible time.
Use the MQL5 Cloud Network!
The phrase "Time is money" becomes even more topical with each passing year, and we cannot afford to wait for important computations for tens of hours or even days. At the time of this writing, the MQL5 Cloud Network provides increase of calculations in a hundred times. With its further increase, the gain in time can grow to a thousand times or more. In addition, the network of distributed computing allows you to solve not only strategy optimization tasks.
You can develop a program in MQL5 that implements massive mathematical calculations and requires a lot of CPU resources. The MQL5 language, in which programs for the MetaTrader 5 terminal are written, is very close to C++ and allows you to easily translate algorithms written in other high level languages.
An important feature of the MetaTrader 5 terminal tester is that hard mathematical tasks aimed at finding solutions with large sets of input variables are easily parallelized among testing agents. And you do not need to write any special code for that - just connect to the MQL5 Cloud Network of distributed computing!
Translated from Russian by MetaQuotes Software Corp.
Original article: http://www.mql5.com/ru/articles/341
Cloud is safe? Or My EA will get stolen if I use cloud?
thanks
You should read the article The Fundamentals of Testing in MetaTrader 5:
The Data Exchange between the Terminal and the Agent
...
The agents never record to the hard disk the EX5-files, obtained from the terminal (EA, indicators, libraries, etc.) for security reasons, so that a computer with a running agent could not use the sent data. All other files, including DLL, are recorded in the sandbox. In remote agents you can not test EAs using DLL.
The testing results are added up by the terminal into a special cache of results (the result cache), for a quick access to them when they are needed. For each set of parameters, the terminal searches the result cache for already available results from the previous runs, in order to avoid re-runs. If the result with such a set of parameters is not found, the agent is given the task to conduct the testing.
All traffic between the terminal and the agent is encrypted.
You should read the article The Fundamentals of Testing in MetaTrader 5:
I have tried the cloud, at the beginning, it worked very fast, but about 700 pass done, the cloud stop, no pass done any more. I have enough balance in my account.
It looks like the sandbox had not distributed the task to the cloud.
What I can do to make it fast?
Thanks.
I have tried the cloud, at the beginning, it worked very fast, but about 700 pass done, the cloud stop, no pass done any more. I have enough balance in my account.
It looks like the sandbox had not distributed the task to the cloud.
What I can do to make it fast?
Thanks
It's VERY surprising that nothing is mentioned about paying to use the Cloud Network. I just wanted to test it out one day not realizing that they were tracking usage and charging for it. Everything I had seen had said it was all still in beta, including the money component. I was VERY surprised when I could no longer use the Cloud Network because I had accumulated charges that exceeded the max limit.
Actually, you can install several agents in your computer, and use them for your own, without the need of any networks (internet, LAN, WAN or any cloud), but I found it even much more slow >_<
Here's how : run MetaTester.exe (MetaTester64.exe if you're using 64 bit), uncheck everything under "Overview" and "MQL5 Cloud Network" tabs, and add several agents in "Service" tab, after you're done adding agents, right click on list of agent and select "Export".
Open MT5 and Strategy Tester (Ctrl + R), under "Agents" tab, right click and select "Import" and import the list of agent and now you will see that you have several agents under "Remote". This agents actually installed in the same computer running as MT5 but the same MT5 consider this agent as remote.
Again, - maybe because I use Dual Core and installing lots of agents for it - when I test this, it does NOT speed up anything, it actually slowing down even more >_<.