Metatrader5 Strategy Tester Limitations on the Cloud Network: how to fix?

[siolibros]

Hi there, this message is about two important declared limitations of Metatrader5 Strategy Tester:

1 Customs symbols cannot be used in the Cloud Network

2 Genetic Optmization cannot be used in the Cloud Network

In the following rows I will describe the importance of these limitations for any serious professional user:

-1
Custom Symbols not allowed in the Cloud Network

Thanks to 21th july update, now it's possible to add&test custom symbols, importing external CSV price data history, wow.

https://www.metatrader5.com/en/releasenotes/terminal

But when running backtest optmization in the Cloud Network, imported price data history is not alllowed to transfer to Cloud Agents.

https://www.metatrader5.com/en/terminal/help/trading_advanced/custom_instruments Watch at the bottom box

In LAN mode it works fine!

We are used to put emphasis in backtesting optimization, using external data history.

If you want to make professional backtest optmization, you need to import high quality price data, like Dukascopy / Oanda.

No holes in their data..   instead price data downloaded from any retail Broker has a very poor quality.

So we need to transfer OUR price data history to the agents in the Cloud.

But it seems, by now, that Metaquotes does not allow it.

By the way, 14 years of price history data on EurUsd is about 15MB on m15 timeframe (the one I use)

The imported CSV into MT5 becomes hcc file extension into this path.

C:\mt5\Bases\Custom\history\eurusd2

When preparing the Backtest optimization, files are transferred to remote agents (in LAN), hcc becomes hcs (compressed file about 1/4 of the hcc original file), and stored in this path:

C:\mt5\Tester\bases\Broker_X\history\EURUSD2

So, the 15 MB of hcs file becomes 4 MB only.

I mean, why Metaquotes does not allow to transfer such a tiny file (4MB only) to Agents in the Cloud Network?

Running a simple backtest optimization, the journal shows immediately this row:

"CUSTOM SYMBOLS NOT ALLOWED IN CLOUD NETWORK"

We believe this is a great limitation....

From one side, the Cloud, there is the chance to get thousand of cpu computing.

On the other side, there is no chance to use affordable price data history.

What the hell?

-2
No Genetic Optimization in the Cloud Network

Watch here: https://www.metatrader5.com/en/terminal/help/mql5cloud_use

Which is the tech limitation to not allow Genetic in the Cloud Network?

regards
SIOLIBROS

[Stanislav Korotky]

siolibros:

-2
No Genetic Optimization in the Cloud Network

Watch here: https://www.metatrader5.com/en/terminal/help/mql5cloud_use

Which is the tech limitation to not allow Genetic in the Cloud Network?

Why do you think genetics are not available in the cloud? It surely works.

[Alain Verleyen]

siolibros:

Hi there, this message is about two important declared limitations of Metatrader5 Strategy Tester:

...

-2
No Genetic Optimization in the Cloud Network

Watch here: https://www.metatrader5.com/en/terminal/help/mql5cloud_use

Not sure where you read that genetic optimization is not possible with the Cloud, as Stanislav said it should work.

The main difference is it can't use all agents as there is some links between different passes :

During genetic optimization, only agents of one access point are used. It is connected with the specific features of the genetic algorithm.

So you will use "Europe 1" or "USA" or ... but not all at the same time.

[siolibros]

Alain Verleyen:

Not sure where you read that genetic optimization is not possible with the Cloud, as Stanislav said it should work.

The main difference is it can't use all agents as there is some links between different passes :

So you will use "Europe 1" or "USA" or ... but not all at the same time.

Hi guys, ty for quic reply

Before writing this post I already read all official guideline.

About 2nd question (GENETIC) the linked page explain:
"The entire power of the MQL5 Cloud Network is used only for complete slow optimization."

So, it seems there is no chance to use Genetic optimization with Cloud network.

Am I wrong?



About 1st question
Do you think there is a way to force using imported price data history in the cloud network?

tx

[Sergey Golubev]

Yes, wrong - it may be generic optimization as well.

-------------

This link?
MetaTrader 5 Help → MQL5 Cloud Network → How to Use

Features of the MQL5 Cloud Network

• The entire power of the MQL5 Cloud Network is used only for complete slow optimization.
• During genetic optimization, only agents of one access point are used. It is connected with the specific features of the genetic algorithm.
• The genetic optimization mode is automatically enabled when the total number of optimization steps exceeds 100 million.
• MQL5 Cloud Network can be used in 64 bit systems only.
• In addition to using the MQL5 Cloud Network, you can provide your CPU computing power in the network. To install the remote agents and include them into the network, use a special utility MetaTester.
• Read more about the MQL5 Cloud Network on the official site.

So,  genetic optimization works with Cloud Network but all the power of the Cloud is used in case of slow optimization only.
It means - if you want to use all power of the Cloud so you will try slow optimization.
If you do not want to get all the power of the Cloud so you may use generic optimization for example.

[siolibros]

Sergey Golubev:

This link?
MetaTrader 5 Help → MQL5 Cloud Network → How to Use

Features of the MQL5 Cloud Network

• The entire power of the MQL5 Cloud Network is used only for complete slow optimization.
• During genetic optimization, only agents of one access point are used. It is connected with the specific features of the genetic algorithm.
• The genetic optimization mode is automatically enabled when the total number of optimization steps exceeds 100 million.
• MQL5 Cloud Network can be used in 64 bit systems only.
• In addition to using the MQL5 Cloud Network, you can provide your CPU computing power in the network. To install the remote agents and include them into the network, use a special utility MetaTester.
• Read more about the MQL5 Cloud Network on the official site.

So,  genetic optimization works with Cloud Network but all the power of the Cloud is used in case of slow optimization only.
It means - if you want to use all power of the Cloud so you will try slow optimization.
If you do not want to get all the power of the Cloud so you may use generic optimization for example.

tx Sergey, I will try!

Could someone reply about question #1?

regards

[Carl Schreiber]

May I ask some additional question about how mt5 creates/uses 'selfmade' history data?

(I don't have mt5 on my laptop yet, so I can't try it myself.)

For mt4 I created my history tick-data file (fxt) for the backtests on the base of the birts script (eareview). My source are the tick-data from Dukascopy. I realised that whatever time frame I created all the fxt-files had the same size. I think this Is owed to the fact that each tick created its own 'data-point'. Therefore its size doesn't depend on the timeframe.

This is different if one uses the mt4 fxt files. As mt4 normally uses at least op,hi,lo,cl (maybe as well tick_volume..) to create 'artificial' ticks, which saves a lot of space and - I guess - calculation time.

Now you, siolibros, said:

By the way, 14 years of price history data on EurUsd is about 15MB on m15 timeframe (the one I use)

The imported CSV into MT5 becomes hcc file extension into this path.

C:\mt5\Bases\Custom\history\eurusd2

When preparing the Backtest optimization, files are transferred to remote agents (in LAN), hcc becomes hcs (compressed file about 1/4 of the hcc original file), and stored in this path:

C:\mt5\Tester\bases\Broker_X\history\EURUSD2

So, the 15 MB of hcs file becomes 4 MB only.

This really surprises me!

My fxt-file with tick data (EURUSD60_0.fxt) has about 10 years and has a size of ~3 GB!!

So - am I right that you aren't using tick data - as 15 years EurUsd m15 has only 15 MB and an the end becomes 4 MB.

Due to the data formate of the csv-file of the tickdata (see here):

<DATE>          <TIME>          <BID>   <ASK>   <LAST>  <VOLUME>
2017.07.03      00:03:47.212    1.14175 1.14210 0.00000 0       
2017.07.03      00:03:47.212    1.14168 1.14206 0.00000 0       
2017.07.03      00:03:47.717    1.14175 1.14206 0.00000 0       

This needs a lot more space for one tick than in the fxt-file.

---

Ok now here are my questions about mt5, custom symbols, third party quotes and tickdata:

1. siolibros: you aren't using tickdata - correct
2. Has anybody experience with the strategy tester of mt5 and tick data: "Every tick based on real ticks":
     a) What is the size of the relevant data files?
     b) Is there a difference between size of the tick-data files for the tester for e.g. m1, m30, h1, h4?

Thanks in advance!

Calli

[Enrique Dangeroux]

Carl Schreiber:

May I ask some additional question about how mt5 creates/uses 'selfmade' history data?

(I don't have mt5 on my laptop yet, so I can't try it myself.)

For mt4 I created my history tick-data file (fxt) for the backtests on the base of the birts script (eareview). My source are the tick-data from Dukascopy. I realised that whatever time frame I created all the fxt-files had the same size. I think this Is owed to the fact that each tick created its own 'data-point'. Therefore its size doesn't depend on the timeframe.

This is different if one uses the mt4 fxt files. As mt4 normally uses at least op,hi,lo,cl (maybe as well tick_volume..) to create 'artificial' ticks, which saves a lot of space and - I guess - calculation time.

Now you, siolibros, said:

This really surprises me!

My fxt-file with tick data (EURUSD60_0.fxt) has about 10 years and has a size of ~3 GB!!

So - am I right that you aren't using tick data - as 15 years EurUsd m15 has only 15 MB and an the end becomes 4 MB.

Due to the data formate of the csv-file of the tickdata (see here):

This needs a lot more space for one tick than in the fxt-file.

---

Ok now here are my questions about mt5, custom symbols, third party quotes and tickdata:

1. siolibros: you aren't using tickdata - correct
2. Has anybody experience with the strategy tester of mt5 and tick data: "Every tick based on real ticks":
     a) What is the size of the relevant data files?
     b) Is there a difference between size of the tick-data files for the tester for e.g. m1, m30, h1, h4?

Thanks in advance!

Calli

2): Size does not matter. Every tick means every single tick. There is more ticks within 15 timeframe vs 1m timeframe. File size is therefor not relevant. More ticks = bigger file. Test with a broker or two (MT5) and you will note different brokers, different quotes and different quote volume.

[Carl Schreiber]

Size does not matter.

Size does matter! Nobody knows the mt5 file formate and the size is a hint whether in the data file has single ticks or not but some data and a system to create virtual ticks (again)!

There is more ticks within 15 timeframe vs 1m timeframe.

No - as you should have the same amount of ticks within a 15 min bar and 15 bars of 1 min!

More ticks = bigger file.

Correct, but different timeframes built on the same tick data source should have the same size!

Test with a broker or two (MT5) and you will note different brokers, different quotes and different quote volume.

I have one tick data source (Dukascopy) so what you said it correct but irrelevant for me and my problem!

---

Again the formate of the data files for the tester is still not known, so practically nobody knows from where the ticks are coming - and therefore the size of the data files gives a hint.

Otherwise if you are optimizing a scalper on tick data you'll get 'optimal' parameters for the tick creating algorithm of mt5 but not for the reality - that why it matters!

[Fernando Carreiro]

siolibros: About 1st question: Do you think there is a way to force using imported price data history in the cloud network?

Custom Symbols are still something very new and recent on MT5/MQL5 and are still being developed further, so there are currently some limitations on its use:

New MetaTrader 5 Platform Build 1640: Creating and testing custom symbols

More possibilities will be available in future platform versions
The development of custom symbols has not completed yet, and more functions will be added in the next builds of the platform. You will be able to import history to custom symbols straight from Expert Advisors, as well as broadcast data (add quotes) of such symbols in real time.

Here are some other sources of information on Custom Symbols:

• Article: Creating and testing custom symbols in MetaTrader 5
• Documentation: MQL5 Reference - Custom Symbols

[Fernando Carreiro]

Carl Schreiber: Ok now here are my questions about mt5, custom symbols, third party quotes and tickdata:

1. 
   
2. Has anybody experience with the strategy tester of mt5 and tick data: "Every tick based on real ticks":
     a) What is the size of the relevant data files?
     b) Is there a difference between size of the tick-data files for the tester for e.g. m1, m30, h1, h4?

In MT4 FXT files, you have multiple very large files, each for its own time-frame.

In MT5, the files are stored as yearly history data files ("yyyy.hcs") and monthly tick data files ("yyyymm.tkc") files.

So there is no repetition for each time-frame, which greatly reduces the amount of disk-space as apposed to MT4 with its combination of HST files and rather large FTX files containing Dukascopy tick data.

Here is an example of the folder:

Tester/Bases ->

+---history
|   \---EURUSD
|           2016.hcs           1.932.297 bytes
|           2017.hcs           1.121.033 bytes
|           
\---ticks
    \---EURUSD
            201701.tkc        12.426.207 bytes
            201702.tkc         9.514.406 bytes
            201703.tkc        10.556.968 bytes
            201704.tkc         6.816.280 bytes
            201705.tkc         8.093.303 bytes
            201706.tkc         8.080.273 bytes
            201707.tkc         7.429.441 bytes