debugging MQL5 EA using artificial tick data

[shloss]

Hi, 

I need to deb an MQL5 EA using tick data that I would create on purpose, so to understand a strange behaviour i have noticed during real testing.

If I use backtesting or real testing I may not find extreme situations that make the bug happen again.

I know in 2012 it was not possible to backtest in MT5 with tick data other than that provided by MT5 itself. https://www.mql5.com/en/forum/9600#comment_5457186

Has the situation changed?

If not this is still for me a big disadvantage of MT5 respect to MT4...

[Alain Verleyen]

Please don't double or triple post.

[shloss]

With last build 1643 it is theoretically possible to import custom history, bars or ticks (thanks Alain for telling me!).

https://www.mql5.com/en/forum/206432

Anyway, I can't use it because once imported, the tick history doesn't seem to be available for backtesting. The strategy tester tries to "syncronize history" and it seems not to allow to use the locally stored data.

Besides, when I reopen the custom symbol "Ticks" tab, the previously imported history is gone.

Same disappearing of the imported data happens with "Bars".

I guess the "custom symbol" feature is not completely implemented, anyone can confirm?

[Alain Verleyen]

shloss:

With last build 1643 it is theoretically possible to import custom history, bars or ticks (thanks Alain for telling me!).

https://www.mql5.com/en/forum/206432

Anyway, I can't use it because once imported, the tick history doesn't seem to be available for backtesting. The strategy tester tries to "syncronize history" and it seems not to allow to use the locally stored data.

Besides, when I reopen the custom symbol "Ticks" tab, the previously imported history is gone.

Same disappearing of the imported data happens with "Bars".

I guess the "custom symbol" feature is not completely implemented, anyone can confirm?

It's very new feature, I didn't try yet but it's very probably not complete.

[shloss]

Sorry, I have to correct part of my previous post, Bars and Ticks history does not disappear, you just have to "request" it again.

The sync error is still there:

DE 0 21:49:07.299 Symbols EURUSD_mql5: symbol to be synchronized

ER 0 21:49:07.300 Symbols EURUSD_mql5: symbol synchronized, 3544 bytes of symbol info received

IQ 0 21:49:07.300 History EURUSD_mql5: history synchronization started

HG 3 21:49:07.301 History no data synchronized, 37 bytes read

GO 2 21:49:07.301 History symbol EURUSD_mql5 synchronization error

RE 2 21:49:07.301 Tester cannot get history EURUSD_mql5,M1

I don't understand, what is this syncronization needed for when I have local history data?

[shloss]

I wonder if anyone has made it yet to backtest on the last version of MT5 with his own data (tick history even better!)...

I am sure I am not the only one badly needing that to work...

[Icham Aidibe]

shloss:

I wonder if anyone has made it yet to backtest on the last version of MT5 with his own data (tick history even better!)...

I am sure I am not the only one badly needing that to work...

Hi ! 

I've spend lot of time with that so I can share my experience : 

• Importing tick data in MQL5 is possible, but you won't find no tick datas - even datas provided by TickStory aren't correctly imported, it's probably due to the proprietary format (no tick flag for example)
• It's possible to use tick datas from the broker, but only a few months of data are well preserved, bad quality datas are replaced by every tick generation & few months only is fairly not enough to be forwarded

You have to rely the standard. Each days' a new one. 

[shloss]

Icham Aidibe:

Hi ! 

I've spend lot of time with that so I can share my experience : 

• Importing tick data in MQL5 is possible, but you won't find no tick datas - even datas provided by TickStory aren't correctly imported, it's probably due to the proprietary format (no tick flag for example)
• It's possible to use tick datas from the broker, but only a few months of data are well preserved, bad quality datas are replaced by every tick generation & few months only is fairly not enough to be forwarded

You have to rely the standard. Each days' a new one. 

Hi Icham,

actually I was able to import tick data. I used the same file format MT5 uses when you save its own tick data in csv.

What I can't do yet is backtesting my EA with that tick data. That is making me crazy. You can see above the kind of error I get.

Hopefully with the next version everything will be smooth...

[Icham Aidibe]

shloss:

Hi Icham,

actually I was able to import tick data. I used the same file format MT5 uses when you save its own tick data in csv.

What I can't do yet is backtesting my EA with that tick data. That is making me crazy. You can see above the kind of error I get.

Hopefully with the next version everything will be smooth...

Hi ! 

You succeded using external ticks datas ? How ?! 

Cause I m sure that from tickstory, I ve made no error during the process and it didn't work : tick flag wasn't fill 

Edit: on the other hand, external ticks datas aren t the broker one - duff if u expect to be more accurate 

[shloss]

Icham Aidibe:
Hi ! 

You succeded using external ticks datas ? How ?! 

Cause I m sure that from tickstory, I ve made no error during the process and it didn't work : tick flag wasn't fill 

Edit: on the other hand, external ticks datas aren t the broker one - duff if u expect to be more accurate 

I am succesfully importing files with this format (where you see spaces actually there is one TAB):

He seems to need the 5th and 6th columns even if they are all zeros (and they are zeros also in the data by Metaquotes, what's the use?):

[Icham Aidibe]

shloss:

I am succesfully importing files with this format (where you see spaces actually there is one TAB):

He seems to need the 5th and 6th columns even if they are all zeros (and they are zeros also in the data by Metaquotes, what's the use?):

Err ... 

CSV format is a well known format. But MT5 seems to need the tick flag (bid, ask or bid/ask) to know which kind of ticks it'll generate. 

Here's the ticks as MT5 need it : 

Here's tickstory with data formating matching MT5's one. No tick flags. No last prices (=0 used).

Then, add the header as MT5 read it : 

Import the csv in MT5 : 

As expected everything's correctly filled but the flag, volume and last.

Then run any EA on the custom symbol and ..... BEEP. Nothing happens. 

Retry with the "based on real tick mode" and ..... BEEP. Nothing happens.

So ... 

• Brokers delivers one year maximum of good tick data quality : definitively not enough to be forwarded 
• Importing data on 5 years or even 10 from another source (dukascopy is a reference), with 2 main disadvantages : 
• First, it doesn't work actually (unless I made a error in the procedure, please correct me) 
• Second, even if it worked, it's not the brokers data, none won't win in accuracy
• The only interest I find there is to collect good quality ticks from a year to another and use it in custom symbol, but brokers don't keep these huge files on their server - and so I won't in my laptop. 

That "based on real tick" mode is ... a try, a ideal, a utopia. markets aren't frozen.

qED.