Discussion of article "Custom symbols: Practical basics" - page 2

 
Mr David Frederick Roberts #:

Thanks for the response.  This is very odd, I am indeed using real ticks mode and I wasn't aware that there is a fixed spread option in MT5 so I'm sure that I'm not (I'm more familiar with MT4).  I hadn't spotted that the spreads were the same for each bar, in fact looking through the full file it seems that they only change on 1 minute boundaries.  I assume I am correct in thinking that the calls to SymbolInfoDouble() should return the prices for the tick being processed as a result of the OnTick() event call - not some kind of M1 value?

I tried printing what I expected to be the actual tick time with milliseconds using the code below, but the milliseconds value is always zero...

EDIT:  As I said I'm relatively new to MQL5 - I've just found SymbolInfoTick(), maybe I should be using that instead of what I'm doing!  I'll try that and see what happens...


Ok, that made no difference, exactly the same results, but neither of them tying up with the "Symbol Ticks" dialog in the terminal.  I was wrong in saying that the milliseconds value was always zero - it is non-zero for exactly one minute, namely the one minute before midnight!

This seems to me to be a pretty important issue.  My trading EA is expecting to receive ticks with prices which match the source imported tick-data (as would anyone else's!).  I dare say there is something that I'm doing wrong, but if so I would like to know what it is.  If not then there is a very serious problem here.  I have attached a zip file with the test EA, a spreadsheet of results for the hour either side of midnight, and some screenshots of settings and ticks.

I'm afraid it's not clear how you create your custom symbol. The attached csv-files are the same (BTW, I don't know the reason you need two copies under different names), and both have wrong format: tick files should have the following columns to be imported into MT5 custom symbol: date, time, bid, ask, last, volume. So currently it's not possible to create custom symbol from your ticks without additional manipulations.

It would be more simple if you post images of imported ticks in MT5 UI and the log of your EA side by side. The presented images above are incomplete and the ticks looks different between the images and compared to the csv-files.

 
Stanislav Korotky #:

I'm afraid it's not clear how you create your custom symbol. The attached csv-files are the same (BTW, I don't know the reason you need two copies under different names), and both have wrong format: tick files should have the following columns to be imported into MT5 custom symbol: date, time, bid, ask, last, volume. So currently it's not possible to create custom symbol from your ticks without additional manipulations.

It would be more simple if you post images of imported ticks in MT5 UI and the log of your EA side by side. The presented images above are incomplete and the ticks looks different between the images and compared to the csv-files.

The tick data has been imported to MT5 as a custom symbol from an export from my MT4 'Tick Data Suite' using Dukascopy data (CSV files around 8GB in size). 

The csv files I sent are in no way intended to be imported as tick data, they are the outputs of my test EA (which was enclosed) which is recording the Bid and Ask prices and time for each OnTick event received.  The two are simply derived by different methods - one using SymbolInfoDouble(...) to get the Bid and Ask prices, and the other using SymbolInfoTick(...).  The fact that they are the same is simply proving to myself that there is no difference between the methods.

My whole point is that the Bid and Ask prices and times in the CSV files recorded by the test EA do not match the image of the imported ticks in the MT5 UI (which I also sent).  So...

  1. The times of the ticks in the UI screenshot do not match up with those recorded by the test EA - why not?
  2. The times recorded by the test EA have a zero millisecond component except during the one minute before midnight - why?
  3. Although the Bid prices recorded by the test EA are somewhere close to those in the MT5 UI, the Ask prices can be a lot different (and hence the spread) - why?
  4. The prices recorded by the test EA give a resultant spread which seems to change only on one minute boundaries - why?
  5. There are fewer OnTick events being recorded by the test EA than there are tick records in the MT5 UI - why?

It seems to me that there is something interfering with the tick data of the custom symbol on its way between the repository and the EA OnTick event.

Below I have attempted to match up, as closely as I can, a portion of the data recorded by the test EA (left) and the MT5 UI (right).  It is impossible to align them accurately because there are fewer ticks recorded by the EA than exist in the MT5 UI.  Also note the milliseconds value recorded by the test EA are zero until 23:59.

The highlighted event is probably the same tick but note that the test EA records different prices from the UI...

EA records Bid 1.16504 and Ask 1.16513 (spread = 9)

MT5 UI shows Bid 1.16504 and Ask 1.16518 (spread = 14) 

 
Mr David Frederick Roberts #:


You have a lot of "why"'s, but it's hard to give an answer (without speculations). There is a problem in your procedure or source code.

What you could do is to publish a small part of 8Gb of tick history, which is easy to import to a custom symbol and check your code.

Normally, SymbolInfoTick should receive exactly the same ticks (including milliseconds timing) you see in MT5 GUI for the same custom symbol.

Some ticks can be skipped in your EA if it takes too much time to process previous tick. This "too much" can be very short, because ticks can come very quickly. If you want to make sure all ticks are available in history, you should call CopyTicks function for the period since previous processed tick. But this is not the issue you should address right now - you should first find out what's wrong with the current approach.

 
Mr David Frederick Roberts #:

The tick data has been imported to MT5 as a custom symbol from an export from my MT4 'Tick Data Suite' using Dukascopy data (CSV files around 8GB in size). 

The csv files I sent are in no way intended to be imported as tick data, they are the outputs of my test EA (which was enclosed) which is recording the Bid and Ask prices and time for each OnTick event received.  The two are simply derived by different methods - one using SymbolInfoDouble(...) to get the Bid and Ask prices, and the other using SymbolInfoTick(...).  The fact that they are the same is simply proving to myself that there is no difference between the methods.

My whole point is that the Bid and Ask prices and times in the CSV files recorded by the test EA do not match the image of the imported ticks in the MT5 UI (which I also sent).  So...

  1. The times of the ticks in the UI screenshot do not match up with those recorded by the test EA - why not?
  2. The times recorded by the test EA have a zero millisecond component except during the one minute before midnight - why?
  3. Although the Bid prices recorded by the test EA are somewhere close to those in the MT5 UI, the Ask prices can be a lot different (and hence the spread) - why?
  4. The prices recorded by the test EA give a resultant spread which seems to change only on one minute boundaries - why?
  5. There are fewer OnTick events being recorded by the test EA than there are tick records in the MT5 UI - why?

It seems to me that there is something interfering with the tick data of the custom symbol on its way between the repository and the EA OnTick event.

Below I have attempted to match up, as closely as I can, a portion of the data recorded by the test EA (left) and the MT5 UI (right).  It is impossible to align them accurately because there are fewer ticks recorded by the EA than exist in the MT5 UI.  Also note the milliseconds value recorded by the test EA are zero until 23:59.

The highlighted event is probably the same tick but note that the test EA records different prices from the UI...

EA records Bid 1.16504 and Ask 1.16513 (spread = 9)

MT5 UI shows Bid 1.16504 and Ask 1.16518 (spread = 14) 

The ticks your EA is collecting seem to be simulated ticks and not real ticks. Check the Strategy Tester logs.
 
Alain Verleyen #:
The ticks your EA is collecting seem to be simulated ticks and not real ticks. Check the Strategy Tester logs.

Thank you Alain, I entirely agree.  The tick data is there in the UI but by the time it gets to the EA it looks as though it's taking the first Bid/Ask pair it finds and applies that spread for the duration of the next minute - or something.

Anyway I've had a look at the log (enclosed) but I'm not too sure what I expect to see, so I would be obliged if you could take a look and see if you can see what's going wrong.  I've also enclosed the generated CSV file for info.

Much thanks, David.

 
Mr David Frederick Roberts #:

Thank you Alain, I entirely agree.  The tick data is there in the UI but by the time it gets to the EA it looks as though it's taking the first Bid/Ask pair it finds and applies that spread for the duration of the next minute - or something.

Anyway I've had a look at the log (enclosed) but I'm not too sure what I expect to see, so I would be obliged if you could take a look and see if you can see what's going wrong.  I've also enclosed the generated CSV file for info.

Much thanks, David.

Your log reads "every tick generating" which means artificial ticks. It should be "generating based on real ticks". Please, consult with the tester documenation.

Again, if you want someone else to test your custom symbol, provide compatible CSV-file, not of your own format.

Strategy Testing - Algorithmic Trading, Trading Robots - MetaTrader 5 Help
Strategy Testing - Algorithmic Trading, Trading Robots - MetaTrader 5 Help
  • www.metatrader5.com
The Strategy Tester allows you to test and optimize trading strategies ( Expert Advisors ) before using them for live trading. During testing, an...
 
Stanislav Korotky #:

Your log reads "every tick generating" which means artificial ticks. It should be "generating based on real ticks". Please, consult with the tester documenation.

Again, if you want someone else to test your custom symbol, provide compatible CSV-file, not of your own format.

Thanks Stanislav, I do appreciate everyone's help but I'm getting nowhere.

I've checked the documentation and can find nothing to tell me why the tester would generate artificial ticks when it has a full set of tick data available!  In fact it states...

Testing and optimization on real ticks are as close to real conditions as possible. Instead of generated ticks based on minute data, it is possible to use real ticks accumulated by a broker. These are ticks from exchanges and liquidity providers.

I repeat that the CSV files are the output of my test EA which is simply writing the prices of the ticks it receives and have nothing to do with the tick data I have imported into the custom symbol - so their format is completely irrelevant.

Ok, what I have done...

  1. Exported EURUSD tick data from my MT4 "Tick Data Suite" in MT5 format ( Tick Data Suite (eareview.net)) which generates a huge CSV file (screenshot below).
  2. Imported that file to a custom symbol in MT5 which I have called EURUSD_TDS.
  3. Taken a screenshot of a portion of the resultant ticks as seen in the MT5 UI.
  4. Written a test EA which simply writes the Bid & Ask price and time in each OnTick() call to a CSV file (I should have called it a TXT file!).
  5. Run the test EA on EURUSD_TDS  using the settings shown in the screenshot I supplied in the ZIP file.
  6. Compared the results in the file with the ticks in the MT5 UI for the same dates - they don't match.
  7. Sought help on why they don't match.
  8. Discover that apparently the tester is generating artificial ticks - but I'm no further forward.



Tick Data Suite
  • eareview.net
Everything you need to make your Metatrader 4 backtests accurate, get 99% modeling quality and to immediately spot which expert advisor is worth your time. From download to backtest, everything has been streamlined in an easy-to-use format. Updated often and offering premium-level support, the Tick Data Suite also has an associated recurring...
 
Mr David Frederick Roberts #: I've checked the documentation and can find nothing to tell me why the tester would generate artificial ticks when it has a full set of tick data available!  In fact it states..

Because you have not selected "Every tick based on real ticks " in the the tester settings:


 
Fernando Carreiro #:

Because you have not selected "Every tick based on real ticks " in the the tester settings:


Aa, at last :-) 

Thank you Fernando, I suspected it was something stupid I was doing but I just couldn't see it!  As I said I'm far more familiar with MT4 (around 8 years) and I hadn't spotted there were two "£Every Tick" options in MT5.  I just started her up, saw it was set to "Every Tick" and assumed that if it had real ticks it would use them.

It had to be something simple and I thank you again for spotting it.

David.

 
Mr David Frederick Roberts #: Thank you Fernando, I suspected it was something stupid I was doing but I just couldn't see it!  As I said I'm far more familiar with MT4 (around 8 years) and I hadn't spotted there were two "£Every Tick" options in MT5.  I just started her up, saw it was set to "Every Tick" and assumed that if it had real ticks it would use them. It had to be something simple and I thank you again for spotting it.
You are welcome!
Reason: