Puzzled at how to prepare tick data for EA backtesting....

 

Hello Gents.. I am completely baffled at how to properly get tick data set up to in MT4 backtest my simple EA. I wish to get about only 6 months worth of EURUSD tick data to do some accurate backtesting, and I basically followed the instructions found here:


https://forums.babypips.com/t/how-i-get-99-9-modelling-quality-on-mt4/64592


I did the following:

- Successfully downloaded 6 months data using SQ tick downloader.

- Copied the entire relevant EURUSD folder (containing all the "bi5" files for all the days) over to the \TickStory Lite\Data folder.

- Ran Tickstory Lite (v1.5.3.0), configured it to point to my correct Mt4 platform foldersm then ran to convert that data to FXT and place it in my proper MT4 tester folder. That worked fine. I have a defunct demo account with "XM" Forex broker that no longer makes any server connection, so am using that as my testing MT4 platform. Tickstory Lite placed 3 files in that \tester\history folder - EURUSD1_0.fxt, EURUSD5_0.fxt, and EURUSD15_0.fxt. Each about 650 megs big. So things were looking up at this time.

- Tried to "launch" my XM MT4 platform from Tickstory Lite, but it gave an error - "Suitable launcher could not be found" for my MT4 version (which is Build 1090).

- Figured I'd just bypass that... couldn't see why it was so necessary to launch my MT4 platfrom using Tickstory Lite, and just launched it manually. Went to the Strategy Tester and tried a small backtest choosing only about 5 days of my data (selected M1). Clicked on "Symbol Properties" before running it and noticed it wasn't reflecting the spread figure I had specifically put in using the FXT Editor of TickStory Lite. Went ahead and tried the backtest anyway, finished in about 2 seconds so I knew something was quite wrong. Viewed the report and showed a modelling quality of "n/a". Tried again in "Visual Mode" and it was evaluating a whole bar at a time, and not tick by tick. I guess it was somehow using the built-in data already that was there within that XM platform and just ignoring all my new FXT data.


So there I am - stuck. Now I notice that my M1 650 meg FXT file is overwritten to just 213 KB. It overwrote all my data. I must be missing something very very basic here. I know there are supposed to be HST files involved in this process somehow as well, but the instructions I followed didn't mention anything about that at all... only the FXT files. TickStory Lite didn't seem to generate any HST files anyway - just the FXT ones. Could anyone offer any insight please as to where I went off the rails?


Thanks!

   Shawn


P.S: Am running Windows Vista.

How I get 99.9% modelling quality on MT4
How I get 99.9% modelling quality on MT4
  • 2014.05.02
  • forums.babypips.com
Now I have your attention, because it's a pretty big claim. After following this thread http://forums.babypips.com/free-forex-trading-systems/65546-millionaire-ea-417-million-profit-5-years.html I am offering my process for criticing by those who know better and as a starting point for those who don't. At the end I will provide a link to where...
 

You have to generate the fxt files again and mark them with the "Read only" Windows property (right click \ Properties \ Read only) so that MT4 does not overwrite them when starting each new backtest

On the other hand, each version of Tickstory is valid for certain MT4 builds. You have to check if you have the latest version of Tickstory, and if it supports build 1090 of MT4. If you do not launch MT4 through the Tickstory launcher, it will always mark the modeling quality as  n/a (and not as 99%), although this has no relevance.

Regards.

 
Jose Francisco Casado Fernandez:

You have to generate the fxt files again and mark them with the "Read only" Windows property (right click \ Properties \ Read only) so that MT4 does not overwrite them when starting each new backtest

On the other hand, each version of Tickstory is valid for certain MT4 builds. You have to check if you have the latest version of Tickstory, and if it supports build 1090 of MT4. If you do not launch MT4 through the Tickstory launcher, it will always mark the modeling quality as  n/a (and not as 99%), although this has no relevance.

Regards.

You have to generate the fxt files again and mark them with the "Read only" Windows property (right click \ Properties \ Read only) so that MT4 does not overwrite them when starting each new backtest


>> Thank you Jose, good tip!


On the other hand, each version of Tickstory is valid for certain MT4 builds. You have to check if you have the latest version of Tickstory, and if it supports build 1090 of MT4. If you do not launch MT4 through the Tickstory launcher, it will always mark the modeling quality as  n/a (and not as 99%), although this has no relevance.


>> I am using TickStory Lite v1.5.3.0 and I guess maybe this is the most up-to-date version of it? Attached is the error message shown when I try to launch MT4 from it. My MT4 platform is Build 1090 and I did try to replace it with an older version, but it would always automatically update - even if I added the "/skipudate" extension at the end of the run command in the shortcut. I would like to run an older version of MT4, but I don't know how to keep it from automatically being updated to 1090. If I could do that, then my current version of TickStory Lite *should* be abe to launch it.


Thank You!

    Shawn

Files:
 
shawnM:

You have to generate the fxt files again and mark them with the "Read only" Windows property (right click \ Properties \ Read only) so that MT4 does not overwrite them when starting each new backtest


>> Thank you Jose, good tip!


On the other hand, each version of Tickstory is valid for certain MT4 builds. You have to check if you have the latest version of Tickstory, and if it supports build 1090 of MT4. If you do not launch MT4 through the Tickstory launcher, it will always mark the modeling quality as  n/a (and not as 99%), although this has no relevance.


>> I am using TickStory Lite v1.5.3.0 and I guess maybe this is the most up-to-date version of it? Attached is the error message shown when I try to launch MT4 from it. My MT4 platform is Build 1090 and I did try to replace it with an older version, but it would always automatically update - even if I added the "/skipudate" extension at the end of the run command in the shortcut. I would like to run an older version of MT4, but I don't know how to keep it from automatically being updated to 1090. If I could do that, then my current version of TickStory Lite *should* be abe to launch it.


Thank You!

    Shawn


I have not used Tickstory for 3 years (now I use MT5). It used to be free. I see that now the free version only supports until build 765 of MT4. To launch build 1090 through Tickstory, it is assumed that you would need the paid version (v 1.8.9.1). I advise you to do what I have told you with the .fxt files and then you open MT4 directly (without Tickstory). When I used MT4, the results of the backtest were identical if you used the Tickstory launcher as if you were not using it, the only difference is that it marks the modeling quality as  n/a (instead of 99%), but that does not matter, because the results were the same.   Regards.

 
Jose Francisco Casado Fernandez:


I have not used Tickstory for 3 years (now I use MT5). It used to be free. I see that now the free version only supports until build 765 of MT4. To launch build 1090 through Tickstory, it is assumed that you would need the paid version (v 1.8.9.1). I advise you to do what I have told you with the .fxt files and then you open MT4 directly (without Tickstory). When I used MT4, the results of the backtest were identical if you used the Tickstory launcher as if you were not using it, the only difference is that it marks the modeling quality as  n/a (instead of 99%), but that does not matter, because the results were the same.   Regards.

Thanks Jose, but without the MT4 launch from TickStory Lite, I think MT4 is simply not "finding" the folder where my new FXT files are, and using the existing M1 data in the MT4 History Center. As I mentioned in my first post:


"Clicked on "Symbol Properties" before running it and noticed it wasn't reflecting the spread figure I had specifically put in using the FXT Editor of TickStory Lite. Went ahead and tried the backtest anyway, finished in about 2 seconds so I knew something was quite wrong. Viewed the report and showed a modelling quality of "n/a". Tried again in "Visual Mode" and it was evaluating a whole bar at a time, and not tick by tick. I guess it was somehow using the built-in data already that was there within that XM platform and just ignoring all my new FXT data."



Thanks

   Shawn

 
shawnM:

Thanks Jose, but without the MT4 launch from TickStory Lite, I think MT4 is simply not "finding" the folder where my new FXT files are, and using the existing M1 data in the MT4 History Center. As I mentioned in my first post:



No, you have to put your hst files into 'history\(broker name)'  folder  and your fxt files into 'tester\history'  folder.  All of them with the "Read Only" property selected.  It's all.  You can open MT4 (without Tickstory launcher) and backtest what you want.  Regards.

 
shawnM:

You have to generate the fxt files again and mark them with the "Read only" Windows property (right click \ Properties \ Read only) so that MT4 does not overwrite them when starting each new backtest

>> Thank you Jose, good tip!

>> I am using TickStory Lite v1.5.3.0 and I guess maybe this is the most up-to-date version of it? Attached is the error message shown when I try to launch MT4 from it. My MT4 platform is Build 1090 and I did try to replace it with an older version, but it would always automatically update - even if I added the "/skipudate" extension at the end of the run command in the shortcut. I would like to run an older version of MT4, but I don't know how to keep it from automatically being updated to 1090. If I could do that, then my current version of TickStory Lite *should* be abe to launch it.

  1. Actually you have to generate both FXT and all HST files and set them both to read-only (manually, as the application has a bug in it when setting R/O), especially if you are using Multi-Time-frame or Multi-currency (HST files for other symbols).
  2. You cannot use "TickStory Lite v1.5.3.0" to launch build 1090, as that is a very old free version. For current build you have to use the later versions (currently v1.8.9.1). However, you don't have to launch MetaTrader from TickStory. You can just call MetaTrader directly. The only difference will be that report files will show "n/a" in the test quality instead of "99%". Also, v1.5.3 still has many other bugs that need to be considered.
  3. You should really be asking these questions in the TickStory forum and not here.
 
Jose Francisco Casado Fernandez:


No, you have to put your hst files into 'history\(broker name)'  folder  and your fxt files into 'tester\history'  folder.  All of them with the "Read Only" property selected.  It's all.  You can open MT4 (without Tickstory launcher) and backtest what you want.  Regards.


Ahh yes, now it is starting to work! The "read-only" was the trick! In the backtest report, it is showing a full completely green progress bar for "Modelling quality" - even though it does say "n/a". Thank you!

So Jose, regarding the HST files - I do not see where TickStory is generating them - I get only FXT. Also please, should I first delete all the HST files that are pre-existing in the 'history\(broker name)'  folder ?


And finally one other strange thing I notice (please see attachment circled in blue). It appears that some trades profit/loss are not being evaluated correctly. Here circled in blue I have a trade that reaches my full profit target of 15 pips, yet the backtest report shows it was a loss of minus $14.00. Here is what I am thinking is the problem - when I was in TickStory Lite creating the FXT files, I did some modifications of the FXT Header regarding commissions, spread etc. (please see attached). I wasn't quite sure what those fields meant circled in blue but I changed some values there. I think maybe that is the culprit. I am going to find what the default values are again and put it back to those values and try again. Do you know if there is a webpage somewhere that explains what these various FXT Header parameters mean? I could not find anything to explain it oon the Tickstory website.


Thanks!

   Shawn

Files:
backtest.jpg  249 kb
FXTheader.jpg  90 kb
 
Fernando Carreiro:
  1. Actually you have to generate both FXT and all HST files and set them both to read-only (manually, as the application has a bug in it when setting R/O), especially if you are using Multi-Time-frame or Multi-currency (HST files for other symbols).
  2. You cannot use "TickStory Lite v1.5.3.0" to launch build 1090, as that is a very old free version. For current build you have to use the later versions (currently v1.8.9.1). However, you don't have to launch MetaTrader from TickStory. You can just call MetaTrader directly. The only difference will be that report files will show "n/a" in the test quality instead of "99%". Also, v1.5.3 still has many other bugs that need to be considered.
  3. You should really be asking these questions in the TickStory forum and not here.


Thank you for your input Fernando - regarding point (1), I am still not sure how to get Tickstory to generate HST files - all I get is FXT files.


(2) - yes I will launch MT4 manually... no problem if modelling quality says "n/a" !


(3) - Well, I think the support at Tickstory would probably not have revealed to me the fact that it's not necessary to have Tickstory do the launching of MT4!


Thanks!

   Shawn

 
shawnM:

(1) I am still not sure how to get Tickstory to generate HST files - all I get is FXT files.

(3) - Well, I think the support at Tickstory would probably not have revealed to me the fact that it's not necessary to have Tickstory do the launching of MT4!

(3) No support forum is not just answered by the developers but also other users, many of which would even have recommended that you call it directly.

(1) See below, but this is on the latest build. I do not remeber where it is on such an old version, but I do now that it is there somewhere because I once used that version before buying an official license.

 
 
shawnM: And finally one other strange thing I notice (please see attachment circled in blue). It appears that some trades profit/loss are not being evaluated correctly. Here circled in blue I have a trade that reaches my full profit target of 15 pips, yet the backtest report shows it was a loss of minus $14.00. Here is what I am thinking is the problem - when I was in TickStory Lite creating the FXT files, I did some modifications of the FXT Header regarding commissions, spread etc. (please see attached). I wasn't quite sure what those fields meant circled in blue but I changed some values there. I think maybe that is the culprit. I am going to find what the default values are again and put it back to those values and try again. Do you know if there is a webpage somewhere that explains what these various FXT Header parameters mean? I could not find anything to explain it oon the Tickstory website.

You can't just use TickStory as is. You have to first build the MetaTrader Broker Info Config files for each group of Based Currency Symbols, which includes things like Broker Server, Broker Name, Stops level, Swap values, Margin Currency, etc.

This is done by the Configuration MQL4 Script supplied by TickStory, but you will still need to set some values manually, such as the Margin Currency, etc.


However, as I stated before, this should be discussed at the TickStory forum. This is not exactly the place to discuss a 3rd party tool that is not exactly sanctioned by MetaQuotes.

Reason: