Creating a public database of all ticks, for better backtesting
general idea is good, but you should know a few things first.
I think collecting and sharing tick data is not legal, even publishing these data is forbidden by some brokers if my mind doesn't trick me.
All platforms will generate sometimes different tick data, so on which one rely?
You need a good piece of software merging these together.
Tick data is going to be big. So you might need large disk space and if this data is going to be published and good also a large bandwidth.
just 2 cents
//z
Last year a company programmed an EA for me and addittionally to my EA I got the minute-datas for the last two years.
I gave another EA to them for professional backtesting and optimization which runs on the M1 Chart, and they did the test with tick data, now it works great!
Through the minute-data I could improve my slower EA which runs on the 15M timeframe successfully. Now I want to try a tick-data test myself. I already saved the tickdata from some brokers and I believe it should be OK if I use it for my own pruposes.
How can you use the tick-data with MetaTrader 4? I know in former times you could convert the csv into an fxt-File and disable the recalculation at the strategy tester. But this way it isn't possible anymore. How could you do this?
Hello
Thank for you remarks.
1/ About legal aspects I will verify on terms and conditions. But as soon as these data have been published and broadcasted... As a matter of fact, let's consider that only contributors will have access to data. As soon as they are shared in a private group for non commercial purposes, nobody can prevent you from doing it.
2/ All platforms generate different ticks, that's true. Therefore I asked people to tell me which ones they are interested in (Mine is FxPro). What we do for one platform may be done for others, provided they are enough "listeners" to constitute a good quality database. On one dedicated PC used for recording ticks, I think you may work simultaneously on several platforms.
3/ Merging is not hard to do. But it is time-consumering.
4/ There will be a lot of ticks, but data to store for one single tick is poor : time, bis, ask. For one year and one pair I expect 100 millions rows. With disks of 1 To (or more) there is not problem to store it. The easiest way would be a MySQL database. Using these data need good computering ability. But it's the price to pay if you want to perform EA backtesting in true conditions.
I hereby answer question of sunshined : I have not had a look on how to use such data in MT4. I consider that backtesting should be realized outside MT4, espacially with so many incoming data. I built myself a testing software in C++. By the way, my EAs are also writen in C++ as DLLs. This DLL is invoked by MT4, which role is limited to an interface with trading platform. In the future, I might share some pieces of C++ code (as soon as it is bugfree and user-friendly.)
Hello
Thank for you remarks.
1/ About legal aspects I will verify on terms and conditions. But as soon as these data have been published and broadcasted... As a matter of fact, let's consider that only contributors will have access to data. As soon as they are shared in a private group for non commercial purposes, nobody can prevent you from doing it.
2/ All platforms generate different ticks, that's true. Therefore I asked people to tell me which ones they are interested in (Mine is FxPro). What we do for one platform may be done for others, provided they are enough "listeners" to constitute a good quality database. On one dedicated PC used for recording ticks, I think you may work simultaneously on several platforms.
3/ Merging is not hard to do. But it is time-consumering.
4/ There will be a lot of ticks, but data to store for one single tick is poor : time, bis, ask. For one year and one pair I expect 100 millions rows. With disks of 1 To (or more) there is not problem to store it. The easiest way would be a MySQL database. Using these data need good computering ability. But it's the price to pay if you want to perform EA backtesting in true conditions.
I hereby answer question of sunshined : I have not had a look on how to use such data in MT4. I consider that backtesting should be realized outside MT4, espacially with so many incoming data. I built myself a testing software in C++. By the way, my EAs are also writen in C++ as DLLs. This DLL is invoked by MT4, which role is limited to an interface with trading platform. In the future, I might share some pieces of C++ code (as soon as it is bugfree and user-friendly.)
Hello Pebs,
I might be interested in this project. I have created most of my EA's by modifying other's scripts -- or creating my own. I am not a C++ programmer. would you be interestd in allowing me to particiapte?
regards,
spgandau
- Free trading apps
- Over 8,000 signals for copying
- Economic news for exploring financial markets
You agree to website policy and terms of use
For backtesting purposes, I am currently looking for a database of all ticks.
I seems that nothing serious and costless exists.
Then I proposed myself to create those databases myself. By letting my omputer runnng 5/7 24/24, memorising every tick it receives.
But it appears that I can miss a lot of ticks (maintenance, network breakdowns...). Then it may be a good idea that we create a group in order to collect all these ticks.
- I created a program that memorises ticks, on a personnal platform (it runs like an EA). The less the computer is used, the more ticks it will receive. Then people who can have a dedicated (old) computer running 24/24 in a corner of their home are welcome.
- I created another one to reconciliate every file coming from different contributors. This will generate a CSV file with the greatest number of ticks we will be able to collect.
- I will also create a little website (or blog) so that contributors can upload their files, and download resulting tick compilations. Obviously all this will be free (at least for contributors).
I think we must be at least 3 or 4 per pair and provider (On myself I work with FxPro and AAAFx).
Should we collect these datas from free accounts or from real ones ? I hope that ticks are the same on both... but this should be discussed.
Please let me know who is interested in this adventure ? and which provider you use or may use / which pairs or products you are interested in...
My e-mail : pebssoft AT gmail DOT com