Backtesting Issue...

EdFuk
137

Hi,


I'm encountering a data loading issue when trying to backtest on UK100 symbol. The errors log from the journal is shown below. Looks like the error is "Cannot add tick event (events' array size is 35782656)" I also encounter the same problem when trying another broker account. Any ideas what is going on? I switched to using a broker example EA for simplicity to rule out a problem with my EA. Same thing happened. Once started there is a lot of processer crunching then the following error sequence. It appears there is a problem with retrieving the data? Seems strange.


Backtest Settings


Backtest Journal

algui91
60
algui91  
To me it seems you ran out of memory.
EdFuk
137
EdFuk  
algui91 #:
To me it seems you ran out of memory.

Which is bizarre because I'm only running 2 and half years worth of 1 minute OHLC. Works fine with FX

Alexandru Casian
435
EdFuk #:

Which is bizarre because I'm only running 2 and half years worth of 1 minute OHLC. Works fine with FX

Did you backtest other symbols before that? I often run into a similar problem after testing different symbols. MT keeps the old data and keeps storing it till you run out of memory. Deleting it manually could solve the problem.
EdFuk
137
EdFuk  
Alexandru Casian #:
Did you backtest other symbols before that? I often run into a similar problem after testing different symbols. MT keeps the old data and keeps storing it till you run out of memory. Deleting it manually could solve the problem.

Hello. Well I found the problem. I’m using 1 minute OHLC and have found that the data only goes back to 28/03/2017. Before this date the data interval is all over the place and eventually ends up daily. Obviously I’m looking for as much good data as possible and my plan now is to create a custom symbol with better quality data. I wondered if you could suggest where the best place to find good quality data for FX/Stocks/CFDs/Indices etc is? I've heard of Tick Data Suite and Tickstory. Can anyone provide me with some advice as to which one is best for use with MT5.


Cheers,

 Ed

Alexandru Casian
435
EdFuk #:

Hello. Well I found the problem. I’m using 1 minute OHLC and have found that the data only goes back to 28/03/2017. Before this date the data interval is all over the place and eventually ends up daily. Obviously I’m looking for as much good data as possible and my plan now is to create a custom symbol with better quality data. I wondered if you could suggest where the best place to find good quality data for FX/Stocks/CFDs/Indices etc is? I've heard of Tick Data Suite and Tickstory. Can anyone provide me with some advice as to which one is best for use with MT5.


Cheers,

 Ed

I can't tell too much about Tickstory besides that I tried it once, it looked well, but backtestings were showing N/A quality. I didn't bothered searching for a solution as I liked Tick Data Suite 2 more.

Tick Data Suite 2 on the other hand is more "plug and play". It's easy to use and has more data sources. It also has an awesome compression algorithm which stores ~19 years of tick data (with variable spread) for 1 pair into only ~1GB. It's also easy to export that data to then use it on MT5.
One downside is the cost. Tick Data Suite 2 (TDS2) costs ~100$. It's one time pay, but for support and further updates you'll need an active subscription which costs ~10$/month, but gladly you'll most probably not need it. And if you need an update or help, you pay 10$, update/ask, and then unsubscribe. But if you have a $1k+ deposit, it's worth investing those 100$ (in my opinion) so that you'd decrease the chance of losing those $1k.
Another downside is that there isn't a spread multiplier when exporting ticks. At least I couldn't find one. It's a problem only if you backtest scalpers, but it's a big problem. I get more realistic results from the "bad" generated ICMarkets ticks from MetaQuotes, than from Dukascopy ticks exported from TDS2. But for non-scalpers TDS2 is awesome.

EdFuk
137
EdFuk  
Alexandru Casian #:

I can't tell too much about Tickstory besides that I tried it once, it looked well, but backtestings were showing N/A quality. I didn't bothered searching for a solution as I liked Tick Data Suite 2 more.

Tick Data Suite 2 on the other hand is more "plug and play". It's easy to use and has more data sources. It also has an awesome compression algorithm which stores ~19 years of tick data (with variable spread) for 1 pair into only ~1GB. It's also easy to export that data to then use it on MT5.
One downside is the cost. Tick Data Suite 2 (TDS2) costs ~100$. It's one time pay, but for support and further updates you'll need an active subscription which costs ~10$/month, but gladly you'll most probably not need it. And if you need an update or help, you pay 10$, update/ask, and then unsubscribe. But if you have a $1k+ deposit, it's worth investing those 100$ (in my opinion) so that you'd decrease the chance of losing those $1k.
Another downside is that there isn't a spread multiplier when exporting ticks. At least I couldn't find one. It's a problem only if you backtest scalpers, but it's a big problem. I get more realistic results from the "bad" generated ICMarkets ticks from MetaQuotes, than from Dukascopy ticks exported from TDS2. But for non-scalpers TDS2 is awesome.

Thankyou, I will take a look at both.


Cheers,

E

Fernando Carreiro
9821
Alexandru Casian #:

I can't tell too much about Tickstory besides that I tried it once, it looked well, but backtestings were showing N/A quality. I didn't bothered searching for a solution as I liked Tick Data Suite 2 more.

Tick Data Suite 2 on the other hand is more "plug and play". It's easy to use and has more data sources. It also has an awesome compression algorithm which stores ~19 years of tick data (with variable spread) for 1 pair into only ~1GB. It's also easy to export that data to then use it on MT5.
One downside is the cost. Tick Data Suite 2 (TDS2) costs ~100$. It's one time pay, but for support and further updates you'll need an active subscription which costs ~10$/month, but gladly you'll most probably not need it. And if you need an update or help, you pay 10$, update/ask, and then unsubscribe. But if you have a $1k+ deposit, it's worth investing those 100$ (in my opinion) so that you'd decrease the chance of losing those $1k.
Another downside is that there isn't a spread multiplier when exporting ticks. At least I couldn't find one. It's a problem only if you backtest scalpers, but it's a big problem. I get more realistic results from the "bad" generated ICMarkets ticks from MetaQuotes, than from Dukascopy ticks exported from TDS2. But for non-scalpers TDS2 is awesome.

Your comments seem to be related to MT4. The OP is using MT5. He does not need to acquire 3rd party tools or outside tick data for his back-tests.
Fernando Carreiro
9821
EdFuk #: Hello. Well I found the problem. I’m using 1 minute OHLC and have found that the data only goes back to 28/03/2017. Before this date the data interval is all over the place and eventually ends up daily. Obviously I’m looking for as much good data as possible and my plan now is to create a custom symbol with better quality data. I wondered if you could suggest where the best place to find good quality data for FX/Stocks/CFDs/Indices etc is? I've heard of Tick Data Suite and Tickstory. Can anyone provide me with some advice as to which one is best for use with MT5.

You are using MT5 which has real tick data from your Broker. TickStory and Tick Data Suite are mostly for MT4. That is not your case.

Select "Every Tick based on Real Tick Data" for modelling field.

Alexandru Casian
435
Fernando Carreiro #:
Your comments seem to be related to MT4. The OP is using MT5. He does not need to acquire 3rd party tools or outside tick data for his back-tests.

Well, it depends. MT5 ticks are from MetaQuotes, not from brokers. From what I know, those ticks aren't even real, they are generated, and some people say they are bad for backtesting. I can't agree with that, I was getting more similar to real results backtests with MetaQuotes ticks than with 3rd party. The period was smaller though, 7 years, but it was ok. And so it was till 1~2 months ago when that 7 years period dropped to about 6~12 months. Now if I backtest on MT5 7 years, I get 0 modelling quality. Even for 1 year I get less than 50%. Backtesting while connected to RoboForex shows much better results, but then I don't like their spread at night.
I contacted IC Markets support, that's when I find you that the MT5 ticks aren't from their servers, but from MetaQuotes. Well, after all that story, I think 3rd party are a more reliable option than MetaQuotes ticks for both MT5 and MT4.

Anyways, EdFuk asked about 3rd party for MT5, I answered.

Fernando Carreiro
9821
Alexandru Casian #: Well, it depends. MT5 ticks are from MetaQuotes, not from brokers. From what I know, those ticks aren't even real, they are generated, and some people say they are bad for backtesting. I can't agree with that, I was getting more similar to real results backtests with MetaQuotes ticks than with 3rd party. The period was smaller though, 7 years, but it was ok. And so it was till 1~2 months ago when that 7 years period dropped to about 6~12 months. Now if I backtest on MT5 7 years, I get 0 modelling quality. Even for 1 year I get less than 50%. Backtesting while connected to RoboForex shows much better results, but then I don't like their spread at night. I contacted IC Markets support, that's when I find you that the MT5 ticks aren't from their servers, but from MetaQuotes. Well, after all that story, I think 3rd party are a more reliable option than MetaQuotes ticks for both MT5 and MT4. Anyways, EdFuk asked about 3rd party for MT5, I answered.

You seem to be misinformed! When MetaTrader 5 is connected to your broker's trading account, you get broker historical data and tick data, not data from MetaQuotes. I have been using MT4 and MT5 for a long time on several brokers and can guarantee that the tick data is different between Metaquotes and brokers and also between different account types, such as between Demo and between Live, and between Cent, Standard, Pro and ECN types of accounts. By the way, night spread depends on the type of account you are using. ECN accounts have much better spread than Standard or Pro accounts.

Obviously, depending on the broker, some keep plenty of history data and other keep less. If a broker does not keep record of tick data far back in time, then simply connect to another broker or account type. So far, I have only needed tick data from Dukascopy and with TickStory when testing on MT4. On MT5 I have never needed to do that.

Also, there is also a difference between "Every tick" and "Every tick based on real ticks". The former is virtual ticks and the later are real ticks.