Yet another unmatched data errors post -- but with a TWIST!

New comment
 

Before I go any further, let me say that I have searched carefully using the forum search feature and using Google, and while I am aware that there are many threads over the years about unmatched data errors, NONE that I found addressed the issue I am facing.

Following advice in many of those posts, I have set up data for back testing a strategy designed to work on daily candles. I may expand this strategy later to work on smaller time frames too but at the moment my focus is on daily candles. The strategy combines some simple price-action related rules with some simple tick volume-related rules to determine entries, and uses a simple but very rigid risk management strategy.

As to the providence of my data. I run a process I wrote that downloads tick data continuously while the market is open using FXCM's Java API. I store the tick data to a database and use it to calculate one-minute bid and ask price data each week together with tick volume data. In this way I've been building up a database of one minute price data and tick volume since September 2012.

I took the bid data from this database for USDJPY for the entire period, including tick volume, and washed it through some home-grown programs I wrote that analyse the data for errors eg duplicate bars, bars where the high isn't the highest price / low isn't lowest price, etc. It also watches out for gaps in the data that don't occur at the weekend, and strips out pre-market data on Monday mornings (I am in Japan, market opens at 6am Monday in summer, 7am Monday in winter local time). I converted the time stamps on the data to Europe/Nicosia time zone, which is the time zone used by my broker, as it results in the market open being at midnight and thereby avoids Sunday bars.

I then followed the following steps:

  1. Open MT4, choose File --> Open Data Folder to get a Windows Explorer open on the data folder for this MT4 instance
  2. Make sure no charts of USDJPY open.
  3. Close MT4. Check in Task Manager to make sure the terminal.exe process is gone.
  4. In Windows Explorer, navigate to the history folder for the demo account I am using.
  5. Delete all data relating to USDJPY (USDJPY1.hst etc)
  6. Disable network adaptor.
  7. Start MT4 again, verify it has been unable to connect to the broker (because the network is down)
  8. Use the History Center to upload my 1-min data with volume. Make sure volume is checked on.
  9. <Maybe unnecessary, I tried it with and without this step to be sure, made no difference> Shutdown MT4, wait for process to die, restart MT4
  10. Pull up USDJPY chart. It comes up 1H by default, and shows *Waiting for Data*
  11. Change to 1M. It shows the data I loaded. Check data is there all the way back to September 2012
  12. Run the Period Converter script in turn for 5M, 15M, 30M, 60M, 240M and 1440M (Daily)
  13. Shutdown MT4. Wait for process to die. Check all files in history folder and that their sizes look sensible relative to each other.
  14. Re-enable network.
  15. Start MT4. In Strategy Tester, run a backtest of my strategy from 2012 September to July 8th 2016. Using every tick, and a spread of 30 pipettes since the strat will trade at daily open if it trades at all, so spread can be quite bad. *THIS BACKTEST WORKS FINE with good quality -- modelling quality about 80-85% and no unmatched data errors. I wonder why this isn't the usual 90% though?*
  16. Run another backtest, either the same one or a subset of the total time eg September 2012 to December 24th 2013. Now I start getting lots of unmatched data errors. Once this starts happening it keeps happening forever until I wipe the data and reload.

If I omit step 14, ie don't turn the network back on, the unmatched data errors do not happen, even with repeated backtests, but then I get a warning in the log saying No connect to trade server, default environment will be applied, and I have no idea what "default environment" means and what bearing it has on my results.

I assume the problem is something to do with data MT4 is getting from the broker (not FXCM) where the volume data and/or price data on a daily time frame doesn't exactly agree with the data I've uploaded. What I don't understand is why it is looking, since I supplied all required data and am not trying to test outside the date range I loaded. Is there a way to prevent this clash, and if the only way to do so is to keep the network down, what implications are there of using the "default environment" ie what in precise terms does "default environment" mean?

Thanks for your help if you are still reading :)

 
  1. You must connect once (with a chart) to get spread, swap, stoplevel, margin required, etc. If you don't reconnect you use the last known values the "default environment"
  2. Once you connect (step 14), the terminal tries to reloads the last 2048 bars on your chart's TF. Thus that TF is now out of step with all your previous work. This has nothing to do with your testing. And how far back 2048 bars are, depends on your open charts.
 
WHRoeder:
  1. You must connect once (with a chart) to get spread, swap, stoplevel, margin required, etc. If you don't reconnect you use the last known values the "default environment"
  2. Once you connect (step 14), the terminal tries to reloads the last 2048 bars on your chart's TF. Thus that TF is now out of step with all your previous work. This has nothing to do with your testing. And how far back 2048 bars are, depends on your open charts.


Good to know "default" means "last used". That is a relief. I was afraid "default" would equate to "total bollocks".

On point 2 -- just let me make sure I have understood correctly, if you don't mind. Are we saying that on loading a chart (either a real one or a virtual one for the purposes of the Strategy Tester) the last 2048 candles' worth of data will be downloaded even if it has already been provided in the HST files? For a Daily chart that's 2048 days' worth of data that I uploaded in 1-min form and then translated to daily using the Period Converter, which is just getting trampled...

Aaactually I've just realised what might be going on here. Since I only started recording tick data and using it to construct my one minute bars in 2012, I only have volume-included history back to then (I have price history from FXCM back to 2002, but don't have volume data to go with it. That hasn't mattered until the EA I am testing now). On the daily chart that is 1000-odd bars. Less than 2048. Perhaps that is prompting MT4 to download the whole 2048 bars instead of only the 10 or so bars from where my uploaded data ends to the current day.

OK I'm starting to talk myself into believing that my situation is a little bit of a corner case. I guess I'll just have to backtest with the network disconnected for this EA for now (or test on smaller time frames!) or see if I can get my hands on more price history with volume... It's a major pain to switch data providers though...

Anyway thanks.

 
Once you connect the 2048 candles worth are download to your open charts.
New comment