Spooky Data - large chunks missing though Chart displays fine???

New comment
  
int start()
{
 
     Print(Time[0]);
      
return(0);
}

Hi, I was examining my back-test trades and noticed some spots that should have traded but didn't. So I started the investigation...

AUDUSD, June 4the 2013, 1 minute chart back-testing on every tick setting.

Here's what I found.. After verifying all parameters were satisfied I reduced the code to just a simple print output on the suspect date. (code is above)

The data still wouldn't print on certain times, it wouldn't print on almost the entire day of June 4th. I've pasted the pics below of the beginning and end of the Journal printout so you can view it yourself.

It only prints a few minutes of data for June 4th. I've also tried only printing the data once a minute to reduce the amount of print outs, however when I do that it only prints a small portion of June 4th, like up to 10 am on June 4th. I've also included a pic of the testing settings. It will print June 3rd and june 5th.

Investigations results: Spooky data that EA can't find.

My biggest concern is that the entire data set could be acting this way leaving blank spots that are just missing, even though the chart itself displays completely!

How could any results be trusted?

Any thoughts or ideas on further tests I could run would be appreciated. Are there small limits as to the amount of printable data that the journal will accept?

Note* I did check the history center but it only showed me the minute data, not tick data.

Thanks in advance.

settings

 

Have you looked in the history center to see what 1 minute data it has for June 4th ?

 

moneycode:

leaving blank spots that are just missing, even though the chart itself displays completely!
Note* I did check the history center but it only showed me the minute data, not tick data.

  1. Mt4 doesn't display missing bars, so of course it displays completely.
  2. MT4 doesn't HAVE tick data.
  3. Have you downloaded history?
 
moneycode:

Hi, I was examining my back-test trades and noticed some spots that should have traded but didn't. So I started the investigation...

AUDUSD, June 4the 2013, 1 minute chart back-testing on every tick setting.

Here's what I found.. After verifying all parameters were satisfied I reduced the code to just a simple print output on the suspect date. (code is above)

The data still wouldn't print on certain times, it wouldn't print on almost the entire day of June 4th. I've pasted the pics below of the beginning and end of the Journal printout so you can view it yourself.

Did you look in the actual log file ? not just the Journal/Experts tab . . . the log file will have all the prints whereas the Journal/Experts tab sometimes cannot update fast enough . . .
 
I
WHRoeder:
  1. Mt4 doesn't display missing bars, so of course it displays completely.
  2. MT4 doesn't HAVE tick data.
  3. Have you downloaded history?

1. Good point, however the time at bottom of chart didn't appear to be missing any time chunks, which would tell me all the bars were there, but the "inside the bar data" is what I thought would be missing.

I did look in the History section and it showed me 1 minute bars seemed to all be there.

2. Doesn't have tick data? But I thought the setting on the last image I posted has the setting for "tick" data, which is how I have it set. So I must not understand the details of the data correctly.

3. Yet another good point and I think also was the solution. Because just to double check if there was more data, I tried to download it again today, and at first is said no but then a second box came up and it seemed to give me some more data. But it only ran for a few seconds. Not like the original time I downloaded it, which took a few minutes.

I did have to go into the Log file in order to see it, as Raptor suggested.

So the case of Spooky Data has been solved!

Thanks

P.S. I spent most of the day reading up on mql4 and watching videos for mql4 to try and accelerate my learning curve and be less of a PIA.

15 years with easy language hasn't really helped matters..

 
moneycode:
I

1. Good point, however the time at bottom of chart didn't appear to be missing any time chunks, which would tell me all the bars were there, but the "inside the bar data" is what I thought would be missing.

I did look in the History section and it showed me 1 minute bars seemed to all be there.

2. Doesn't have tick data? But I thought the setting on the last image I posted has the setting for "tick" data, which is how I have it set. So I must not understand the details of the data correctly.

The ticks are synthesised by the Strategy Tester from the M1, and above, history data, MT4 does not save any tick data.
 
RaptorUK:
The ticks are synthesised by the Strategy Tester from the M1, and above, history data, MT4 does not save any tick data.



Oh thanks. . Are there any obvious draw backs with that type of data from your experience, that I should be aware of in my back testing.? (My present strategies typically make only a few trades per day. )

Also, is there a short cut key stroke for "compile"?

 
moneycode:


Oh thanks. . Are there any obvious draw backs with that type of data from your experience, that I should be aware of in my back testing.? (My present strategies typically make only a few trades per day. )

Also, is there a short cut key stroke for "compile"?

1. Do not use tick sensitive strategies.

2. F5 and F7

 
moneycode:
Are there any obvious draw backs with that type of data from your experience, that I should be aware of in my back testing.?

Since your "tick" data is simulated from the Open/High/Low/Close values of M1..M?? candles it means that you will not be getting the historically correct Bid, Ask and Spread for the simulated ticks. What that means is that you will not see in the tester events like dealer/broker/MM widening the spread during news announcements. And these are often source of bad entries or bad exists for scalping EAs.

How this works (AFAIK anyway):

Instead of the correct Bid and Ask (and Spread since it is the diff between these two) the historical data stored contains only 1 minute candles (OHLC) with Bid prices. The tester will simulate random ticks which will fall somewhere within this OHLC price range. So you are guaranteed that within that candle you will have at least one occurrence of High and Low respectively and that it will start with the Open and end with the Close value. Anything else in between is random (but of course within the Hi-Lo boundaries).

When trying a strategy in the tester you have basically 2 options to simulate the Spread (or the Bid Ask difference): 1. use current live spread of the pair which provides good random spread (but of course will generate results which are not 100% reproducible); 2. Use a fixed spread - controlled by the Spread combo-box on the Settings tab of the tester. This will generate repeatable results but of course it lacks the randomness of the real tick data.

- - -

One thing I am not sure about:

When the candles were recorded on the server (for later download and use as historical data) - was the actual LIVE SPREAD at that time taken into account? Or was just the OHLC of the Bid price recorded? In other words - assuming we are talking about M1 data - if the spread during that minute was widened UP i.e. the Ask price rose significantly and the Bid price did not move as much - will the OHLC of the recorded candle reflect this? Will the High of that candle be the highest Ask during that minute? Or will it just store the Bid movement as OHLC? The latter of course would be the worse case scenario as there would be absolutely NO INDICATION of price surges in the tester...

Can someone knowledgeable about the tester and the way historical data is generated please answer this?

 
4x4ever:

Since your "tick" data is simulated from the Open/High/Low/Close values of M1..M?? candles it means that you will not be getting the historically correct Bid, Ask and Spread for the simulated ticks. What that means is that you will not see in the tester events like dealer/broker/MM widening the spread during news announcements. And these are often source of bad entries or bad exists for scalping EAs.

How this works (AFAIK anyway):

Instead of the correct Bid and Ask (and Spread since it is the diff between these two) the historical data stored contains only 1 minute candles (OHLC) with Bid prices. The tester will simulate random ticks which will fall somewhere within this OHLC price range. So you are guaranteed that within that candle you will have at least one occurrence of High and Low respectively and that it will start with the Open and end with the Close value. Anything else in between is random (but of course within the Hi-Lo boundaries).

When trying a strategy in the tester you have basically 2 options to simulate the Spread (or the Bid Ask difference): 1. use current live spread of the pair which provides good random spread (but of course will generate results which are not 100% reproducible); 2. Use a fixed spread - controlled by the Spread combo-box on the Settings tab of the tester. This will generate repeatable results but of course it lacks the randomness of the real tick data.

If you use Current or a fixed spread the Spread is fixed for the duration of the Strategy Test run . . . Current just means the current spread at the start of the test, so from one test to the next it can be different.


4x4ever:


When the candles were recorded on the server (for later download and use as historical data) - was the actual LIVE SPREAD at that time taken into account? Or was just the OHLC of the Bid price recorded? In other words - assuming we are talking about M1 data - if the spread during that minute was widened UP i.e. the Ask price rose significantly and the Bid price did not move as much - will the OHLC of the recorded candle reflect this? Will the High of that candle be the highest Ask during that minute? Or will it just store the Bid movement as OHLC? The latter of course would be the worse case scenario as there would be absolutely NO INDICATION of price surges in the tester...

Can someone knowledgeable about the tester and the way historical data is generated please answer this?

The chart, and the bars/candles are drawn by Bid price so the OHLC data is all based on Bid prices.
New comment
Reason: