Download MetaTrader 5

Large M1 data disappears from MT4 between re-launches, but .hst files all present

To add comments, please log in or register
Join MQL5 Cloud Network. Improve your rating!
jmb1
86
jmb1 2013.02.09 22:27 

I wonder if anyone else has come across this.   Is it a bug?

I'm able to import _SP500.csv which contains 30 years' worth of minute data successfully through History Center (for the method followed see https://forum.mql4.com/51224), and create all my .hst files for the market.  I'm then able to run back tests using M1 and all other time frames created.

However, when I close and re-launch MT4, the M1 time frame data is gone, and has to be re-imported from _SP500.csv.  BUT the _SP5001.hst file (the M1 history data file) is still present.

Through trial and error, I found that this does not happen if the M1 data is no more than around 750,000 lines long, which History Center says is 297899 records.

The work around is that I have to re-import each time I re-launch MT4, which is quick work.  But its very annoying!   :) 

Simon Gniadkowski
Moderator
18001
Simon Gniadkowski 2013.02.09 22:51  
jmb:

I wonder if anyone else has come across this.   Is it a bug?

I'm able to import _SP500.csv which contains 30 years' worth of minute data successfully through History Center (for the method followed see https://forum.mql4.com/51224), and create all my .hst files for the market.  I'm then able to run back tests using M1 and all other time frames created.

However, when I close and re-launch MT4, the M1 time frame data is gone, and has to be re-imported from _SP500.csv.  BUT the _SP5001.hst file (the M1 history data file) is still present.

Through trial and error, I found that this does not happen if the M1 data is no more than around 750,000 lines long, which History Center says is 297899 records.

The work around is that I have to re-import each time I re-launch MT4, which is quick work.  But its very annoying!   :) 

What do you have Tools>Options>Charts  Max Bars in History set to ?
jmb1
86
jmb1 2013.02.09 22:58  
RaptorUK:
What do you have Tools>Options>Charts  Max Bars in History set to ?


I followed the method in https://forum.mql4.com/51224, so 99999999999999999999 (2147483647)!  Should be plenty!   :)
Simon Gniadkowski
Moderator
18001
Simon Gniadkowski 2013.02.10 09:54  
jmb:

I followed the method in https://forum.mql4.com/51224, so 99999999999999999999 (2147483647)!  Should be plenty!   :)
Perhaps it's a UAC issue ?  where do you have MT4 installed and what OS are you using ?
jmb1
86
jmb1 2013.02.10 11:08  

I've tried various installs:

  • Windows XP Pro - C:\tester1
  • Windows XP Pro - C:\program files\tester1
  • Wine on Linux (CentOS 6)

Simon Gniadkowski
Moderator
18001
Simon Gniadkowski 2013.02.10 11:22  
jmb:

I've tried various installs:

  • Windows XP Pro - C:\tester1
  • Windows XP Pro - C:\program files\tester1
  • Wine on Linux (CentOS 6)

UAC is not applicable in XP.

 

Have you verified that you have ALL the data imported ?  how many M1 bars should you have ?  how many M1 bars do you have ?

jmb1
86
jmb1 2013.02.10 17:30  

I've been checking through the data, and have found something strange.  I wonder if what I've found is at the root of this problem.

With the help of BREW_MissingData_sep.mq4 found on this site, I narrowed down where there seemed to be gaps in the data, the most recent being October 2012.  I then examined the file on a Linux box with Joe text editor - its too big for anything I have under Windows.  The gap in the raw data isn't there.  So for some reason MT4 didn't import this part.

So I isolated the missing bit, and saved it to "oct15_31-2012.txt".  I also reverse sorted it using Sed and created "oct15_31-2012-r.txt".  I've attached the first hereto, and the second in the next comment.

I'm completely unable to see why this data won't import into MT4.  It all looks fine to me.  But I must be missing something, so I need all the help I can get!  Sorry!   :(

Files:
jmb1
86
jmb1 2013.02.10 17:31  
...here's oct15_31-2012-r.txt...
Files:
whroeder1
14545
whroeder1 2013.02.10 21:15  
jmb: ...here's oct15_31-2012-r.txt...
10/15/2012,0930,1428.75,1430.68,1428.75,1430.62,0
10/15/2012,0931,1430.81,1431.54,1430.81,1431.3,0
10/15/2012,0932,1431.49,1432.16,1431.49,1432.16,0
10/15/2012,0933,1432.68,1432.73,1432.37,1432.59,0
The data I got through Problem importing data for backtesting - MQL4 forum looks like
<TICKER>,<DTYYYYMMDD>,<TIME>,<OPEN>,<HIGH>,<LOW>,<CLOSE>,<VOL>
AUDJPY,20010102,230100,64.30,64.30,64.30,64.30,4
AUDJPY,20010102,230300,64.29,64.29,64.29,64.29,4
AUDJPY,20010102,230400,64.29,64.29,64.29,64.29,4

No volume.

You didn't check the volume button did you?

Simon Gniadkowski
Moderator
18001
Simon Gniadkowski 2013.02.10 21:17  
WHRoeder:
The data I got through Problem importing data for backtesting - MQL4 forum looks like
<TICKER>,<DTYYYYMMDD>,<TIME>,<OPEN>,<HIGH>,<LOW>,<CLOSE>,<VOL>
AUDJPY,20010102,230100,64.30,64.30,64.30,64.30,4
AUDJPY,20010102,230300,64.29,64.29,64.29,64.29,4
AUDJPY,20010102,230400,64.29,64.29,64.29,64.29,4

No volume.

You didn't check the volume button did you?

Was just about to type the same thing ;-)

Volume doesn't usually matter much,  it doesn't actually mean Volume,  it means tick count.  It is needed for the Strategy Tester though,  the Strategy Tester uses the "Volume" to determine how many ticks to synthesise when it creates the .fxt tick data file.. 

jmb1
86
jmb1 2013.02.11 17:52  

I'm extremely happy (not least relieved!) to be able to report that I've resolved this problem.

The raw minute data I have is formatted thus:

"Date","Time","Open","High","Low","Close","Volume"

06/28/2004,0931,37.49,37.50,37.45,37.46,1049200 

Over almost all of the 30 years' worth, MT4 is happy to import this straight and without editing into its history for back testing.  Almost!  There must be something about certain dates it just can't cope with.

The answer is to reformat the whole 30 years' worth of data, or 2,824,276 records(!) thus:

"Date","Time","Open","High","Low","Close","Volume"

2004/06/28,0931,37.49,37.50,37.45,37.46,1049200 

So, Sed to the rescue:

$ sed -e 's_\(..\)\/\(..\)\/\(....\)_\3/\1/\2_' in_file.txt > out_file.txt 

Goodness!  What a steeeep learning curve I'm on..!  Hope this info proves useful to someone else too!

Still, keeps me off the streets... 

12
To add comments, please log in or register