Problem loading historical data

 
Hello,

I have some issues loading historical data. I want to test an expert advisor for which I need lots of 1Min Historical data.
1. When I keep pressing the home button on the 1 min chart to load more historical data, there is only a few weeks back I can go to. After that, pressing the home button does not load any more historical data.
2. I loaded up an historical data file in the history center. However the time in the past, from where my brokers data stops and the historical data from the file takes effect, my indicator values get messed up. Attaching a screenshot to which shows the problem with the indicator lines.

I am not sure why this is happening. Can someone please help with the problem or suggest ways to get large amounts of historical data for backtesting.

Regards,
Arooba

 
Tool > Options > Charts tab > change 'Max bars in history' and 'Max bars in chart' to 9999999999 > OK > Restart MT4.
 
gordon:
Tool > Options > Charts tab > change 'Max bars in history' and 'Max bars in chart' to 9999999999 > OK > Restart MT4.


Hi Gordon,
Thanks for the response. However this does not solve my problem. The indicator is still messed up as in the above chart. This only happens with data imported through the history center.
 
Check the data in the history center... Maybe there are big gaps?
 
Arooba wrote >>
1. When I keep pressing the home button on the 1 min chart to load more historical data, there is only a few weeks back I can go to. After that, pressing the home button does not load any more historical data.



Just FYI as someone who use to do what you describe here I know the pain you are experiencing trying to download historical data.

There is a better/easier way to get the data from the server from your broker and that is:

Step 1: Open an M1 chart
Step 2: Go to History Center (press F2)
Step 3: Navigate to the currency pair and the "1 Minute (M1)" history data for the chart you have open in your terminal (be sure to "double click" the "1 Minute (M1)" selection in history center so it actually becomes the active selection)
Step 4: Then left click once on any of the data rows in the right hand window of the History Center (it doesn't matter which one)
Step 5: Click the "Edit" button, this pops up another window titled "Bar" and the default selection is on the "year"
Step 6: Push the down arrow on your keyboard until the year reads something silly like 1970
Step 7: Then select "OK"...this doesn't actually edit the data you selected but rather it creates a new bar with the timestamp and price info you just edited...don't worry about this as you will delete this fake bar in the end
Step 8: Now select "Close" in the History Center
Step 9: Navigate to your M1 chart of interest and then from the terminal menu select "Charts -> Refresh"...this will refresh the chart and attempts to fill any time gaps in the chart data (which now includes your year 1970 data point) with server data
Step 9 continued...Now your chart will have pulled anywhere from 2048 to 65536 M1 bars from your broker's server (not Metaquotes server) and is usually far more data than you can get the broker to cough up when you try and manually scroll your M1 chart back in time
Step 10: Go back to History Center (press F2) and navigate to your currency pair and its M1 data
Step 11: scroll down the data to the bottom of the list, your will see your 1970 candle there, click it once and then click the "Delete" button in the History Center window

Now you have really quickly downloaded all the broker's M1 data on your currency pair that the broker is going to let you have, much faster than holding down the home or pg-up key for minutes and minutes to download even less data. It works with charts of other timeframes as well.

 
1005phillip:

There is a better/easier way to get the data from the server from your broker and that is (...)

Excellent! Good find phillip. And it actually makes sense.

 
1005phillip:


There is a better/easier way to get the data from the server from your broker and that is:

Downloading history is NOT getting it from your broker, it is getting it from metatrader as the pop up message tells you.
Only scrolling back gets it from the broker's server and most brokers only keep a few weeks-months worth of M1 data.
 
WHRoeder:
Downloading history is NOT getting it from your broker, it is getting it from metatrader as the pop up message tells you.
Only scrolling back gets it from the broker's server and most brokers only keep a few weeks-months worth of M1 data.

You didn't understand his method... He is not using the 'Download' button. He is just inventing a 'last date' that is so old that the server tries to fill the gap to the max available data. It's actually brilliant... Much better than pressing a button repeatedly like a robot.

 
gordon wrote >>

Excellent! Good find phillip. And it actually makes sense.


LoL I'm glad it makes sense to somebody! It's 'Greek' to me.
The NoDD, ECN, STP broker that I'm going to use: FxOpen.Com only keeps a few days of m1 data on their server that I harvested with your 'scroll back' method and they don't have any 'Data DownLoad' files at all. Lame
. .
I hope they do a better job with v5 so that the strategy tester doesn't have this same lame problems.
.
Thanks for the info and the assistance all.
 
1005phillip wrote >>



Just FYI as someone who use to do what you describe here I know the pain you are experiencing trying to download historical data.

Thanks,
I did it and actually understnad what it was doing.
I'll add a suggestion: Do it in 'batch mode' :
. . . When you are in the history center, go through all of the currency pairs and back date one in each pair as per instructed and then close teh history center and then go through all of the charts. Then back into the history cneter to delet all the really old ones added for the purpose.
. .
Now if I can nly figure out how to convert them wihtout messing them up.
. .
Perhaps one of you experienced savy programmers can automate this and make it into a script?
Splice Period_Converter_ALL_Rev01.mq4 from https://www.mql5.com/en/forum/118275/page2 onto the end of it and it will automate the whole thing!
. .
That would be great for everyone!
Would only have to give one simple easy instruction to newbies like me and would do away with these constant recuring incidents for everyone once and forall.
I'll be happy to do it, IF you want to wait a couple of years so that I actually know enought to do it! LoL
. . .
Thanks to one and all for all of the help (< 8)

 
FourX wrote >>

Thanks,
I did it and actually understnad what it was doing.
I'll add a suggestion: Do it in 'batch mode' :
. . . When you are in the history center, go through all of the currency pairs and back date one in each pair as per instructed and then close teh history center and then go through all of the charts. Then back into the history cneter to delet all the really old ones added for the purpose.
. .
Now if I can nly figure out how to convert them wihtout messing them up.
. .
Perhaps one of you experienced savy programmers can automate this and make it into a script?
Splice Period_Converter_ALL_Rev01.mq4 from https://www.mql5.com/en/forum/118275/page2 onto the end of it and it will automate the whole thing!
. .
That would be great for everyone!
Would only have to give one simple easy instruction to newbies like me and would do away with these constant recuring incidents for everyone once and forall.
I'll be happy to do it, IF you want to wait a couple of years so that I actually know enought to do it! LoL
. . .
Thanks to one and all for all of the help (< 8)


Yeah I do it in batch mode just as you noted...which if you are a MIG or CitiFX user then you will find the whole process rather time consuming nonetheless (they have some 60+ currency pairs). But you should only find yourself in the position of needing to do it once and never more than once (per broker).

I thought about making the process all the more automated but the truth is if you think about you really only need to extract as much historical data as possible from your broker just once. After that point you will be accumulating history data as time progresses anyways (I'm assuming you have MT4 demo accounts setup whose sole purpose is simply to harvest M1 forex data...if not then you should look into setting up something like that anyways).

Whereas the convert M1 -> other time period data is more of a routine thing you'll probably be doing on a weekly or monthly basis anyways (whatever your backtesting learning cycle rate is).
Reason: