Accurate historical data years backward
I am looking for accurate historical prices of tick by tick or 1-minute resolution as much as 10 years backward or alike. Ans> Search for Dukascopy Data on mt4.com or google.com
1. I was told the data represents only bid prices - the ask prices are not taken into consideration. I guess that simply adding a fixed spread to the prices would be misleading. Therefore I am looking for either a source of both bid and ask data, or alternatively transaction prices. Ans> True
2. The data contains many 'holes' - tens of minutes, even more, without any data, in the middle of the week. I need a consistent sequence of prices. Ans> True
3. I would like to understand what the open and close prices represent in practice. Since the close price of a bar and the open price of the next bar usually differ, it makes sense they probably refer to first/last price changes within a time frame - not the price offered at a given time. Ans> True
My Bias,
*Unless you're looking to test a Scalper, looking for Tick data is Pointless.
*If you are looking to test a Scalper, downloading Tick data which is Not from your broker is Pointless.
*Tick data is Server and Settings sensitive in the Real Forex world from what I hear. Two MT's running on the same computer, with the same broker, will show different bid/ask Ticks.
*Downloading Tick data is Pointless unless you want to be (what was your word) impractical. Thinking is gonna give you a better estimate is just Not True. Nor worth the effort.
*Unless you're looking to test a Scalper, looking for Tick data is Pointless.
*If you are looking to test a Scalper, downloading Tick data which is Not from your broker is Pointless.
*Tick data is Server and Settings sensitive in the Real Forex world from what I hear. Two EA's running on the same computer, with the same broker, will show different bid/ask Ticks.
*Downloading Tick data is Pointless unless you want to be (what was your word) impractical. Thinking is gonna give you a better estimate is just Not True. Nor worth the effort.
Hello,
The system is not scalping-based. My signals are based on 5, 10, 15 minutes etc., however using 1-minute data enable me to build bars in multiple time-frames, and enable me to divide execution prices into many portions, resulting in some kind of the average price that could be actually obtained within the time frame, rather than select a specific point of time and assume the entire transaction is to be executed at that price. I believe (you might have good reasons not to agree, though) such conditions are more difficult to reliably simulate over time.
Therefore, tick-by-time is not critical. I would do very well with 1-minute bars. The problem is that as I said, MT data is meant to be bid prices only. I would like to have ask prices as well, since simply adding a spread for all sell operations is unreliable.
Thank you. (why bias?)
The spread for EUR/USD is usually 2 pips, I'm no expert, but I'm thinking it means bid/ask are about 2 pips apart. 1-Minute bars can have as much as 100 Ticks or more within them but all u get is High,Low,Open and Close. Tick is Not time based, it's price movement based. 1-Minute charts however is Time based and broker time at that.
I guess the question I would ask myself is which is more reliable 2-pips difference or up-to-over 100 pips difference. The missing data with Metaquotes data would be my biggest concern not the difference in Bid/Ask. If your broker does not have Fixed (they always reserve the right to change) spreads. That makes matters worse.
Seems you're looking for something reliable but for what? For the 1-Minute Data or For the Real-Broker? Bias because this is Just My Opinions.
The spread for EUR/USD is usually 2 pips, I'm no expert, but I'm thinking it means bid/ask are about 2 pips apart. 1-Minute bars can have as much as 100 Ticks or more within them. Tick is Not time based, it's price movement based. 1-Minute charts however is Time based and broker time at that.
I guess the question I would ask myself is which is more reliable 2-pips difference or up-to-over 100 pips difference. The missing data with Metaquotes data would be my biggest concern not the difference in Bid/Ask. If your broker does not have Fixed (they always reserve the right to change) spreads. That makes matters worse.
Seems you're looking for something reliable but for what? For the 1-Minute Data or For the Real-Broker? Bias because this is Just My Opinions.
The broker is connected to ECN. It is not market-maker, therefore there is no fixed spread. Adding a spread to the bid prices can give some estimation but it would probably not reach the level of reliability I'm expecting. That's the reason I'm looking for both bid and ask prices that represent the real ECN prices. The average spread for EURUSD would be generally smaller than 2 pips and even from 1 pip, but for certain pairs it can differ widely.
What I was saying is that if, as you said, 1-minute bars would be only time based, then there were not (or were rarely) differences between close and following open prices. I tend to understand that the open and close prices represent first and last price change, NOT the best price at the first and last second of the bar.
I am looking for reliability of 1-minute bars: need to know what was the bid/ask prices at a specific time (or alternatively last transaction prices over the ECN), and as you correctly said, without holes in the data. MT's data has many holes - you sometimes need to interpolate whole hours, which is not exactly my idea for reliability of simulation.
https://www.mql5.com/en/forum/118275
You won't find bid and ask price data. Nobody stores this info from what I have been able to determine.
Why is that you may ask? Mostly because that degree of accuracy is irrelevant. If your strategy is such that it needs to be finely optimized on historical data that is correct down to the volatile spread and specific bid and ask prices then you've pretty much got yourself a guaranteed useless EA when it comes time to make profit from future unknown timeseries.
Backtesting is useful for proofing out the concepts of your EA in a controlled environment. Don't fall for the "quality/quantity of data" fallacy.
At the same time, yes having ridiculously lousy historical data (i.e. the metaquotes data accessed via the download button) is not what anyone considers to be a "controlled environment" worth proofing out your EA's mechanical soundness.
The simple answer in my mind to what you're looking for but I didn't want to just say in the first response is that there's NO reliable way to know what the bid/ask price is at a specific time.
*First you have to find a Data Source which have both Bid/Ask 1-Minute (I'm not sure if Dukascopy Data comes in both Bid/Ask. There's no other source providing m1 data that I'm aware of for that period of time)
*The Source would need to be Ecn broker who is providing the best Bid/Ask at that particular top of Minute. (Not sure if Dukascopy is Ecn, therefore their Bid/Ask could vary form what the Best Price would have been.)
*The Spreads for Ecn can go anywhere between 0-5 during news time and alike. (So what are you going to add to your Bid/Ask perfect data your download)
Forex is an OTC market, there's no centralized Bid/Ask anywhere at anytime. Trying to simulate a good estimate is Pointless. I'd just assume the worse case scenario. Give myself a range of 2-pips and call it a day.
Yep, and they are bid prices. Always bid prices.
If you are using MT4's built-in strategy tester then you do not have to add spread to anything, it will use the most recently quoted spread from the broker for the given currency pair when determing the ask price during the backtest.
Actually I've been written my own genuine platform, the simulation itself does not run over MT. If the prices are bid, I guess I would like to add that extra spread only when selling (or double it when selling).
So what do you think about 'guessing' a spread to add according to a factor of recent volatility? or a 'typical price' (average of H/L) etc.?
My question about the meaning of open/close prices still remains open, though. Does the close price represents last change and open price represents first change, or are they assumed to represent the price at the second or so when the minute ends/starts?
This is crucial, because there are two options:
1. If the prices represent first and last *change* of price within the time frame, execution price may well be calculated by adding a spread to the close price (which is typically used as the signal price).
2. If the prices represent bids at the exact beginning or end of the time frame (so open and close prices represent a change within an intra-second frame, or any other really small frame), open prices should be used to calculate execution price. This is my current policy.
What do you think?
Ah, thanks for explaining. I should have added the caveat in my original post that I do not use the precompiled forexite data. I used their QuoteRoom program to download the data and exported myself. I assumed the precompiled stuff was the same but I never verified.
Regardless, I recommend using the FXDD data. I only use the forexite data to plug holes in my FXDD and diskcopy data sets and have never encountered issue doing such. Maybe I just missed the gaping holes?
Ah, thanks for explaining. I should have added the caveat in my original post that I do not use the precompiled forexite data. I used their QuoteRoom program to download the data and exported myself. I assumed the precompiled stuff was the same but I never verified.
Regardless, I recommend using the FXDD data. I only use the forexite data to plug holes in my FXDD and diskcopy data sets and have never encountered issue doing such. Maybe I just missed the gaping holes?
Hi,
Probably too late for this thread, but, anyone tested the historical data from: http://www.histdata.com ?
The guy's have FREE historical data MetaTrader, 66 pairs, FREE (M1 and Tick).
Thanks in advance for the help.
Cheers!
- Free trading apps
- Over 8,000 signals for copying
- Economic news for exploring financial markets
You agree to website policy and terms of use
Hello,
I am looking for accurate historical prices of tick by tick or 1-minute resolution as much as 10 years backward or alike.
I downloaded EURUSD data from MT, but there are some issues with that:
1. I was told the data represents only bid prices - the ask prices are not taken into consideration. I guess that simply adding a fixed spread to the prices would be misleading. Therefore I am looking for either a source of both bid and ask data, or alternatively transaction prices.
2. The data contains many 'holes' - tens of minutes, even more, without any data, in the middle of the week. I need a consistent sequence of prices.
3. I would like to understand what the open and close prices represent in practice. Since the close price of a bar and the open price of the next bar usually differ, it makes sense they probably refer to first/last price changes within a time frame - not the price offered at a given time. I think a reliable simulation should separate between decision price and execution price, i.e. a system makes a decision according to close prices and calculates execution price according to the open price of the next bar. This enables a simulator to avoid the arbitrary assumption that the last price at the time of the signal will still be available. It is better to tune the simulator to see what would happened if you were to react to the signal, rather than wrongly assume any price will remain available until you perform.
Now, and this is critically important, if open prices represent the first bid or ask price change within the time frame of the bar, this is way different than representing the first trade.
In an ideal world you would have to maintain two series of bars: bid and ask, and calculate execution prices according to the operation - buy or sell. Alternatively, transaction prices can provide a good approximation of average price movement.
Since MT historical prices are bid-only and full of holes, where is reliable, consequent data can be found - without holes and for the ask prices as well?
I think using bid prices only for backward test is futile, and any serious backtests (years) in small time frames on MT data is impractical.
Answers will be appreciated.
Thanks.