Looking for quality 1 Min OHCL data (doesn't matter the broker)

 
Tried Alpari (wrong volumes and crazy spreads), tried IC Markets (wrong swaps and huge gaps), tried Metaquotes (few months of data only); tried to import tick data from dukascopy on a custom symbol, but when i try to test it the "stop" sound triggers, and no feedback is provided on the journal, so i have no idea what is going on...

Why even create such powerful backtesting platform if you are not going to provide or hint the most elemental thing needed to use it?  Hell, you may even SELL the damn thing for thousands (are you selling it somewhere?), i really don't understand how everything else is built in a "fool's proof" manner, but "God knows" where you can get a few MB's of OHCL that isn't completely messed up.

Any suggestions? (don't mind the rant). 
 
abaldenegro:
Tried Alpari (wrong volumes and crazy spreads), tried IC Markets (wrong swaps and huge gaps), tried Metaquotes (few months of data only); tried to import tick data from dukascopy on a custom symbol, but when i try to test it the "stop" sound triggers, and no feedback is provided on the journal, so i have no idea what is going on...

Why even create such powerful backtesting platform if you are not going to provide or hint the most elemental thing needed to use it?  Hell, you may even SELL the damn thing for thousands (are you selling it somewhere?), i really don't understand how everything else is built in a "fool's proof" manner, but "God knows" where you can get a few MB's of OHCL that isn't completely messed up.

Any suggestions? (don't mind the rant). 

Beep!

 
abaldenegro: Beep!

What is it you would like us to say? Your query is much too general to be able to answer in just a few posts!

Many traders and users are successful in using and importing OHLC and Tick data into MetaTrader v4 without problems. Obviously that is not a task for beginners nor for casual users, but for power users.

If you are unable to learn it on your own or you want someone to put in the hours to teach you, then you will have to consider compensating them for it (such as via the Freelance section).

Alternatively consider using Metatrader v5.

 
Fernando Carreiro:

What is it you would like us to say? Your query is much too general to be able to answer in just a few posts!

Many traders and users are successful in using and importing OHLC and Tick data into MetaTrader v4 without problems. Obviously that is not a task for beginners nor for casual users, but for power users.

If you are unable to learn it on your own or you want someone to put in the hours to teach you, then you will have to consider compensating them for it (such as via the Freelance section).

Alternatively consider using Metatrader v5.

Dude, i want something general ---> 1M OHCL data for forex that is not messed up, either from a broker or external source that is not included on the list i provided. Obviously it is for MT5 since this is in the MQL5 forums. I already have and use high quality tick data for MT4, but that is not what i am asking for. I am trying to test multi-currency EA's on MT5, but as i said, there are no pointers on how to get decent data anywhere. Already tried to import ticks on a custom MT5 symbol and it just crashes before the test with no feedback whatsoever. When i ask the MT5 staff directly they point me to this forum.

I can also use google, research and test things by myself, but the reason i am asking here is because i assume experienced veterans may have tried long term backtesting/optimizing only to find these flaws/holes that make their efforts completely unreliable and point out what i may be missing. :)  

 
abaldenegro:

Dude, i want something general ---> 1M OHCL data for forex that is not messed up, either from a broker or external source that is not included on the list i provided. Obviously it is for MT5 since this is in the MQL5 forums. I already have and use high quality tick data for MT4, but that is not what i am asking for. I am trying to test multi-currency EA's on MT5, but as i said, there are no pointers on how to get decent data anywhere. Already tried to import ticks on a custom MT5 symbol and it just crashes before the test with no feedback whatsoever. When i ask the MT5 staff directly they point me to this forum.

I can also use google, research and test things by myself, but the reason i am asking here is because i assume experienced veterans may have tried long term backtesting/optimizing only to find these flaws/holes that make their efforts completely unreliable and point out what i may be missing. :)  

On MT5 you don't need to import anything! It will download the necessary data automatically (provided your EA is correctly programmed to so).

So, the question should not be "How to import OHLC data?", but rather "How to fix EA to function correctly with multi-symbol?" (see link below for synchronizing the data)!

Are you the coder of the EA being tested?

Documentation on MQL5: Timeseries and Indicators Access / Organizing Data Access
Documentation on MQL5: Timeseries and Indicators Access / Organizing Data Access
  • www.mql5.com
Before price data become available in the MetaTrader 5 terminal, they must be received and processed. To receive data, connection to the MetaTrader 5 trade server must be established. Data are received in the form of packed blocks of minute bars from the server upon the request of a terminal. The mechanism of server reference for data doesn't...
 
Fernando Carreiro:

On MT5 you don't need to import anything! It will download the necessary data automatically (provided your EA is correctly programmed to so).

So, the question should not be "How to download OHLC data", but rather "How to fix my EA do it" (see link below for synchronizing the data)!

Are you the coder of the EA being tested?

EDIT: Synchronization of the Terminal Data and Server Data

I downloaded it like this from various brokers:


But that's how i get the horrible spreads, gaps, uneven volume data from Alpari, IC markets and Metaquotes.

The way you say it downloads data (automatically, before the test), is not reliable either, as it fills the missing 1M bars with 1H ones, doesn't tell me that the period is uncomplete (if i tell it to go from 2009, it just uses what is available from the broker and doesn't even hints at you anything), the problem with this is that there is not a way to see the data itself, you can only inferr the quality from the test results by watching it go candle by candle. Any EA i have tested on the platform tries to pre-download data, i have not included anything to the code for that (i use Fxdreema, so it may very well be added), so you are telling me i would get different results if i add that line of code to an EA??

 
abaldenegro: I downloaded it like this from various brokers:


But that's how i get the horrible spreads, gaps, uneven volume data from Alpari, IC markets and Metaquotes.

The way you say it downloads data (automatically, before the test), is not reliable either, as it fills the missing 1M bars with 1H ones, doesn't tell me that the period is uncomplete (if i tell it to go from 2009, it just uses what is available from the broker and doesn't even hints at you anything), the problem with this is that there is not a way to see the data itself, you can only inferr the quality from the test results by watching it go candle by candle. Any EA i have tested on the platform tries to pre-download data, i have not included anything to the code for that (i use Fxdreema, so it may very well be added), so you are telling me i would get different results if i add that line of code to an EA??


You can easily see the quality of the data by first pulling up charts for the symbols in question for the time period required. You can even analyse the data at the tick level as well in the symbols section. If it is no good then have a look at data from other brokers in order to compare the quality. If your broker is not providing quality data, then consider another broker. However, I would first rule out other possible problems first. I don't use either of those brokers, so cannot comment, but I have no problems with my brokers data (not allowed to advertise, so you will have to do some research).

Any kind of code produced by EA generators or builders is just not going to cur it. They produce bad, buggy, bloated code of low quality. A properly working EA has be coded by a skilled high-quality human coder (and I don't mean paying $30 for an EA). High quality code does not come cheap (think $500+) nor is it developed in just a couple of days. Skilled High-Quality coders are also a very scarce commodity and you will not find many on the Freelance section as most of them work by word-of-mouth. So be very careful when selecting a developer in the Freelance section. Do your homework and verify what is to be accomplished.

That being said, you will need to refine and build up your skills a great deal more. Think years (not months nor weeks, but years) of research, development, testing, etc. to reach a skilled, profitable trading level.

EDIT: PS! On another of your threads, a moderator (who is a highly skilled coder), informed you that Alpari data did not seem to present any problems in quality. So maybe you should also check your system to make sure the fault is not on your end instead of the broker's data.

 
Fernando Carreiro:


You can easily see the quality of the data by first pulling up charts for the symbols in question for the time period required. You can even analyse the data at the tick level as well in the symbols section. If it is no good then have a look at data from other brokers in order to compare the quality. If your broker is not providing quality data, then consider another broker. However, I would first rule out other possible problems first. I don't use either of those brokers, so cannot comment, but I have no problems with my brokers data (not allowed to advertise, so you will have to do some research).

Any kind of code produced by EA generators or builders is just not going to cur it. They produce bad, buggy, bloated code of low quality. A properly working EA has be coded by a skilled high-quality human coder (and I don't mean paying $30 for an EA). High quality code does not come cheap (think $500+) nor is it developed in just a couple of days. Skilled High-Quality coders are also a dime-a-dozen and you will not find many on the Freelance section as most of them work by word-of-mouth. So be very careful when selecting a developer in the Freelance section. Do your homework and verify what is to be accomplished.

That being said, you will need to refine and build up your skills a great deal more. Think years (not months nor weeks, but years) of research, development, testing, etc. to reach a skilled, profitable trading level.

That was precisely the thing i needed help with (which broker gives reliable data), but nevermind. Right now, i am just downloading it from Dukascopy, it will be painfully slow to get the history for 28 symbols but at least i don't have to bother eyeing every single candle for huge broker "woopsies" . That wouldn't be practical at all since it is 10 years with 525600 (or so) minutes each x 28 symbols. The good brokers i thought could store some data don't use MT5 sadly (i wonder why).

You are right in saying that EA builders may generate some unnecessary bloat code, but i would rather pay 100 bucks a year for the ability to build, modify and test the ideas myself (most of it being simple stuff really). Of course i can read and write basic stuff but i find it a lot faster and effective to edit the logic using just diagrams, as i matter of fact the only bugs i have found so far are related to my own logic, not by the builder itself. I don't really want to pay someone $500 to build M.A systems or something i can build myself in a few hours, are you joking? Maybe if i consider getting into more complex/tricky stuff i would pay someone. 

 
abaldenegro:

That was precisely the thing i needed help with (which broker gives reliable data), but nevermind. Right now, i am just downloading it from Dukascopy, it will be painfully slow to get the history for 28 symbols but at least i don't have to bother eyeing every single candle for huge broker "woopsies" . That wouldn't be practical at all since it is 10 years with 525600 (or so) minutes each x 28 symbols. The good brokers i thought could store some data don't use MT5 sadly (i wonder why).

You are right in saying that EA builders may generate some unnecessary bloat code, but i would rather pay 100 bucks a year for the ability to build, modify and test the ideas myself (most of it being simple stuff really). Of course i can read and write basic stuff but i find it a lot faster and effective to edit the logic using just diagrams, as i matter of fact the only bugs i have found so far are related to my own logic, not by the builder itself. I don't really want to pay someone $500 to build M.A systems or something i can build myself in a few hours, are you joking? Maybe if i consider getting into more complex/tricky stuff i would pay someone. 

I will answer you via PM what my broker is!
 
Fernando Carreiro:


You can easily see the quality of the data by first pulling up charts for the symbols in question for the time period required. You can even analyse the data at the tick level as well in the symbols section. If it is no good then have a look at data from other brokers in order to compare the quality. If your broker is not providing quality data, then consider another broker. However, I would first rule out other possible problems first. I don't use either of those brokers, so cannot comment, but I have no problems with my brokers data (not allowed to advertise, so you will have to do some research).

Any kind of code produced by EA generators or builders is just not going to cur it. They produce bad, buggy, bloated code of low quality. A properly working EA has be coded by a skilled high-quality human coder (and I don't mean paying $30 for an EA). High quality code does not come cheap (think $500+) nor is it developed in just a couple of days. Skilled High-Quality coders are also a very scarce commodity and you will not find many on the Freelance section as most of them work by word-of-mouth. So be very careful when selecting a developer in the Freelance section. Do your homework and verify what is to be accomplished.

That being said, you will need to refine and build up your skills a great deal more. Think years (not months nor weeks, but years) of research, development, testing, etc. to reach a skilled, profitable trading level.

EDIT: PS! On another of your threads, a moderator (who is a highly skilled coder), informed you that Alpari data did not seem to present any problems in quality. So maybe you should also check your system to make sure the fault is not on your end instead of the broker's data.

Here is an example of what i am talking about (unexplainable spreads), it's on tuesday, no weekend and no bank holiday. 


Also unexplainable volumes! (tick volume isn´t showing quite right). 



So, it is not good enough, unless you are testing daily or above and using closes to trade (no TP/SL). The would effect magnify if this happens on multiple currencies.  

 
abaldenegro: Here is an example of what i am talking about (unexplainable spreads), it's on tuesday, no weekend and no bank holiday. 


Also unexplainable volumes! (tick volume isn´t showing quite right).

So, it is not good enough, unless you are testing daily or above and using closes to trade (no TP/SL). The would effect magnify if this happens on multiple currencies. 

NB! I can confirm that the Alpari feed data on my end was exactly the same as the OPs. Therefore, it is not a issue on the OPs end but indeed a problem with the brokers data!
Reason: