The most effective way to store Tick Data

New comment
 

Hello Everyone,

I'm trying to store tickdata and not sure what would be the most effective way to store this data. The Tick Data I want to store will be up to 1 minute of data and based on this I would like to do some calculations therefore I will require to access that data at any millisecond time interval.

But I'm not sure if I should store it in an

1) Array

2) File

3) Database

4) Better method i'm not aware of?

Any advice in the direction I should head would be highly appreciated.

Thank you in advance all.

Audai,

 
Audai Louri:

Hello Everyone,

I'm trying to store tickdata and not sure what would be the most effective way to store this data. The Tick Data I want to store will be up to 1 minute of data and based on this I would like to do some calculations therefore I will require to access that data at any millisecond time interval.

But I'm not sure if I should store it in an

1) Array

2) File

3) Database

4) Better method i'm not aware of?

Any advice in the direction I should head would be highly appreciated.

Thank you in advance all.

Audai,

Wouldn't harm if you tried the array with milsec fields and watch out for Ticks that get lost in high frequency hours.
 
Depends... on the size of tick data, why you're storing it and how frequently you need to access this data.
 
Raymond Ogechukwu Omeh #:
Depends... on the size of tick data, why you're storing it and how frequently you need to access this data.

Thank you Raymond.

Honestly if I'm storing tick data in milliseconds I assume its over 60,000 chars and I know that the array can only hold 40,000 chars max.

So there is an option of creating multiple arrays, as receiving data will be faster from RAM rather than file or DB.

The short of it I'm storing data in milliseconds, or the closest I can get to storing in milliseconds, and requesting them per tick.

 
pennyhunter #:
Wouldn't harm if you tried the array with milsec fields and watch out for Ticks that get lost in high frequency hours.

Hi Pennyhunter,

That was one of my concerns, if I set the timer to 1 milisec that tickdata can be lost.

But I believe array is the fastest option there.

My only concern is that arrays can only hold 40,000 chars.

 
Audai Louri #:

My only concern is that arrays can only hold 40,000 chars.

Would you care to back that up with a source?
 
pennyhunter #:
Would you care to back that up with a source? 
void OnTick()
  {
   double test[70000];
  }

Its really more of a warning


 
Audai Louri #:

Its really more of a warning


Oh... thanks for the concern, I might have to reevaluate the scalability of a current project...

It would be nice to know if there are possibilities to not have that limitations... Local variable as in opposed to global?
 
Audai Louri #:

Hi Pennyhunter,

That was one of my concerns, if I set the timer to 1 milisec that tickdata can be lost.

But I believe array is the fastest option there.

My only concern is that arrays can only hold 40,000 chars.

Dynamic arrays can have 2147483647 elements.

https://www.mql5.com/en/docs/array/arrayresize

Documentation on MQL5: Array Functions / ArrayResize
Documentation on MQL5: Array Functions / ArrayResize
  • www.mql5.com
ArrayResize - Array Functions - MQL5 Reference - Reference on algorithmic/automated trading language for MetaTrader 5
 
Alexandre Borela #:

Dynamic arrays can have 2147483647 elements.

https://www.mql5.com/en/docs/array/arrayresize

Thanks Alexandar,

I decided to go with the file storage option. Was to worried about the warning I got.

 

Have you tried global variables just to be sure?


Audai Louri #:

Hi Pennyhunter,

That was one of my concerns, if I set the timer to 1 milisec that tickdata can be lost.


Also did you look into how in MQL5/Indicators/Examples the calculations are limited to the last bar?

New comment