How to speed up big array reading? - page 2

[R4tna C]

Lativ_:

Hello 

I'm trying to read a news file on back test from 2000 until present.

Is there a way to read the Array quicker? because what I'm trying for now is painfully slow

As file I/O is often the largest bottleneck, have you tried reading all lines into memory (an array, or set of arrays if the data set is very large) and then performing the complex processing while in-memory?

Could be faster if you pre-fetch and cache into RAM then do the complex manipulations, but you would have to test out whether reading the file first in one loop is better.

Also file I/O can be affected by fragmentation - if you can place the file on a separate unencrypted file system and defragment it, it could also help (if you want to more info on this, "clustering factor" in Oracle gives a good explanation). Such disk I/O optimization techniques used to be a big deal some years back, but in this age of SSD drives it may be not be as effective now

[R4tna C]

R4tna C #:

As file I/O is often the largest bottleneck, have you tried reading all lines into memory (an array, or set of arrays if the data set is very large) and then performing the complex processing while in-memory?

Could be faster if you pre-fetch and cache into RAM then do the complex manipulations, but you would have to test out whether reading the file first in one loop is better.

Also file I/O can be affected by fragmentation - if you can place the file on a separate unencrypted file system and defragment it, it could also help (if you want to more info on this, "clustering factor" in Oracle gives a good explanation). Such disk I/O optimization techniques used to be a big deal some years back, but in this age of SSD drives it may be not be as effective now

I just discovered MQL5 allows parallel processing - that was another way we used to speed up processing of large datasets.

https://www.mql5.com/en/articles/197

[Lativ_]

William Roeder #:

Stop doing the same things multiple times
Simplify your code

Wow

[Lativ_]

R4tna C #:

I just discovered MQL5 allows parallel processing - that was another way we used to speed up processing of large datasets.

https://www.mql5.com/en/articles/197

Thank you for this

[Lativ_]

If you read this and also learning mql5. Having to deal with a large Array looping? I do not recommend this way of looping. I have found the most easiest and efficient way using ROM and its OOP. Please read this article

https://www.mql5.com/en/articles/719

[R4tna C]

Lativ_ #:

If you read this and also learning mql5. Having to deal with a large Array looping? I do not recommend this way of looping. I have found the most easiest and efficient way using ROM and its OOP. Please read this article

https://www.mql5.com/en/articles/719

Could you elaborate please? Did you mean RAM rather than ROM?

The article I posted was not this one 

[Lativ_]

R4tna C #:

Could you elaborate please? Did you mean RAM rather than ROM?

The article I posted was not this one 

Yes I meant RAM.

I didn't know about  MQL5 CArrayObj class