MT5 RAM memory voraciousness, problems with reading/writing large files - page 5

 
Maxim Dmitrievsky:

so it's a 2-dimensional array, a matrix. Write it.

Can you show me an example of how to write and read multidimensional arrays? It's clear in theory, but the implementation is not clear.

 
Aleksey Vyazmikin:

Can you show me an example of how to write and read multidimensional arrays? It's clear in theory, but the implementation is unclear.

Just like one-dimensional arrays, with WriteArray()

 
Aleksey Vyazmikin:

I'm comfortable using the class, I've been using it for 3 years now, since MT4 - it's handy, but damn, it's slow.

Of course it's easy to read a string, but turning it all into a working cell structure is much harder.

What do you mean I can't use classes? I do, and besides, the metaquote employee wrote the class. And then, I actually hired a person, paid and expected to get a finished product, but I was told that it's OK to be slow... I believed them, but now I've got the memory overkill as well - I'm outraged.

The class is designed for small files. It doesn't matter at all who created the class. Now we need a different approach to reading a file, so that class handles that file poorly.

 
Roffild:

The class is designed for small files. It does not matter at all who created the class. Now you need a different approach to reading a file, so that class doesn't handle that file well.

I'll perpetuate it. Fucked up, thank you.

 
Maxim Dmitrievsky:

just like one-dimensional ones, through WriteArray()

Found an example, I'll have to try it, I'll have to write two files - for general understanding and probably to speed up processing.

Thank you.

Added: I understand that you can write only 1 array at a time, which of course is not very convenient.
 
Roffild:

The class is designed for small files. It does not matter at all who created the class. Now you need a different approach to reading a file, so that class does not handle that file well.

If memory serves me correctly, I immediately said that we need to work with large files, and for example, a large file was given - tens of megabytes, if not a hundred (we need to watch the correspondence, and in the current format it is not convenient to do).

 
Aleksey Vyazmikin:

Found an example, have to try it, will have to write two files - for general understanding, and probably to speed up processing.

Thank you.

Added: I understand that you can write only 1 array at a time, which of course is not very convenient.

Very convenient :)

 
Maxim Dmitrievsky:

very handy :)

Not convenient because you have to create a separate file for different types of data.

 
Aleksey Vyazmikin:

Not convenient, as you have to create a separate file for different types of data.

Well, you can put it in the same directory so that it is not scattered. You won't have any problems. That's what I always do for experts. I immediately create a directory called program name/account number and I put all the files in it.

 

I recommend making a minimal change first, so that memory reallocation is done less frequently. Two lines

m_total_rows++;
ArrayResize(m_cells,m_total_rows*m_total_columns,10000);

in bool CSVReader::AddData(string data_str,bool header) replace with

m_total_rows++;
if (m_total_rows*m_total_columns>ArraySize(m_cells)) ArrayResize(m_cells,2*m_total_rows*m_total_columns);

The number of memory reallocations with copying should become O(log(n,2)) instead of O(n). 20 instead of 600 thousand. Maybe that's enough for you now.

Reason: