MT5 RAM memory voraciousness, problems with reading/writing large files - page 6

New comment
 
Vladimir:

I recommend making a minimal change first, so that memory reallocation is done less frequently. Two lines

m_total_rows++;
ArrayResize(m_cells,m_total_rows*m_total_columns,10000);

in bool CSVReader::AddData(string data_str,bool header) replace with

m_total_rows++;
if (m_total_rows*m_total_columns>ArraySize(m_cells)) ArrayResize(m_cells,2*m_total_rows*m_total_columns);

The number of memory reallocations with copying should become O(log(n,2)) instead of O(n). 20 instead of 600 thousand. Maybe that's enough for you now.

Thank you! I want to let you know what I've got:

1. No change in memory - by 10 gigabytes the current code ate RAM there and there.

2. By speed:

2.1 Old version 574 seconds

2018.07.22 02:08:18.694 Scripts script Pred_Ocenka (Si Splice,M1) loaded successfully
2018.07.22 02:17:52.059 Scripts script Pred_Ocenka (Si Splice,M1) removed

2.2 New version: 138 seconds.

2018.07.22 02:22:16.728 Scripts script Pred_Ocenka (Si Splice,M1) loaded successfully
2018.07.22 02:24:34.569 Scripts script Pred_Ocenka (Si Splice,M1) removed

So you get a 4-time gain, which is pretty good! However, memory is tight, and this is far from all that needs to be loaded....

 
Maxim Dmitrievsky:

very handy :)

So I converted CSV to binary, minus the date.

What it turns out, when running the script took up 1 gigabyte of memory, which compared to 10 is very good. However, it's still a lot :)

In terms of speed - only 16 seconds! Quite good!

2018.07.22 02:35:12.338 Scripts script Pred_Ocenka_02 (Si Splice,M1) loaded successfully
2018.07.22 02:35:28.334 Scripts script Pred_Ocenka_02 (Si Splice,M1) removed
 
Vladimir:

I recommend making a minimal change first, so that memory reallocation is done less frequently. Two lines

m_total_rows++;
ArrayResize(m_cells,m_total_rows*m_total_columns,10000);

in bool CSVReader::AddData(string data_str,bool header) replace with

m_total_rows++;
if (m_total_rows*m_total_columns>ArraySize(m_cells)) ArrayResize(m_cells,2*m_total_rows*m_total_columns);

The number of memory reallocations with copying should become O(log(n,2)) instead of O(n). 20 instead of 600 thousand. Maybe that will be enough for you.

Actually the third parameter for ArrayResize() is specified for a reason... It's a bad change.

Read the documentation

 
Roffild:

Actually the third parameter for ArrayResize() is specified for a reason... a feathery change...

Read the documentation

What did you manage to get from the documentation about the third parameter, useful for this case, when solving the task of lifting into memory .csv created in different programs and having arbitrary size?

Feel free to suggest a better, non-binary change, which increases the speed of memory reallocation (reducing the number of ArrayResize calls) more than binary search...

 
Aleksey Vyazmikin:

Thank you! I'll let you know what comes out:

1. No change in memory - by 10 gigabytes the current code ate RAM there and there.

2. By speed:

2.1 Old version 574 seconds

2.2 New version: 138 seconds.

So you get a 4-time gain, which is pretty good! However, memory is tight, and it's not all that much to load....

After reading, in bool CSVReader::Load(int start_line), after the line

FileClose(filehandle);

insert freeing of memory

ArrayResize(m_cells,m_total_rows*m_total_columns);

Frees up unnecessary 0-50% of memory occupied by m_cells. Only m_cells itself, without cell contents.

 

Now I'm making a small library to work quickly with CSV.

On the screenshot is a test run that goes through in 7 seconds!!! Xeon processor, 3.0 frequency.

First the script makes up the data format for each column. There are 6 columns. Then 1000000 rows are added to the table, then they are filled with numbers from 0 to 999999. According to the data format the numbers can be perceived differently. Then everything is saved into a file.

The file size is 65.4 MB. The whole structure took up 232 MB in memory.

 
Aleksey Vyazmikin:

So I converted CSV to binary, minus the date.

What it turns out, when running the script took up 1 gigabyte of memory, which compared to 10 is very good. However, still a lot :)

In terms of speed - only 16 seconds! Quite good!

Well, the script itself is still crippled.

 
Vladimir:

After reading, in bool CSVReader::Load(int start_line), after line

FileClose(filehandle);

insert memory release

ArrayResize(m_cells,m_total_rows*m_total_columns);

Frees up unnecessary 0-50% of memory occupied by m_cells. Only m_cells itself, without cell contents.

Thanks, but after closing the file/finishing the script, the memory is quickly freed anyway. Here's how to reduce consumption while running....

 
Aliaksandr Hryshyn:

Now I'm making a small library for fast CSV handling.

On the screenshot is a test run that goes through in 7 seconds!!! Xeon processor, 3.0 frequency.

First the script makes up the data format for each column. There are 6 columns. Then 1000000 rows are added to the table, then they are filled with numbers from 0 to 999999. According to the data format the numbers can be perceived differently. Then everything is saved into a file.

The file size is 65.4 MB. The whole structure took up 232 MB in memory.

Quite interesting. Are you planning to publish your programming achievements publicly?

 
Maxim Dmitrievsky:

Well, the script itself is still a mess.

Can you tell me what to fix in it?

#property version   "1.00"
#property indicator_chart_window
#property strict
//---
#property script_show_inputs
//#include <CSVReader.mqh>                              //Класс по чтению информации из файла
#include <CsvWriter_2.mqh>                            //Класс по записи информации в файл

input string  FileLoadSetup="PredFind\\Test\\Pred_Ocenka_Read.csv";
input int Prediktor=1; //Целевая
input double Support=0.01;//Поддержка
input double Relevantnost=70.0;//Достоверность

//---Переменные
int ArrTest[][57];

CsvWriter Printer;
int Statistic;

int StrokTotal=0;
int arrSize=0;
//+------------------------------------------------------------------+
//| Script program start function                                    |
//+------------------------------------------------------------------+
void OnStart()
  {
   int arrTest2[][57];
   int r=FileOpen("PredFind\\Test\\test.bin",FILE_BIN|FILE_READ);
   if(r!=INVALID_HANDLE) 
     { 
      FileReadArray(r,arrTest2,0,WHOLE_ARRAY);
      Print(arrTest2[1][1]);
      arrSize=ArraySize(arrTest2)/57;
      StrokTotal=arrSize;
      Print(arrSize);

     ArrayFree(ArrTest); 
     ArrayResize(ArrTest,arrSize);

      for(int i=1;i<StrokTotal; i++)
        {
         for(int s=1;s<56+1; s++)
           {   
            //Print("s=",s);
            ArrTest[i,s]=arrTest2[i,s];
           }
        }
         Print(ArrTest[2,1]);
      for(int i=1;i<StrokTotal; i++)
        {
         for(int s=3;s<56+1; s++)
           {               
            if (ArrTest[i,s]>=0)ArrTest[i,s]=ArrTest[i,s]*ArrTest[i,1];
            else ArrTest[i,s]=ArrTest[i,s]*ArrTest[i,2];
           }
        }
   }
     else Print("File open failed, error ",GetLastError()); 
/*    
   int h=FileOpen("PredFind\\Test\\test.bin",FILE_BIN|FILE_WRITE);
   FileWriteArray(h,ArrTest,0,WHOLE_ARRAY);
   FileClose(h);
*/
   
    
   string TimeF=TimeToString(TimeLocal(),TIME_DATE|TIME_MINUTES);
   StringSetCharacter(TimeF,13,'_');
   Statistic=Printer.FileCreate(Symbol()+"_"+TimeF+"_""Pred_Ocenka_Write","PredFind\\Test",false,false,EvryTick); //Создание файла для записи     
   Printer.Write(
                 "arr_Data",
                 "arr_Buy_Sell",
                 "arr_Buy_Sell"
                 );

   for(int i=1;i<arrSize; i++)
     {
      Printer.Write(
                    IntegerToString(0),
                    IntegerToString(ArrTest[i,1]),
                    IntegerToString(ArrTest[i,2]),
                    IntegerToString(ArrTest[i,3]),
                    IntegerToString(ArrTest[i,4]),
                    IntegerToString(ArrTest[i,5]),
                    IntegerToString(ArrTest[i,6]),
                    IntegerToString(ArrTest[i,7]),
                    IntegerToString(ArrTest[i,8]),
                    IntegerToString(ArrTest[i,9]),
                    IntegerToString(ArrTest[i,10]),
                    IntegerToString(ArrTest[i,11]),
                    IntegerToString(ArrTest[i,12]),
                    IntegerToString(ArrTest[i,13]),
                    IntegerToString(ArrTest[i,14]),
                    IntegerToString(ArrTest[i,15]),
                    IntegerToString(ArrTest[i,16]),
                    IntegerToString(ArrTest[i,17]),
                    IntegerToString(ArrTest[i,18]),
                    IntegerToString(ArrTest[i,19]),
                    IntegerToString(ArrTest[i,20]),
                    IntegerToString(ArrTest[i,21]),
                    IntegerToString(ArrTest[i,22]),
                    IntegerToString(ArrTest[i,23]),
                    IntegerToString(ArrTest[i,24]),
                    IntegerToString(ArrTest[i,25]),
                    IntegerToString(ArrTest[i,26]),
                    IntegerToString(ArrTest[i,27]),
                    IntegerToString(ArrTest[i,28]),
                    IntegerToString(ArrTest[i,29]),
                    IntegerToString(ArrTest[i,30]),                                                                                
                    IntegerToString(ArrTest[i,31]),
                    IntegerToString(ArrTest[i,32]),
                    IntegerToString(ArrTest[i,33]),
                    IntegerToString(ArrTest[i,34]),
                    IntegerToString(ArrTest[i,35]),
                    IntegerToString(ArrTest[i,36]),
                    IntegerToString(ArrTest[i,37]),
                    IntegerToString(ArrTest[i,38]),
                    IntegerToString(ArrTest[i,39]),
                    IntegerToString(ArrTest[i,40]),
                    IntegerToString(ArrTest[i,41]),
                    IntegerToString(ArrTest[i,42]),
                    IntegerToString(ArrTest[i,43]),
                    IntegerToString(ArrTest[i,44]),
                    IntegerToString(ArrTest[i,45]),
                    IntegerToString(ArrTest[i,46]),
                    IntegerToString(ArrTest[i,47])
                    );
     }
  }
12345678
New comment