Question regarding mql is getting slower with size

[Marcin Rutkowski]

Hi  ...I'm in the middle of developing algorithm which is involving a bit of data, and I noticed  on multiple occasions  that adding size to the array is  getting slower with size if we pass certain size of array (each case is different) . below I created example what i mean

This is just a simple example of the problem what i have. (in this case) If  the size of the string value pass 10000  lines then adding lines after 10000 is much  much  slower then first 10000 ("string" is just example)   ...I used this code just for the example but the same is happening with arrays, after certain size to add records to the array is getting slower and slower, and with algorithms which are calculating many permutations this is becoming a problem

And my real question is : is this a nature of mql and I have to move to different environment (c#) or I can do something about it, or this is just my computer :)  (i7 4gen)

int OnInit()
  {

   string locFileName="TestReport.txt";
   if(!FileDelete(locFileName))
      Print("No File");

   string stringBase="01234567890123456789012345678901234567890123456789";

   string stToFile="";

   int locTotalLines=30000;

   for(int i=0;i<locTotalLines;i++)
     {

      if(i%100==0)
         Comment("line added = "+(string)i+"      total of = "+(string)locTotalLines);   

      stToFile=stToFile+stringBase+"\n";

     }

   int handle=FileOpen(locFileName,FILE_TXT|FILE_READ|FILE_WRITE," ");// File opening
   FileSeek(handle,0,SEEK_END);

   uint saved=FileWriteString(handle,stToFile);
   FileClose(handle);
   if(saved>0)
     {
      Print(" bytes written = "+(string)saved);
      Print ("File created :  MQL5\\Files\\"+locFileName);
     }

   return(INIT_SUCCEEDED);
  }

Thank You  .

I'm far from  being Pro  so I hope I'm not asking trivial question on this forum :)   ...but i noticed if size (in memory) of an array of string is over 1Mbytes then everything is getting quickly slower

[Flavio Javier Jarabeck]

Marcin Rutkowski:

Hi  ...I'm in the middle of developing algorithm which is involving a bit of data, and I noticed  on multiple occasions  that adding size to the array is  getting slower with size if we pass certain size of array (each case is different) . below I created example what i mean

This is just a simple example of the problem what i have. (in this case) If  the size of the string value pass 10000  lines then adding lines after 10000 is much  much  slower then first 10000 ("string" is just example)   ...I used this code just for the example but the same is happening with arrays, after certain size to add records to the array is getting slower and slower, and with algorithms which are calculating many permutations this is becoming a problem

And my real question is : is this a nature of mql and I have to move to different environment (c#) or I can do something about it, or this is just my computer :)  (i7 4gen)

Thank You  .

I'm far from  being Pro  so I hope I'm not asking trivial question on this forum :)   ...but i noticed if size (in memory) of an array of string is over 1Mbytes then everything is getting quickly slower

Your code is very risky... Work in Chunks of data. The probability of having a memory bloat or a disk corruption is bigger the way you are doing it...

;)

[Marcin Rutkowski]

Minions Labs:

Your code is very risky... Work in Chunks of data. The probability of having a memory bloat or a disk corruption is bigger the way you are doing it...

;)

Thanks' for response

For contrast, if I create objects using new operator, I can fill the all memory super quick and release it  , Only when I increase size of single array or string in above example this slowdown is happening. Now Im trying to decide how to store the results, and also the speed is important.  (saving to the file is less important, I included it only to easy display size of  stored string)  … I will be surprise if problem is with my equipment  ...did You run the code on Your machine and did You noticed slowdown after 10000 ??   ...Im curious ??

[Alain Verleyen]

Marcin Rutkowski:

Thanks' for response

For contrast, if I create objects using new operator, I can fill the all memory super quick and release it  , Only when I increase size of single array or string in above example this slowdown is happening. Now Im trying to decide how to store the results, and also the speed is important.  (saving to the file is less important, I included it only to easy display size of  stored string)  … I will be surprise if problem is with my equipment  ...did You run the code on Your machine and did You noticed slowdown after 10000 ??   ...Im curious ??

It will happen on any machine and any language.

Each time you add to a string (or resize an array), the code need to allocate memory for it, it needs to find a contiguous memory area and eventually copy the old content to the new place. You are doing that 30,000 times with increasing size.

Instead of that you should reserve memory once for all or at least in small step, and in this particular case with strings, use StringAdd() instead of concatenation.

   string stringBase="01234567890123456789012345678901234567890123456789";

   string stToFile="";
   StringInit(stToFile,1531000,'\0');
   int locTotalLines=30000;

   for(int i=0;i<locTotalLines;i++)
     {
      if(i%100==0)
         Comment("line added = "+(string)i+"      total of = "+(string)locTotalLines);

      StringAdd(stToFile,stringBase+"\n");
     }

Original :                 145344 ms.
With StringAdd :     3938 ms.
With buffer init :    1593 ms.

100 times faster.