Will the EA crash ?

New comment
 

I've got an ea that downloaded all available ticks from a broker for a symbol . 

It reached 250 million ticks , and , the MqlTick structure is 52 bytes per tick. (13 gigabytes)

It placed it all in memory .I've got decent RAM on the machine so it held.

If someone with 8GB of RAM runs it will they get OutOfMemory , Array out of range or will the swap file be used ? (if its called the swap file , the virtual memory)

 
Lorentzos Roussos:

I've got an ea that downloaded all available ticks from a broker for a symbol . 

It reached 250 million ticks , and , the MqlTick structure is 52 bytes per tick. (13 gigabytes)

It placed it all in memory .I've got decent RAM on the machine so it held.

If someone with 8GB of RAM runs it will they get OutOfMemory , Array out of range or will the swap file be used ? (if its called the swap file , the virtual memory)

Why are you thinking it placed all in memory ?

 
Alain Verleyen #:

Why are you thinking it placed all in memory ?

that's why it was slowing down ?

 
Lorentzos Roussos #:

that's why it was slowing down ?

Slowing down ? It's hard to know what you are talking about exactly.

You can check the memory used by MT5 with Windows Task Manager.

If you used CopyTicks or CopyTicksRange, the data will be cached (so in memory), but I guess that if the computer memory is not sufficient only a part will be cached (this would need to be checked). The cache is then cleared after 30 minutes.

If you downloaded from the Symbols window, then it's not in memory at all.

 
Alain Verleyen #:

Slowing down ? It's hard to know what you are talking about exactly.

You can check the memory used by MT5 with Windows Task Manager.

If you used CopyTicks or CopyTicksRange, the data will be cached (so in memory), but I guess that if the computer memory is not sufficient only a part will be cached (this would need to be checked). The cache is then cleared after 30 minutes.

If you downloaded from the Symbols window, then it's not in memory at all.

these files are the cache ?


 
Lorentzos Roussos #:

these files are the cache ?


That's the ticks data files, yes.

So when you use CopyTicks() and there is no data they will be download. If the data files are present they will be loaded in memory. That's what I am calling the cached data. They will remain in memory for 30 minutes (unless you used them of course).

I just tried to download a lot of ticks at once with CopyTIcksRange() and I got an error 4004 (as documented).

ERR_NOT_ENOUGH_MEMORY

4004

Not enough memory to perform the system function
 
Alain Verleyen #:

That's the ticks data files, yes.

So when you use CopyTicks() and there is no data they will be download. If the data files are present they will be loaded in memory. That's what I am calling the cached data. They will remain in memory for 30 minutes (unless you used them of course).

I just tried to download a lot of ticks at once with CopyTIcksRange() and I got an error 4004 (as documented).

ERR_NOT_ENOUGH_MEMORY

4004

Not enough memory to perform the system function

Yeah , i noticed it keeps stacking them up in the array until it hits the memory limit , so i assume there will be gaps in the array , logically , since it does not crash.


#property version   "1.00"

int OnInit()
  {
//---
  EventSetMillisecondTimer(44); 
//---
   return(INIT_SUCCEEDED);
  }
void OnTimer(){
  EventKillTimer();
  int attempts=0;
  int ping=-1;
  datetime cursor=flatten(TimeTradeServer());
  long cursorMSC=((long)cursor)*1000;
  long jump=2592000000;//60*60*24*30*1000;
  MqlTick receiver[];
  long oldest=LONG_MAX;
  Comment("PleaseWait");
  while(attempts<5){
       ping=CopyTicks(_Symbol,receiver,COPY_TICKS_ALL,cursorMSC,1);
       if(ping==1){
         if(receiver[0].time_msc==oldest){attempts++;}else{attempts=0;}
         if(receiver[0].time_msc<oldest){oldest=receiver[0].time_msc;}
         cursorMSC-=jump;
         }
       else{
         attempts++;
         }
       Sleep(44);
       Comment("Oldest Tick : "+TimeToString((datetime)(oldest/1000),TIME_DATE|TIME_MINUTES|TIME_SECONDS)+"\nCursor("+TimeToString((datetime)(cursorMSC/1000),TIME_DATE|TIME_MINUTES|TIME_SECONDS)+")\nAttempts("+IntegerToString(attempts)+")\nPlease wait for response...");
       }
  //we have the oldest tick here 
  //request all ticks from the oldest 
    if(oldest!=LONG_MAX){
    Print("Requesting all ticks since "+TimeToString((datetime)(oldest/1000),TIME_DATE|TIME_MINUTES|TIME_SECONDS));
    int all_ticks=CopyTicks(_Symbol,receiver,COPY_TICKS_ALL,oldest,-1);//1000000);//-1);
    Print("All ticks downloaded "+IntegerToString(all_ticks));
    Print("Oldest "+TimeToString(receiver[0].time,TIME_DATE|TIME_MINUTES|TIME_SECONDS));
    Print("Newest "+TimeToString(receiver[all_ticks-1].time,TIME_DATE|TIME_MINUTES|TIME_SECONDS));

    }else{
    Print("Cannot find oldest tick");
    }
  Print("DONE");
  ExpertRemove();
  }
datetime flatten(datetime _time){
MqlDateTime mqt;
if(TimeToStruct(_time,mqt)){
  mqt.day=1;
  mqt.hour=0;
  mqt.min=0;
  mqt.sec=0;
  _time=StructToTime(mqt);
  }
return(_time);
}

void OnDeinit(const int reason){}

void OnTick(){}

//BASIC TICKS PACKER
 
Lorentzos Roussos #:

Yeah , i noticed it keeps stacking them up in the array until it hits the memory limit , so i assume there will be gaps in the array , logically , since it does not crash.


Gaps ? It should not have gaps. Unless you make it on purpose.

By the way, just for information the MqlTick structure size is 60, not 52.

 
Alain Verleyen #:

Gaps ? It should not have gaps. Unless you make it on purpose.

By the way, just for information the MqlTick structure size is 60, not 52.

is there an extra 8 byte member in it not documented? 

Ow datetime is 8 bytes ... damn 
 
Lorentzos Roussos #:

Yeah , i noticed it keeps stacking them up in the array until it hits the memory limit , so i assume there will be gaps in the array , logically , since it does not crash.


You are copying ticks one by one ?
 
Alain Verleyen #:
You are copying ticks one by one ?

It makes a "poke" pass at first going back month by month requesting just one tick.

Then it finds the earliest tick and requests everything . 

12
New comment