What is it, that soaks up all my RAM? - Strategy Tester - MQL4 and MetaTrader 4

rv 2009.09.02 20:47 #1

schnappi wrote >>

Hi,

I'm currently working on an EA which is causing me some headaches... In the strategy tester the EA soaks up all my RAM, something I've never experienced before - and I'm working with MT for quite a while now.

This screenshot was taken while running a backtest. Terminal.exe soaks up alle the RAM it can get. Win starts to increase virtual RAM.

If the time span of the backtest is too long, the terminal even crashes because there's no more memory. This only takes around 15 minutes.

Normally, the lower line (virtual RAM) is a constant - even when I'm running backtests, that run for more than 24 hours.

The CPU usage looks pretty normal.

The funny thing is, somehow I can isolate the problem. It's about a new feature I've added. But I wasn't yet able to define what exactly makes the terminal need so much RAM!

The EA does not include any rocket science stuff, no neural net, no interfaces, dlls or anything. The new feature (which causes this problem) is packed into 2 functions and just contains some simple arithmetics. I just can't figure out, what's so special about it...

Have you made any experiences like that? Can you give me a hint?

Is there anything particular, that needs a lot of RAM?

endless loop somewhere in your code ?!

Pilot65 2009.09.03 06:15 #2

schnappi:

Hi,

I'm currently working on an EA which is causing me some headaches... In the strategy tester the EA soaks up all my RAM, something I've never experienced before - and I'm working with MT for quite a while now.

This screenshot was taken while running a backtest. Terminal.exe soaks up alle the RAM it can get. Win starts to increase virtual RAM.

If the time span of the backtest is too long, the terminal even crashes because there's no more memory. This only takes around 15 minutes.

Normally, the lower line (virtual RAM) is a constant - even when I'm running backtests, that run for more than 24 hours.

The CPU usage looks pretty normal.

The funny thing is, somehow I can isolate the problem. It's about a new feature I've added. But I wasn't yet able to define what exactly makes the terminal need so much RAM!

The EA does not include any rocket science stuff, no neural net, no interfaces, dlls or anything. The new feature (which causes this problem) is packed into 2 functions and just contains some simple arithmetics. I just can't figure out, what's so special about it...

Have you made any experiences like that? Can you give me a hint?

Is there anything particular, that needs a lot of RAM?

Yes, check carefully that you release all unused handles/files and objects.

Quick check: right-click on chart and look at the list of objects. Sometimes you may see just one of your object on the chart, but they can be stacked 1000s high !

Is this a bug Changing the time on how to draw objects

schnappi 2009.09.03 08:22 #3

No, no endless loops, no objects, handles...

The funny thing is: the code that causes the problems is disabled! The code is never executed!

The problem occurs just by having the code included.

Yesterday night I've spent 2 hours trying to isolate the specific code piece. I'm getting nearer...

Custom indicator in tester I need HEEEELP, please, OnTick() Seems to not

rv 2009.09.03 08:30 #4

schnappi wrote >>

No, no endless loops, no objects, handles...

The funny thing is: the code that causes the problems is disabled! The code is never executed!

The problem occurs just by having the code included.

Yesterday night I've spent 2 hours trying to isolate the specific code piece. I'm getting nearer...

in this case i guess you need to post your code for others can find something...

schnappi 2009.09.03 10:50 #5

I've isolated the piece of code, but I still haven't found the actual reason or a solution.

It's all about a single function-call:

ProcessOrderErrors(GetLastError());

The function is placed in a separate library and looks like this:

void ProcessOrderErrors (int Error) {
string fname="ProcessOrderErrors()";
Log(3, fname, "Error occurred: #"+Error+" "+getErrorDescription(Error));
switch(Error)
{
// Not crucial errors
case 4:
{
Log(3, fname, "Trade server is busy. Trying once again..");
Sleep(3000); // Simple solution
return;
}
case 135:
{
Log(3, fname, "Price changed. Trying once again..");
RefreshRates(); // Refresh rates
return;
}
case 136:
{
Log(3, fname,"No prices. Waiting for a new tick..");
while(RefreshRates()==false) // Till a new tick
Sleep(1); // Pause in the loop
return;
}
case 137:
{
Log(3, fname,"Broker is busy. Trying once again..");
Sleep(3000); // Simple solution
return;
}
case 146:
{
Log(3, fname,"Trading subsystem is busy. Trying once again..");
Sleep(500); // Simple solution
return;
}

// Critical errors
case 2: Log(3, fname,"Common error.");
return; // no simple solution, exit the function
case 5:
{
Log(3, fname,"terminal version too old!");
eaWork=false; // Terminate program
return; // Exit the function
}
case 64:
{
Log(3, fname,"Account blocked.");
eaWork=false; // Terminate operation
return; // Exit the function
}
case 133:Log(3, fname,"Trading forbidden.");
return; // Exit the function
case 134:
{
Log(3, fname,"Not enough money to execute operation. Free margin="+AccountFreeMargin()+" Min. needed margin="+MarketInfo(Symbol(),MODE_MARGINREQUIRED));
if (getTotalOrders(appID)>1) Log(3, fname,"There are still open orders -- NO EXIT");
else {Log(3, fname,"No open orders --> EXIT"); eaWork=false; }
return; // Exit the function
}
default: return; // Exit the function

return; // Exit the function
}
}

This function is probably the one I use most in my EAs. I've counted the function calls in this EA.

If I call it 25 times - no problems.

If I call it 26 times - terminal soaks up the RAM - allthough the 26th function call is never executed! It's just, that it's in the code.

Isn't this weird?

So is there a maximum number of allowed function calls or what???

Why is there NO Cannot compile as variable Error messages - in

schnappi 2009.09.03 12:01 #6

Finally I've found a solution - but I still don't understand what caused the problem.

It wasn't the function, the number of calls or anything: I did not change a single line in the code - just placed the functions in a library. So far, the two functions were placed in the main-mq4-file.

This means the compiler makes some difference between functions placed in the main file and functions placed in libraries - just in case this might help someone else...

Now the RAM usage is absolutely constant.

[ARCHIVE!] Any rookie question, Questions from Beginners MQL5 help me....error ambiguous call

[Deleted] 2009.09.03 12:38 #7

schnappi wrote >>

I've isolated the piece of code, but I still haven't found the actual reason or a solution.

It's all about a single function-call:

ProcessOrderErrors(GetLastError());

The function is placed in a separate library and looks like this:

void ProcessOrderErrors (int Error) {
string fname="ProcessOrderErrors()";
Log(3, fname, "Error occurred: #"+Error+" "+getErrorDescription(Error));
switch(Error)
{
// Not crucial errors
case 4:
{
Log(3, fname, "Trade server is busy. Trying once again..");
Sleep(3000); // Simple solution
return;
}
...

...

case 134:
{
Log(3, fname,"Not enough money to execute operation. Free margin="+AccountFreeMargin()+" Min. needed margin="+MarketInfo(Symbol(),MODE_MARGINREQUIRED));
if (getTotalOrders(appID)>1) Log(3, fname,"There are still open orders -- NO EXIT");
else {Log(3, fname,"No open orders --> EXIT"); eaWork=false; }
return; // Exit the function
}
default: return; // Exit the function

return; // Exit the function
}
}

So is there a maximum number of allowed function calls or what???

assume by Library you mean .ex4 and if so, does your terminal experts tab show that is loading the file >1 time? just a thot...

schnappi - I am curious/always interested to see how others code so what is first formal param in function Log(3,...) ? could it be priority or code to add eg, header.3 ?

Why is there NO Cannot compile as variable Error messages - in

schnappi 2009.09.03 14:48 #8

Hi fbj,

libraries for me are mqh-files which I place into the \include subdirectory.

Speaking of my logging: I've found it useful to structure my logging. I'm using loglevels. I never just use Print().

0 = DEBUG

1 = INFO

2 = WARNING

3 = ERROR

4 = CRITICAL

When I'm running my EA, I can switch the loglevel to increase/decrease the logging output. Debug for example is the highest level and results in a huge logfile.
When I'm running backtests, I sometimes switch it to 3 only.

When I'm running an EA on a real-money account, I always use Debug.

This is my logging-function:

void Log(int level, string src, string txt)
{
if (level>=loglevel) {
if (level==0) Print("DEBUG "+src+": "+txt);
if (level==1) Print("INFO "+src+": "+txt);
if (level==2) Print("WARNING "+src+": "+txt);
if (level==3) Print("ERROR "+src+": "+txt);
if (level==4) {Print("!CRITICAL ERROR! "+txt); SendMail(getProdKons()+": Critical Error",Symbol()+" "+Period()+" "+src+": "+txt);}
}
return;
}

Ah, and I also found it useful to know, which function is currently logging. So I always tell the source...

Makes life easier :)

Question: WORKING WITH LIBRARIES Writing when I want Please help: MT4 FileOpen

[Deleted] 2009.09.03 16:21 #9

Thank you very much for your detailed reply schnappi - is appreciated and interesting.

.

Well, since your function was .mqh type, is indeed a strange 26th event!

Wonder what that event has unique to it that previous ones do not?

.

on closer inspection the following ideas hit me:
(1)

  case 136:
  {
    Log(3, fname,"No prices. Waiting for a new tick..");
    while(RefreshRates()==false) // Till a new tick
    Sleep(1); // Pause in the loop 
    return;
  }

Is possible that incoming ticks have time gaps, yes?
- could this have potential side effect?
- calling RefreshRates() in a hard loop each millisecond. Could this have issue? eg, is 1ms long enough?
or could this cause intensive buffering within interpreter and cause ramp up of storage requirements...
.
(2)
Could the log shed light regarding the last caller to ProcessOrderErrors()?
What was the error it met? This would map to a case nn: block, yes? maybe this block is the issue...
.
(3)
Help me here if you will:
I see that this is the logging part of the error discovery journey...
I am trying to say: since the caller 'sees' called builtin error code and then calls ProcessOrderErrors(),
how do all these 25+ callers deal with reality of "// Critical errors". Eg, do they each have detection of error codes
associated with their specific line of work and following the logging call, they act according to the error code?
.
(4)
Additionally, I do not know of course, but is there calling code that effectively does a while(1) block if the error code
is not serious. Ie, on ProcessOrderErrors() return, it keeps retrying...
.
Again, thanks for your detailed reply!

iCustom not working Loops and Closing or Need some help with

schnappi 2009.09.03 22:30 #10

Hi fbj,

actually I don't want to spend too much time to get too deep into details with this...

For 2 reasons:

1. understanding each and every detail of the compiler won't make my systems more profitable (and that's what counts for me)

2. i think, they will release mt5 in a while. then we all have to spend months.. for what? to get to the point where we all are standing at the moment. you know what i mean? we'll have to rewrite EVERYTHING. and there will be a new compiler, containing new bugs...

take care

schnappi

Please help. Newbie programmer. What would it take Open Order using amount