Help: Using Multiple Timeframes and MT4 freezing and load history issues - page 2

New comment
 

I have encountered a similar problem with ArrayCopyRates: it hangs the entire terminal. I put an EA which does calls to ArrayCopyRates at five time frames onto a chart and everything is fine. But if I put it on three charts simultaneously then sometimes the terminal hangs, and when I put it on five charts simultaneously, it hangs almost every time. It seems that ArrayCopyRates is not re-entrant or thread-safe. On a single chart the functions calls are sequential, but when the EA is on multiple charts there is the possibility of it being entered by multiple charts at once.

I have implemented a mutex to prevent simultaneous access to ArrayCopyRates and it appears to have fixed the problem.


Edit: Okay, upon further testing it has improved the situation but not vanquished the problem entirely. Odd thing is it worked reliably when used on only one or two charts. But when stress tested on many charts it vaporizes itself. If I solve it I'll post a follow-up here.

 

I have resolved this problem now. It is caused by a bug in MetaTrader's implementation of ArrayCopyRates. ArrayCopyRates is not re-entrant or thread-safe. Translation from geek-speak: Each EA runs as a separate thread, its stream of instructions execute by themselves outside of and parallel to every other thread. It means that ArrayCopyRates can be interrupted before it has finished, by another EA running in another chart. ArrayCopyRates is not designed to permit this (a flaw in the design), so when it occurs the interruption corrupts some of the internal data structures and the whole system crashes.

I know of only two workarounds, but neither one is elegant. Both workarounds require avoiding the use of ArrayCopyRates in an EA.

The first workaround is to split the EA into two parts, a client and a server. The server part is implemented as an indicator, not as an EA. This is the approach I have taken, and after several months of use it has never crashed. In my system the heavy lifting is done in a DLL. A server (indicator) does the ArrayCopyRates operation and calls functions in the DLL to transfer data to it. But computation needs to be kept to a minimum in these functions. The EA part calls functions in the DLL that do the actual computation and issue the trade signals. The EA then executes the trades and manages them.

This arrangement works because all indicators (across all charts) run in a single thread. That means that every call to ArrayCopyRates must complete before another call can be started, so no problems with interruptions can occur.

The other workaround is to avoid referring to any other TF than the chart TF by doing your own bar compression. In this approach, the chart would have to be at the lowest TF used, and the DLL or EA would have to compress those bars to higher TFs on its own. This solution isn't always workable, depending on how much low TF data you have and how many high TF bars you need. If you only have 2 weeks of M1 data, it will only compress to 10 bars of D1 data, hardly enough to get any sort of overview or extract meaningful patterns from it.

I haven't tested this issue on MT 5, since my broker of choice uses MT 4, but even though there is no explicit ArrayCopyRates function in MQL5, the equivalent operations still have to be done internally. The developers may have reused the code from ArrayCopyRates rather than rewriting it all from scratch, and if so the same sort of problem might show up.

 

I'm not surprised that ACR could be thread unsafe. I am surprised that it wasn't solved with a mutex. Post your mutex code or check out mine.

Are you calling ACR once during init, or on every tick?

 
WHRoeder:

I'm not surprised that ACR could be thread unsafe. I am surprised that it wasn't solved with a mutex. Post your mutex code or check out mine.

Are you calling ACR once during init, or on every tick?


I call ACR once during init, then if a 4066 error is returned a timer is started with a short delay and it is called again, until it succeeds or a final timeout occurs. I know of no liability to calling ACR more than once, other than the side effect that the address pointer that it returns changes each time it is called. Typically what happens is on subsequent calls ACR will return a success status even though the data has not been updated. This throws my DLL into conniptions, because my DLL cannot tolerate missing bars in the data (it copies all the bars it needs, using interpolation to fill gaps). At startup the data needs to be completely up to date before the copy is started. I banged my head on this one for months and haven't found any clean way to ensure that the data update is complete before starting the copy. A simple time delay would do it of course, but when a lot of charts are running at once and much data needs updating the delay can be several minutes long. Eventually I stuck a big "Start" button on a dialog box with instructions to wait until data update is complete and then press the button. That works, but it gets messy when no data feed is running like on weekends, as there aren't any ticks to trigger re-entry into the functions to complete initialization and start processing. I have to manually initialize the server twice. Anyway, that's straying off topic.

Here is the mutex code I wrote. It should be bulletproof, for example I drop into assembly language and use XCHG, which is an atomic (uninterruptable) instruction. It locks the flag and then checks whether it was already locked, in one uninterruptable operation. If it was already locked then the state of the flag wasn't changed, so it simply exits without granting the lock, otherwise it grants the lock. It also contains a timeout feature to prevent permanent deadlocks. Do you see any flaw in this?

function GetACRLock: integer; stdcall;
  // this is a non-waiting mutex. It returns 1 if the lock is granted and
  // 0 if it is not granted.  It will eventually time out if not released.
  label ISUNLOCKED;
  const
    TIMEOUT = 1.0/1440.0; // one minute
  var
    retval      : integer;
    currentTime : TDateTime;

  begin
    currentTime := Now;

    asm // do XCHG, to prevent race conditions
      mov   AL, $1
      xchg  AL, ACRLock
      cmp   AL, $0
      jz    ISUNLOCKED
    end;

    // It is already locked, but release lock if it timed out
    if (timeLockGranted > 0) and ((currentTime - timeLockGranted) <= TIMEOUT) then
      retval := 0
    else
  ISUNLOCKED:
      begin // grant lock
        retval := 1;
        timeLockGranted := currentTime;
      end;
    GetACRLock := retval;
  end; // GetACRLock
//---------------------------------------------------------------------
procedure ReleaseACRLock; stdcall;
  begin
    ACRLock := 0;
  end; // ReleaseACRLock


BTW, good thread you have going to document Lazarus/Delphi DLL development for MT4. Feel free to copy this info there if you like.

 

Your mutex is in the dll. If you have two separate threads (i.e. separate EA's) then you have two DLLs and two mutex's. So that's useless.

Try the GetTradeContext in my code

 
WHRoeder:

Your mutex is in the dll. If you have two separate threads (i.e. separate EA's) then you have two DLLs and two mutex's. So that's useless.

Try the GetTradeContext in my code

The MT documentation says that a separate DLL is loaded for each instance of the EA, but in practice they share the same memory space. Writing to global variable ABC from one instance will change it for all instances running on a single MT platform, but not for instances of the DLL running on another MT on the same computer (I tested these things). Or maybe that's just Delphi, I don't know. Anyway, the mutex above does work correctly. I will take a look at yours.
 

I'm using one EA in multicurrency on multiframe, the terminal sometimes freeze too, I don't know why? The protype as following:

for (int i = 0; i < TOTALTRADESYMBOLS; i++)
{
  string symbol = AllTradeSymbols[i];
  ...
  double r_mn1_0 = iRSI(symbol, PERIOD_MN1, 3, PRICE_CLOSE, 0);
  double r_mn1_1 = iRSI(symbol, PERIOD_MN1, 3, PRICE_CLOSE, 1);

  double mh4_5_0 = iMA(symbol, PERIOD_H4, 5, 0, MODE_SMA, PRICE_CLOSE, 0);
  double mh4_5_1 = iMA(symbol, PERIOD_H4, 5, 0, MODE_SMA, PRICE_CLOSE, 1);
  double ch4_0 = iClose(symbol, PERIOD_H4, 0);
  ...
}

 If I comment all iRSI, iMA and other indicator calculations, the EA will be fine, but useless. Any solutions or idea, please. 

 
  1. Don't hijack two year old threads with your off topic (non-ACR) problem - open your own.
  2. Reduce max bars in chart
12
New comment
Reason: