Non-determinism in CopyBuffer even when no error is reported and return value is positive.

[Adam W]

This time hopefully I have checked all possible ways CopyBuffer can return something invalid. (unlike last time, when I neglected to GetLastError). Still, in rare cases, CopyBuffer sometimes writes rogue values to its destination array.

This little script, though pointless on its own, proves it.

1. Set debugger options to use specified symbol and period: EURUSD, M1.

2. Set max bars per chart at 500k.

3. Pre-load chart data for EURUSD and AUDJPY.

4. Set breakpoint at the "return 1" statement.

5. Start debugger.

//+------------------------------------------------------------------+
//|                                                      ProjectName |
//|                                      Copyright 2012, CompanyName |
//|                                       http://www.companyname.net |
//+------------------------------------------------------------------+
int OnStart()
  {
   MathSrand(123);
   int handle=iMA("AUDJPY",PERIOD_M1,50,0,MODE_EMA,PRICE_HIGH);
   if(handle==INVALID_HANDLE)
     {
      return 1;
     }
   double previous=0.0;
   double tmp[1];
   int count=0;
   while(true)
     {
     double random_starting_point = MathRand()/100.0;
      tmp[0]=random_starting_point;
      ResetLastError();
      int inds_copied=CopyBuffer(handle,0,D'2011.04.01 21:50',1,tmp);
      int e=GetLastError();
      if(!e && inds_copied==1)
        {
         if(previous!=0.0 && previous!=tmp[0])
           {
            Print("Non-determinism after "+IntegerToString(count));
            return 1;
           }
         previous=tmp[0];
        }
      count++;
     };
   return 0;
  }  // End OnStart.

I find this script stops at the breakpoint after some number of iterations. (be patient). At this point, the local variables are

e = 0

previous = 87.3505777829597 (the correct value according to the chart)

tmp[0] = 87.3517034067397 (consistently, when wrong)

random_starting_point = 203.45 (for example)

 

I've run this a few times. It always encounters the return 1 line eventually.

count = 68463501, random_starting_point = 228.85

count = 19735884, random_starting_point = 15.54

count = 37069616, random_starting_point = 53.04

count = 358285678, random_starting_point = 205.87

 

The strange thing is, I'm always using the same seed (123). So, sometimes the loop works fine through places where it previously hit the breakpoint. So, it is not the case that the value of tmp[0] depends on the value of random_starting_point after CopyBuffer has been executed without error.  Still, there is non-determinism, somehow.

 

Any ideas where the 87.3517034067397 come from?

I can get this to happen at other data points and with other symbols and indicators. This is just an example. I discovered this problem when tracking down inconsistencies in the output of a much more complex indicator. This is a simplified example of a problem that really happens.

[onewithzachy]

Hi adamuu, 

No error reported ?

int e = GetLastError();
if(!e && inds_copied == 1)

Insignificant maybe, but could you re-run the test again using this ?

int e = GetLastError();
if(e == 0 && inds_copied == 1) // -->> no error reported

https://www.mql5.com/en/docs/constants/errorswarnings/errorcodes 

Interesting though :) 

:D

[Adam W]

At the breakpoint, in the expression viewer, I can see e = 0.

But yeah, I made your modification. It stopped on count = 77893011.

Same values for everything else.

random_starting_point for the failed iteration was 103.38 this time. 

[Adam W]

Ping.

[Adam W]

Another clue. 87.3517034067397 is the value of this MA at D'2011.04.01 21:49:00'. CopyBuffer sometimes writes this value to the destination array even though this script only requests the value at D'2011.04.01 21:50:00'.

 

[Adam W]

Suddenly, I am not seeing this behavior. Not much has changed. The market is closed right now, i.e. no new data for AUDJPY is coming in. So, maybe that's it. I'll test this theory when markets re-open.

[Adam W]

The markets are now open, and as I suspected, this problem is reproducible again.

e=0, count=155809609, previous=87.350577782957, tmp[0]=87.35170340675397, inds_copied=1, random_starting_point=138.85

I think you may need one of these. https://en.wikipedia.org/wiki/Semaphore_(programming) :/ only half smiling, here.

 

[Adam W]

Ping.

[Adam W]

It's a scary bug but nothing to hide from....