Synch CopyBuffer in EA using iCustom

New comment
 

Hi,

i have a doubt about reliability of data obtained via CopyBuffer in EA. 

Considering this example. Let's assume we have a custom indicator where each tick in onCalculate sets 3 buffer:

double A[];
double B[];
double C[];
.....
//set buffers in OnInit 
.....
int onCalculate(....){
  //initialize buffer at first call etc.
  ................

  //Setting buffers
  A[0]=x;
  B[0]=y;
  C[0]=z;
}

Now let's consider getting these values in EA:

double A[1];
double B[1];
double C[1];

//getting handle in OnInit
......

void OnTick()
	if(Bars(_Symbol,_Period)!=BarsCalculated(handle)) return; //indi not calculated yet

        CopyBuffer(handle,0,0,1,A);
        CopyBuffer(handle,1,0,1,B);
        CopyBuffer(handle,2,0,1,C);

	//using A[0], B[0] and C[0]
	............
}


My question is: How i can be sure, when reading data copied using CopyBuffer, that i can get A[0],B[0],C[0] setted at the same "call" of OnCalculated? That is, it could happen that i can get A[0] setted on latest OnCalculate excution and getting B[0] and C[0] setted on latest-1 OnCalculated execution? I need to avoid getting inconsistent data from indicator.

Summary, is a single buffer thread safe? Are indicator buffers (all togheter) thread safe?

 
antony23:

Hi,

i have a doubt about reliability of data obtained via CopyBuffer in EA. 

Considering this example. Let's assume we have a custom indicator where each tick in onCalculate sets 3 buffer:

Now let's consider getting these values in EA:


My question is: How i can be sure, when reading data copied using CopyBuffer, that i can get A[0],B[0],C[0] setted at the same "call" of OnCalculated? That is, it could happen that i can get A[0] setted on latest OnCalculate excution and getting B[0] and C[0] setted on latest-1 OnCalculated execution? I need to avoid getting inconsistent data from indicator.

Summary, is a single buffer thread safe? Are indicator buffers (all togheter) thread safe?


No, they are not thread safe. Indicators are run in the chart thread, EAs have their own thread. It is very likely to receive old values from the indicators buffers.

If you need data synchronicity, you must run the calculations inside your EA. - Or, build some inter thread sync mechanism to make the EA wait for the indicator to catch up.

One possible solution could be to have a buffer that holds a serial number for the calculations the indicator has performed (An OnCalculate call counter, so to say).

Since we know OnCalculate will be called for every tick received, we can assume inside the EA, the serial number must have changed, if the indicator has been updated.

Problem is, how to wait, or recall OnTick, without blocking subsequent updates.

Basically this method slows down your EA to the onchart thread.

I suggest, do the calculations inside your EA.
 
Dominik Egert #:

No, they are not thread safe. Indicators are run in the chart thread, EAs have their own thread. It is very likely to receive old values from the indicators buffers.

I would say it's possible, but not "very likely". Anyway, if you need synchronization, the result is the same.


If you need data synchronicity, you must run the calculations inside your EA. - Or, build some inter thread sync mechanism to make the EA wait for the indicator to catch up.

One possible solution could be to have a buffer that holds a serial number for the calculations the indicator has performed (An OnCalculate call counter, so to say).

Since we know OnCalculate will be called for every tick received, we can assume inside the EA, the serial number must have changed, if the indicator has been updated.

Problem is, how to wait, or recall OnTick, without blocking subsequent updates.

Basically this method slows down your EA to the onchart thread.

I suggest, do the calculations inside your EA.
Or just add a "all-at-once" buffer with data you need to be synchronized, so it's one call to CopyBuffer().
 
Alain Verleyen #:
Or just add a "all-at-once" buffer with data you need to be synchronized, so it's one call to CopyBuffer().

Maybe do you mean putting these data on the same buffer with different indexing? 

That coudl be done, of course, but who garantee me that a single CopyBuffer call, example from index 0, count 3, would give me consistent data? Could happen that CopyBuffer copy at example index 0 an 1 values from the latest OnCalculate excution and index 2 from lastest-1 OnCalculate execution?

In fact in the first post i did two questions, the first was: "is a single buffer thread safe?"

 
Dominik Egert #:

No, they are not thread safe. Indicators are run in the chart thread, EAs have their own thread. It is very likely to receive old values from the indicators buffers.

If you need data synchronicity, you must run the calculations inside your EA. - Or, build some inter thread sync mechanism to make the EA wait for the indicator to catch up.

One possible solution could be to have a buffer that holds a serial number for the calculations the indicator has performed (An OnCalculate call counter, so to say).

Since we know OnCalculate will be called for every tick received, we can assume inside the EA, the serial number must have changed, if the indicator has been updated.

Problem is, how to wait, or recall OnTick, without blocking subsequent updates.

Basically this method slows down your EA to the onchart thread.

I suggest, do the calculations inside your EA.

The problem is not having old values inside EA, but that these values, even if old, are consistent, that is calculated in the same "OnCalculate" iteration. 

To me seems very absurd that CopyBuffer function is not thread safe realtive to arrays used to copy from and to. The same documentation advice to divide actions, that is calculated on Indicators and use data calculated on EA, but without data integrity it is impossible.

 
antony23 #:

Maybe do you mean putting these data on the same buffer with different indexing? 

Yes.

That coudl be done, of course, but who garantee me that a single CopyBuffer call, example from index 0, count 3, would give me consistent data? Could happen that CopyBuffer copy at example index 0 an 1 values from the latest OnCalculate excution and index 2 from lastest-1 OnCalculate execution?

In fact in the first post i did two questions, the first was: "is a single buffer thread safe?"

Of course a single buffer is thread safe, otherwise it would be absurd design.
 
Alain Verleyen #:
Yes. Of course a single buffer is thread safe, otherwise it would be absurd design.
Sorry, but that's not how thread safety works. Its as well not thread safe.

Two threads running in parallel, there is no way to say. CopyBuffer is only a read function, no requirement to semaphore these with a write. Maybe an atomic operation, but that's anyways given by the memory controller. But that's it.

In any way, there is no way to tell what gets executed when, without any additional information. That's why I suggested a serial number.

 
Dominik Egert #:
Sorry, but that's not how thread safety works. Its as well not thread safe.

Two threads running in parallel, there is no way to say. CopyBuffer is only a read function, no requirement to semaphore these with a write. Maybe an atomic operation, but that's anyways given by the memory controller. But that's it.

In any way, there is no way to tell what gets executed when, without any additional information. That's why I suggested a serial number.

An indicator is running in 1 thread, with the data. While the indicator is running to update data (write on the buffer), you can't read it at the same time. 

EDIT: Actually we both don't know how it's done in practice, my guess was MQ made it thread safe, but you could be right. I will ask them more information.

 
Alain Verleyen #:

An indicator is running in 1 thread, with the data. While the indicator is running to update data (write on the buffer), you can't read it at the same time. 

EDIT: Actually we both don't know how it's done in practice, my guess was MQ made it thread safe, but you could be right. I will ask them more information.

Yeah, we actually dont know.

My approach therefore was preemptive multithread.

Good idea to ask. Let's see what's coming back.
 
Alain Verleyen #:

An indicator is running in 1 thread, with the data. While the indicator is running to update data (write on the buffer), you can't read it at the same time. 

EDIT: Actually we both don't know how it's done in practice, my guess was MQ made it thread safe, but you could be right. I will ask them more information.

 The things are these:

- or copybuffer is thread safe, but that would mean that also writing data in buffers, inside Oncalculate, should be done using copybuffer and not setting single index value in a loop. And i didn't read this anywhere in documentation.

- or the deisgn is done in the way that OnCalulate and OnTicks, on the same chart, are executing sequentially, even if in its own thread. Using separate threads for EAs could be usefull to execute, instead, more OnTick events on the same symbol, from different EAs (and so from different charts) simultenously.  

I suspect the second is the most likely, otherwise this simultaneous data access problem should have been very well documented, but instead I haven't found anything about it, and I've searched really hard.


Anyway i'll wait for your news.

Thanks.

 
antony23 #:

 The things are these:

- or copybuffer is thread safe, but that would mean that also writing data in buffers, inside Oncalculate, should be done using copybuffer and not setting single index value in a loop. And i didn't read this anywhere in documentation.

- or the deisgn is done in the way that OnCalulate and OnTicks, on the same chart, are executing sequentially, even if in its own thread. Using separate threads for EAs could be usefull to execute, instead, more OnTick events on the same symbol, from different EAs (and so from different charts) simultenously.  

I suspect the second is the most likely, otherwise this simultaneous data access problem should have been very well documented, but instead I haven't found anything about it, and I've searched really hard.


Anyway i'll wait for your news.

Thanks.

Though, no fundamental docs on this, CopyBuffer is for sure not thread safe, else this would have been actually documented.

EAs and OnChart thread is not synchronized, there is no sequential processing implemented.

The lack of documentation leads me to assume, my initial assumption is correct. There is zero thread safety in this process.

You (and I am quite confident this will be the outcome) need to figure a way to ensure you get what you actually are looking for in a different way.

Probably the only real approach is to use global variables from the terminal to achieve a locking mechanism to prevent false "in the middle" processing/reading of data.

Or, transfer your calculations to the EA.
123
New comment