Can a buffer update in the middle of two commands? - page 2

New comment
 
William Roeder #:

Wrong. When a new tick arrives, the event is added to the queue, if it isn't already there. When, and only when, the current event function (OnTick, OnCalculate, OnChart, etc.) returns, will the new event handler be started. This is why you can miss ticks.

 With a sleep of 10s inside the ontick, it waits ? It's stupid. Thread support should definitively be added 
 
Jeepack #: Thanks for your input!

I suggest the following. Don't use sleep! Since it is multi-symbol, instead use the OnTimer() event handler in combination with OnTick() for the most active symbol (e.g. EURUSD), in order to detect time synchronisation of bar openings.

Also, there is no need to wait for a symbol's bar to close. If the bar's close time is up, then it is almost certain that it will close at the most recently received data later on, even if no new opening tick has arrived for that symbol. A way to handle this is to check if the current bar's ( open time + bar duration ) has lapsed. Based on this you will know if the current bar is pending closure or if it is a freshly created new bar.

When a new bar is supposed to have happened, you will check whether a new bar has actually been formed (and thus use the previous bar for the decision) or if the current bar is still pending a new opening tick (then you will use the current bar data as if it were already closed and in the previous bars data).

I hope my convoluted explanation made sense to you.

 
Icham Aidibe #:
 With a sleep of 10s inside the ontick, it waits ? It's stupid. Thread support should definitively be added 

Not sure that would be a good thing since at a certain point you need your computer to be able to keep up with what's going on in real time. Be careful with this as this is one of the reasons it's important to understand that the tester doesn't work like a live environment. The tester processes all ticks it receives with the OnTick function, not a live EA.

Fernando Carreiro #:

I suggest the following. Don't use sleep! Since it is multi-symbol, instead use the OnTimer() event handler in combination with OnTick() for the most active symbol (e.g. EURUSD), in order to detect time synchronisation of bar openings.

Also, there is no need to wait for a symbol's bar to close. If the bar's close time is up, then it is almost certain that it will close at the most recently received data later on, even if no new opening tick has arrived for that symbol. A way to handle this is to check if the current bar's ( open time + bar duration ) has lapsed. Based on this you will know if the current bar is pending closure or if it is a freshly created new bar.

When a new bar is supposed to have happened, you will check whether a new bar has actually been formed (and thus use the previous bar for the decision) or if the current bar is still pending a new opening tick (then you will use the current bar data as if it were already closed and in the previous bars data).

I hope my convoluted explanation made sense to you.

I'm trying to figure this out... I don't use sleep, no worries! I'm curious about what you mean by "use the OnTimer() event handler in combination with OnTick() for the most active symbol (e.g. EURUSD), in order to detect time synchronisation of bar openings", maybe you're thinking of a way to improve my solution idea but I'm not sure I understand.

Your explanation makes sense to me by the way, it's exactly the idea I was recently testing until I realized it presented a huge risk if a buffer could update in the middle of two commands. This is what prompted me to code the previous experiments and write this post. So do you think we can eliminate this risk with some of the ideas mentioned in this post? That's actually what I want to do because I think it makes more sense since it does a way better job at synchronizing events accross multiple symbols (when the server clock reaches 0 seconds, you can get the previous close for all symbols at the same time).

I basically check if the current chart's open time is different than the last on each processed tick. If it is different, I look at all symbols and copy their buffers with shift = i where i is equal to 1 + iBarShift(symbol,TF,new_bar_open_time_on_current_chart,true) which is 1 + 0 = 1 if the symbol formed a new bar or 1 + -1 = 0 if the new bar isn't formed yet. So this tells me if I need to use index 1 or 0 for each symbol like you said, depending on if a new bar has formed on it after the pending close or not.

But then I run into the issue of new bars forming while the code is running, and consecutive buffer queries using identical indexes returning different results because the buffer values can shift during the process before the EA is done with all the queries.

So here's what I'm thinking now:

1) maybe I'm missing something, perhaps when you talk about using OnTimer with OnTick you might be offering me the solution and I haven't understood yet.

2) I'm still trying to figure out a solution since you also believe that using pending closes is still a valid proposition which leads me to think I should keep pursuing a solution to the risk it poses. This is what I'm considering right now: these events happen over milliseconds, so why not use the same code I just created (see previous reply) for OnChartEvents to validate synchronization of bar closes when copying multiple buffers from multiple symbols? I guess it could apply to that as well and perhaps completely fix the issue. Let me modify the code real quick and see what you think if you can check it out! If you could take a look and tell me what you think I would really appreciate it. 

(the following code can't be compiled; anyone is free to use it, but you still need to create your array of symbols and prepare all your indicator buffers in OnInit before the code is complete) ...and I'm not sure this works yet, I just wrote it to get Fernando's input on the idea, maybe there's a better way to do it...

**edit: removed my first draft because it was not sufficiently polished, see my next reply for a new version of the function**
***I realize the 2nd and 3rd for loop can probably be condensed into a single for loop but I'm just a bit tired after writing all this so please don't be too critical ;)
 
Jeepack #:

Not sure that would be a good thing since at a certain point you need your computer to be able to keep up with what's going on in real time. Be careful with this as this is one of the reasons it's important to understand that the tester doesn't work like a live environment. The tester processes all ticks it receives with the OnTick function, not a live EA.

I'm trying to figure this out... I don't use sleep, no worries! I'm curious about what you mean by "use the OnTimer() event handler in combination with OnTick() for the most active symbol (e.g. EURUSD), in order to detect time synchronisation of bar openings", maybe you're thinking of a way to improve my solution idea but I'm not sure I understand.

Your explanation makes sense to me by the way, it's exactly the idea I was recently testing until I realized it presented a huge risk if a buffer could update in the middle of two commands. This is what prompted me to code the previous experiments and write this post. So do you think we can eliminate this risk with some of the ideas mentioned in this post? That's actually what I want to do because I think it makes more sense since it does a way better job at synchronizing events accross multiple symbols (when the server clock reaches 0 seconds, you can get the previous close for all symbols at the same time).

I basically check if the current chart's open time is different than the last on each processed tick. If it is different, I look at all symbols and copy their buffers with shift = i where i is equal to 1 + iBarShift(symbol,TF,new_bar_open_time_on_current_chart,true) which is 1 + 0 = 1 if the symbol formed a new bar or 1 + -1 = 0 if the new bar isn't formed yet. So this tells me if I need to use index 1 or 0 for each symbol like you said, depending on if a new bar has formed on it after the pending close or not.

But then I run into the issue of new bars forming while the code is running, and consecutive buffer queries using identical indexes returning different results because the buffer values can shift during the process before the EA is done with all the queries.

So here's what I'm thinking now:

1) maybe I'm missing something, perhaps when you talk about using OnTimer with OnTick you might be offering me the solution and I haven't understood yet.

2) I'm still trying to figure out a solution since you also believe that using pending closes is still a valid proposition which leads me to think I should keep pursuing a solution to the risk it poses. This is what I'm considering right now: these events happen over milliseconds, so why not use the same code I just created (see previous reply) for OnChartEvents to validate synchronization of bar closes when copying multiple buffers from multiple symbols? I guess it could apply to that as well and perhaps completely fix the issue. Let me modify the code real quick and see what you think if you can check it out! If you could take a look and tell me what you think I would really appreciate it. 

(the following code can't be compiled; anyone is free to use it, but you still need to create your array of symbols and prepare all your indicator buffers in OnInit before the code is complete) ...and I'm not sure this works yet, I just wrote it to get Fernando's input on the idea, maybe there's a better way to do it...

***I realize the 2nd and 3rd for loop can probably be condensed into a single for loop but I'm just a bit tired after writing all this so please don't be too critical ;)

It should not be a blocking thread. You start a computation to decide if you'll buy or sell, and when you have your answer you decide to buy or sell but based on an old calculation, for old prices at a new price. 

Threading support should definitively be added and delegated to the coder. 

Btw, I never took the time to check it... for me it was obviously non blocking

 
You can build a multi threaded environment by using services.

You could build a whole "system" of services to create your own data source.

What I am thinking of would be maybe considered a little overkill in my eyes, but it would be about as follows.

Have a service for every symbol feed the data into a database and let them take measure to make th data be synchronous.

In your EA, you only read out the database and get the ready made datasets from there.

This way you have an environment which "freezes" your time, since you are in control of the forming of datasets. And it is multithreaded.

The whole approach will not be very upfront with the markets changes, due to slow symbols, but you would get a system that serves you only complete datasets.


 
Dominik Christian Egert #:
You can build a multi threaded environment by using services.

You could build a whole "system" of services to create your own data source.

What I am thinking of would be maybe considered a little overkill in my eyes, but it would be about as follows.

Have a service for every symbol feed the data into a database and let them take measure to make th data be synchronous.

In your EA, you only read out the database and get the ready made datasets from there.

This way you have an environment which "freezes" your time, since you are in control of the forming of datasets. And it is multithreaded.

The whole approach will not be very upfront with the markets changes, due to slow symbols, but you would get a system that serves you only complete datasets.


That's relevant, a service <-> a thread ...

edit : ... but untestable


https://www.mql5.com/en/forum/340270

How do you create a function asynchronous without to do call OnTimer?
How do you create a function asynchronous without to do call OnTimer?
  • 2020.05.11
  • www.mql5.com
I would like to create an asynchronous function which will be used in my library. But I don't want to use the OnTimer function...
 
Dominik Christian Egert #:
You can build a multi threaded environment by using services.

You could build a whole "system" of services to create your own data source.

What I am thinking of would be maybe considered a little overkill in my eyes, but it would be about as follows.

Have a service for every symbol feed the data into a database and let them take measure to make th data be synchronous.

In your EA, you only read out the database and get the ready made datasets from there.

This way you have an environment which "freezes" your time, since you are in control of the forming of datasets. And it is multithreaded.

The whole approach will not be very upfront with the markets changes, due to slow symbols, but you would get a system that serves you only complete datasets.


I think I figured it out though with basic code. It's not the same as a real multi thread environment and it may use a lot of CPU for 1-2 seconds but it still fixes my problem. I've been working overtime today trying to polish this idea so I may have missed something, still needs more testing but for now, I have something that compiles and seems to synchronize all my buffers with a few extra lines of code. There's a lot more code involved for it to be able to compile and use but here's the function that takes care of synchronizing the buffer queries with an OnTick event... hope I didn't make any mistakes:

/*
...
*/

void Copy_synchronized_buffers_values() {
        Number_of_failed_synchronized_buffer_copy_attempts = 0; // for testing purposes
        for (int s = 0; s < Number_of_tradeable_symbols; s++) {
                for (int t = 0; t < Number_of_tradeable_timeframes; t++) {
                        int handle_index = s * Number_of_tradeable_timeframes + t;
                        last_close_buffer_index_before_copy[t] = 1 + iBarShift(Tradeable_symbols[s],Tradeable_timeframes[t],last_open_time_on_smallest_timeframe,true);
                        CopyBuffer(indicator_1_buffer_handles[handle_index],0,last_close_buffer_index_before_copy[t],1,indicator_1_values_array);
                        CopyBuffer(indicator_2_buffer_handles[handle_index],0,last_close_buffer_index_before_copy[t],1,indicator_2_values_array);          
                }
                if (last_close_buffer_index_before_copy[0] != 1 + iBarShift(Tradeable_symbols[s],Tradeable_timeframes[0],last_open_time_on_smallest_timeframe,true)) {
                        Number_of_failed_synchronized_buffer_copy_attempts++; // for testing purposes
                        s--;
                }
        }
        Print("Synchronized copy of ",Number_of_handles_for_each_buffer," buffers is successful after ",Number_of_failed_synchronized_buffer_copy_attempts," failed attemps."); // for testing purposes
}

void OnTick() {
	datetime current_open_time_on_smallest_timeframe = iTime(Current_chart_symbol,Smallest_timeframe,0);
	if (current_open_time_on_smallest_timeframe != last_open_time_on_smallest_timeframe) {
		last_open_time_on_smallest_timeframe = current_open_time_on_smallest_timeframe;
		Copy_synchronized_buffers_values();
	}
}

The idea, if my thinking is correct, is that what really matters is that all buffer queries for each symbol/timeframe must be aligned in time and uncompromized by new ticks opening new bars as computations are taking place in the OnTick function. When I compile the full version of my code and run it with M1 as the smallest timeframe, using 28 pairs and 19 timeframes, it fails to synchronize the queries on average perhaps 50000 times per new M1 bar open before being able to produce synchronized results on every symbol. This seems like a lot but it takes on average 1 second on every bar open so the buffers are all copied from previous closes or pending closes on average around the 1 second mark of every minute without having to wait for all symbols to get their new opens. Sometimes it takes 500k or 600k failed attempts at the most, but this still manages to take less than 5 seconds.

Statistically, the more symbols have time to form their new M1 bar opens, the smaller the chances of any synchronized buffer copy attempts failing as fewer M1 bars can open in the middle of the OnTick computation process. So what I did is I also tried it with a 2.5 second delay:

void OnTick() {
        datetime current_open_time_on_smallest_timeframe = iTime(Current_chart_symbol,Smallest_timeframe,0);
        if (current_open_time_on_smallest_timeframe != last_open_time_on_smallest_timeframe) {
                last_open_time_on_smallest_timeframe = current_open_time_on_smallest_timeframe;
                Sleep(new_bar_open_sleep_delay);
                Copy_synchronized_buffers_values();
        }
}

...this is not supposed to speed up the process but shows that you can reduce the processing power required if you're not in a rush to have orders sent as fast as possible. By playing with the timer I also see that in the first second or two, it's almost impossible to synchronize all the buffer queries, they almost always get compromized by new M1 bars opening as computations are taking place. So I don't think that using a timer event to get the function to trigger on the exact minute change would improve anything, you would probably still have to wait until the 1-2 second mark to be able to synchronize your queries. This might be different though if using only 1 or a few timeframes instead of 19 as that would make it less likely to have a bar open in the middle of two buffer query commands.

PS: I realize this is not necessarily an improvement compared to just looping through each symbol until they all receive their new M1 bar's first tick before starting new buffer queries. After all this testing, maybe it just proves that something is possible without showing a good reason to use it. That's something I'll have to give more thought to... but for now, the way I see it, it prevents all symbols from sometimes getting delayed by 10+ seconds when one of your symbols seems to momentarily freeze with your broker since this technique makes it possible to get that symbol's pending close without having to wait 10 seconds for a new open. So it mitigates the risk of having to wait for a long time if 1 or 2 symbols are slow to deliver ticks while making sure that all buffer queries are synchronized.

 

You see ... at first the topic seems idiot but ... it's extremely finical ✌

Delays between ticks vary ; we can suppose that with any hardware today, the ontick() will have time to fully execute before the new one arrives ... but it's just a supposition - ea are now complex, involving many currencies  

 
Jeepack #:  I'm curious about what you mean by "use the OnTimer() event handler in combination with OnTick() for the most active symbol (e.g. EURUSD), in order to detect time synchronisation of bar openings", maybe you're thinking of a way to improve my solution idea but I'm not sure I understand.

The OnTick() event handler gets called on every new tick for the current symbol only, but what will you do if there are no new ticks for the current symbol, but there have been ticks for other symbols you are trading with the EA?

By using the OnTimer() in combination with OnTick(), set at a regular interval, you still get to query the current state of other symbols, even if the current symbol is inactive. Consider it a "heartbeat" function or protocol.

By using EURUSD as your primary chart, you will have plenty of tick activity since its one of the most active symbols, but it also has lull periods compared to other symbols, or not trade at all compared to some other symbols. What if you are also trading BitCoin on your EA which trades during the weekend, but EURUSD does not?

By using the OnTimer() as well, your EA will continue to query all the other symbols and act accordingly even if the primary chart is on pause.

 

btw there was a typo so I just corrected it... this:

                        CopyBuffer(indicator_1_buffer_handles[handle_index],0,last_close_buffer_index_before_copy,1,indicator_1_values_array);
                        CopyBuffer(indicator_1_buffer_handles[handle_index],0,last_close_buffer_index_before_copy,1,indicator_2_values_array);    

Was supposed to be:

                        CopyBuffer(indicator_1_buffer_handles[handle_index],0,last_close_buffer_index_before_copy,1,indicator_1_values_array);
                        CopyBuffer(indicator_2_buffer_handles[handle_index],0,last_close_buffer_index_before_copy,1,indicator_2_values_array);
Icham Aidibe #:

You see ... at first the topic seems idiot but ... it's extremely finical ✌

Delays between ticks vary ; we can suppose that with any hardware today, the ontick() will have time to fully execute before the new one arrives ... but it's just a supposition - ea are now complex, involving many currencies  

I'm glad you see it that way! I would say that even if trading only one symbol and one timeframe, this can be an issue. Like Fernando was saying though, this won't affect most EA's if they always control for a new bar open before copying buffers. But then this still leaves them open, like he clarifies in his last reply, to periods where trading is dull, where ticks aren't delivered quickly, and then you're EA starts being slow to react to pending closes. I think this post addresses this in a way, even though it seems idiotic at first haha ;)

Fernando Carreiro #:

The OnTick() event handler gets called on every new tick for the current symbol only, but what will you do if there are no new ticks for the current symbol, but there have been ticks for other symbols you are trading with the EA?

By using the OnTimer() in combination with OnTick(), set at a regular interval, you still get to query the current state of other symbols, even if the current symbol is inactive. Consider it a "heartbeat" function or protocol.

By using EURUSD as your primary chart, you will have plenty of tick activity since its one of the most active symbols, but it also has lull periods compared to other symbols, or not trade at all compared to some other symbols. What if you are also trading BitCoin on your EA which trades during the weekend, but EURUSD does not?

By using the OnTimer() as well, your EA will continue to query all the other symbols and act accordingly even if the primary chart is on pause.

That makes perfect sense and I'm glad you clarified it. I haven't figured out the details on how to implement it in the most effective way yet but I'll surely give it some thought. Things I'm wondering about is where would you want the timer event to be triggered, would it be on every new minute at 0 seconds if using M1 as the smallest timeframe? Probably. But then I'm wondering if that means OnTick becomes useless? I was thinking of using a timer instead of OnTick but I'm still wondering how they can both be used together and complement each other. Maybe the reason is to have the OnTimer event stop when the market is closed? I'm still trying to figure this out but I understand what you mean. It's true, EURUSD can sometimes go into lull periods during the day so a timer could help with that. Thanks!

1234
New comment
Reason: