Bar (Time) lag in CopyRates on symbol other then current chart in open prices only mode.

Enrique Dangeroux 2022.12.09 11:40 #11

Lorentzos Roussos #:

I get that too , on H1 as well

i'll do a programer's shot in the dark here .
They have coded the ticks to come sequentially from each symbol and since open prices has one tick event it has no time to fire the rest of the symbols .

Thank you for testing. It is clear the issue is not with the code, but with the tester.

2022.12.09 11:32:45.454 2022.12.05 23:59:59   ********************
2022.12.09 11:32:45.454 2022.12.05 23:59:59   Tester TimeFrame PERIOD_H1
2022.12.09 11:32:45.454 2022.12.05 23:59:59   Tester Symbol EURUSD
2022.12.09 11:32:45.454 2022.12.05 23:59:59   Symbol A EURUSD
2022.12.09 11:32:45.454 2022.12.05 23:59:59   Symbol B GBPUSD
2022.12.09 11:32:45.454 2022.12.05 23:59:59   Minial Lag 00:00
2022.12.09 11:32:45.454 2022.12.05 23:59:59   Bars with difference 70 out of 71 bars
2022.12.09 11:32:45.454 2022.12.05 23:59:59   Average lag 99.72 Minutes
2022.12.09 11:32:49.343 2022.12.05 23:59:59   ********************

The lag is proportionally to the timeframe.

2022.12.09 11:33:30.295 2022.12.05 23:59:59   ********************
2022.12.09 11:33:30.295 2022.12.05 23:59:59   Tester TimeFrame PERIOD_M30
2022.12.09 11:33:30.295 2022.12.05 23:59:59   Tester Symbol EURUSD
2022.12.09 11:33:30.295 2022.12.05 23:59:59   Symbol A EURUSD
2022.12.09 11:33:30.295 2022.12.05 23:59:59   Symbol B GBPUSD
2022.12.09 11:33:30.295 2022.12.05 23:59:59   Minial Lag 00:00
2022.12.09 11:33:30.295 2022.12.05 23:59:59   Bars with difference 142 out of 143 bars
2022.12.09 11:33:30.295 2022.12.05 23:59:59   Average lag 70.28 Minutes
2022.12.09 11:33:30.295 2022.12.05 23:59:59   ********************

Even with M1 timeframe

2022.12.09 11:37:59.858 2022.12.05 23:59:59   ********************
2022.12.09 11:37:59.858 2022.12.05 23:59:59   Tester TimeFrame PERIOD_M1
2022.12.09 11:37:59.858 2022.12.05 23:59:59   Tester Symbol EURUSD
2022.12.09 11:37:59.858 2022.12.05 23:59:59   Symbol A EURUSD
2022.12.09 11:37:59.858 2022.12.05 23:59:59   Symbol B GBPUSD
2022.12.09 11:37:59.858 2022.12.05 23:59:59   Minial Lag 00:00
2022.12.09 11:37:59.858 2022.12.05 23:59:59   Bars with difference 4301 out of 4305 bars
2022.12.09 11:37:59.858 2022.12.05 23:59:59   Average lag 19.96 Minutes
2022.12.09 11:37:59.858 2022.12.05 23:59:59   ********************

The lag is a whopping 19 minutes. Granted the M1 is most prone to mismatched timestamp bars during slow hours.

By the way, testing on 1 minute OHLC data yields exactly the same results.

I am starting to think this is a bug. As it is now, it is useless to test/optimize any EA that accesses price data from any symbol other then the current chart in open prices only and 1 minute OHLC mode.

Is there a way When is the new Bugs MT5 backtests price

Fernando Carreiro 2022.12.09 12:34 #12

I haven't been able to test your code myself due to other obligations, but during the weekend I will also run some tests and give you a report on it.

Fernando Carreiro 2022.12.09 12:48 #13

I still have not analysed you code yet, so I don't know if the following situation has been considered or not, but here is one more thought...

Have you considered the delay between the closing and opening of bar?

For example...

... at exactly 12:00:00 a new tick arrives for EURUSD, causing the previous M1 bar to close and a new bar to form, so that the current bar's opening time is 12:00(:00).

However, no tick has arrived yet for GBPUSD, so it's current M1 bar's opening time is still 11:59(:00), meaning a lag of 1 minute.

... at exactly 12:00:09 a new tick arrives for GBPUSD, causing the previous M1 bar to close and a new bar to form, so that the current bar's opening time is 12:00(:00).

Only now are the two symbols is sync where both the current bars have the same opening time of 12:00(:00)

How do you check How does CopyTime work? MQL4 first step

Enrique Dangeroux 2022.12.09 13:17 #14

No worries Fernando.

Test with every tick based on real ticks

2022.12.09 13:07:57.189 2022.12.05 23:59:58   ********************
2022.12.09 13:07:57.189 2022.12.05 23:59:58   Tester TimeFrame PERIOD_M1
2022.12.09 13:07:57.189 2022.12.05 23:59:58   Tester Symbol EURUSD
2022.12.09 13:07:57.189 2022.12.05 23:59:58   Symbol A EURUSD
2022.12.09 13:07:57.189 2022.12.05 23:59:58   Symbol B GBPUSD
2022.12.09 13:07:57.189 2022.12.05 23:59:58   Minial Lag 00:00
2022.12.09 13:07:57.189 2022.12.05 23:59:58   Bars with difference 2041 out of 4305 bars
2022.12.09 13:07:57.189 2022.12.05 23:59:58   Average lag 19.44 Minutes
2022.12.09 13:08:00.330 2022.12.05 23:59:58   ********************

While there are less bars with a difference vs 1 Minute OHLC and open prices, the average lag is relative consistant with the other modes.

Maybe the NewBar test is part of the "lag creation" as per your suggestion/example.

Honestly i do not want to waste anyone's time. I have to retract the bug feeling. It always annoys me people blaming their incompetence to an external source like a bug.

The current state is i fail to understand.

Tester different Bar Open Backtesting/Optimization Questions from Beginners MQL5

Fernando Carreiro 2022.12.09 13:19 #15

Another possibility is the the "Open prices only" modelling is applied only to the current symbol, while for other symbols real-time modelling is applied, causing a discrepancy on the synchronisation of open times.

Enrique Dangeroux 2022.12.09 13:31 #16

Fernando Carreiro #:
Another possibility is the the "Open prices only" modelling is applied only to the current symbol, while for other symbols real-time modelling is applied, causing a discrepancy on the synchronisation of open times.

2022.12.09 13:23:51.724 EURUSD: symbol to be synchronized
2022.12.09 13:23:51.724 EURUSD: symbol synchronized, 3720 bytes of symbol info received
2022.12.09 13:23:51.725 EURUSD: history synchronization started
2022.12.09 13:23:51.727 EURUSD: load 27 bytes of history data to synchronize in 0:00:00.001
2022.12.09 13:23:51.727 EURUSD: history synchronized from 2009.01.02 to 2022.12.06
2022.12.09 13:23:51.895 EURUSD,M1: history cache allocated for 719037 bars and contains 713894 bars from 2021.01.01 00:47 to 2022.11.30 23:59
2022.12.09 13:23:51.895 EURUSD,M1: history begins from 2021.01.01 00:47
2022.12.09 13:23:51.899 EURUSD,M1 (****-Server): OHLC bar states generating. OnTick executed on the bar begin only
2022.12.09 13:23:51.934 GBPUSD: symbol to be synchronized
2022.12.09 13:23:51.935 GBPUSD: symbol synchronized, 3720 bytes of symbol info received
2022.12.09 13:23:51.935 GBPUSD: history synchronization started
2022.12.09 13:23:51.937 GBPUSD: load 27 bytes of history data to synchronize in 0:00:00.001
2022.12.09 13:23:51.937 GBPUSD: history synchronized from 2009.01.02 to 2022.12.06
2022.12.09 13:23:52.108 GBPUSD,M1: history cache allocated for 716649 bars and contains 711506 bars from 2021.01.01 00:01 to 2022.11.30 23:59
2022.12.09 13:23:52.109 GBPUSD,M1: 1 bar from 2022.12.01 00:00 added
2022.12.09 13:23:52.109 GBPUSD,M1: history begins from 2021.01.01 00:01

There is no mention of bar states generating on the non-chart symbol so perhaps you are right.

Fernando Carreiro 2022.12.09 13:32 #17

Enrique Dangeroux #:No worries Fernando. Test with every tick based on real ticks. While there are less bars with a difference vs 1 Minute OHLC and open prices, the average lag is relative consistant with the other modes.

Maybe the NewBar test is part of the "lag creation" as per your suggestion/example. Honestly i do not want to waste anyone's time. I have to retract the bug feeling. It always annoys me people blaming their incompetence to an external source like a bug. The current state is i fail to understand.

For me there is no bug or a problem. The dependencies are due to the bars time of opening not being accurate enough.

The first tick can arrive at any time, even a millisecond just before closing, yet the opening time only reflects the "ideal" time opening and not the true time.

In my multi symbol EAs, to compensate, I always align bar data as I described in posts a made previously on other threads:

Forum on trading, automated trading systems and testing trading strategies

Can a buffer update in the middle of two commands?

Fernando Carreiro, 2022.04.27 02:27

I suggest the following. Don't use sleep! Since it is multi-symbol, instead use the OnTimer() event handler in combination with OnTick() for the most active symbol (e.g. EURUSD), in order to detect time synchronisation of bar openings.

Also, there is no need to wait for a symbol's bar to close. If the bar's close time is up, then it is almost certain that it will close at the most recently received data later on, even if no new opening tick has arrived for that symbol. A way to handle this is to check if the current bar's ( open time + bar duration ) has lapsed. Based on this you will know if the current bar is pending closure or if it is a freshly created new bar.

When a new bar is supposed to have happened, you will check whether a new bar has actually been formed (and thus use the previous bar for the decision) or if the current bar is still pending a new opening tick (then you will use the current bar data as if it were already closed and in the previous bars data).

I hope my convoluted explanation made sense to you.

Forum on trading, automated trading systems and testing trading strategies

Can a buffer update in the middle of two commands?

Fernando Carreiro, 2022.04.28 05:23

You misunderstand. I'm not disputing the reason for creating a multi-symbol, multi-time-frame EAs. I have my own such EAs, and for all the reasons 1, 2 and 3. What I am questioning is the complexity that you are striving to implement.

Even for method 3, I have not needed so much complexity that you seem to want to implement. These are main steps I take.

For the most part, in multi-symbol EAs, I don't use indicators at all. I replicate their calculations in the EA itself in an incremental way, on each new bar event (see next step). I effectively maintain my own buffers (using ring buffers, only for the data I need, which is way more efficient than indicators).

On every new bar detection/pseudo-event (OnTick/OnTimer), I collect the bar data first (current and previous bar only, per symbol), then decide if the current bar is new and the previous one closed, or if it the current bar is stale and its duration lapsed, or even if the duration of several bars has lapsed with no updates. And based on this, I carry out the incremental calculations.

That is it. There is no more to consider. These conditions allow me to have synchronised data, in order to decide whether to enter or exit trades. After that, the management of positions is based on current position closing prices.

There are no worries. Even if there are changes between reading one symbol and the next symbol, it does not matter. Even if the data is not aligned when collected, does not matter. As long as I keep a state machine for each symbol, and do the incremental calculations, my data for each and every symbol will always be correctly aligned and synchronised.

Forum on trading, automated trading systems and testing trading strategies

Can a buffer update in the middle of two commands?

Fernando Carreiro, 2022.04.29 07:19

Remember what I wrote — I calculate things incrementally and I always keep track of the previous state of a symbol.

Lets say the new bar event has occurred at roughly 03h00 (H1 bars). I have in my current state for the symbol the complete data related to the bar that opened 01h00 and closed at 02h00. So, at 03h00, I read the last 2 bars (current & previous) with "CopyRates( _Symbol, _Period, 0, 2, ... )". I then check the open time of the last/current bar from the rates copied. Is it 03h00? If yes, great — then the previous bar is the one I want, which is the bar 02h00-03h00.

Now, what if the last/current bar is not 03h00 but instead 02h00? That would mean that no new tick has arrived to update the state. So I take the current bar and consider that the bar for 02h00-03h00 as closed and store it, but mark it as "uncertain" in my state machine for the symbol. On the next new bar event, when I ran the CopyRates() again, I can update the bar's data, but 99% of the time there would have been no change to the data as the next tick would be for the open of the next bar. On the off chance when due to network latency the missing ticks were actually for that same bar and not the open of the next, then the data will be slightly different and require a recalculation of indicator values, but the difference will be so small and insignificant that it would not affect the decisions previously made, and even if the change was larger, that is part of the strategy rules, and part of the acceptable tolerance and uncertainty of those rules, that will be managed according with the trade and risk management.

Now, for the cases of missing bars. When I have do CopyRates() and the bar is behind the expected value by a full bar, then I store a "blank/missing" bar with the appropriate state of missing and the data as open = high = low = close = previous bar's close, and tick_volume = 0, real_volume = 0, spread = 0. On the next bar event I continue the process as described above. If i discover that due the network/broker problems, I have missed an entire bar, then I update the data as described before, and accept that uncertainty, but I log the situation in the log for analysis. If I lost data due to network/broker problems, then I have to solve the network/broker problem. There is nothing that can be done for the EA data after the fact. The decisions by the EA are correct based on the collected data at the time, and those are part of the strategy rules. I simply accept that. Trade and risk management will take care of the rest.

Can a buffer update Creating fake ticks for Convert an MT5 EA

Enrique Dangeroux 2022.12.09 14:50 #18

Fernando Carreiro #:

For me there is no bug or a problem. The dependencies are due to the bars time of opening not being accurate enough.

The first tick can arrive at any time, even a millisecond just before closing, yet the opening time only reflects the "ideal" time opening and not the true time.

In my multi symbol EAs, to compensate, I always align bar data as I described in posts a made previously on other threads:

I will study your posts, as at a quick glance i already realize i need to switch to a state machine approach.

I do see some challenges regarding test/optimization. The open prices / 1 min OHLC modes seem out of the question because it will seriously deteriorate the states vs what actually happens in a real environment - real ticks. Even when the model recalculates only once a minute.

How to start with Every tick VS Every How to Start with

Lorentzos Roussos 2022.12.09 15:21 #19

Fernando Carreiro #:

For me there is no bug or a problem. The dependencies are due to the bars time of opening not being accurate enough.

The first tick can arrive at any time, even a millisecond just before closing, yet the opening time only reflects the "ideal" time opening and not the true time.

In my multi symbol EAs, to compensate, I always align bar data as I described in posts a made previously on other threads:

Interesting .

@EnriqueDangeroux if it shows up on other test modes too i would not trust the average that much as you have to consider that the leading event is the tick of the asset you chose on the tester.

So , i would expect some gaps there .

I believe the issue in the open prices may be the tester stacks up ticks (from the other symbols) to be "distributed" on the OnTick event , but , being only open prices it allots 1 tick and they have no time for the distro . This is an assumption and also "why would they" distribute other symbol ticks in ontick , and if the would not where would they distribute them ?

So if we have a multisymbol ea right ? does that mean you have access to the other symbol ticks from the previous bar only and you'll get those of the current on the next one ?

how come a line Raw Ideas Wishes for MT5

Bar (Time) lag in CopyRates on symbol other then current chart in open prices only mode. - page 2