Handling market data hiccups: Improving EA resilience.

[Oleksandr Medviediev]

Would like to discuss a challenge I've been facing since 2004 (MT3 at that time). As we all know EAs rely heavily on consistent and reliable market data to perform optimally. However, in the real world, sometimes we face data gaps and anomalies (let alone spread spikes and broker-server disconnects).

Just the other day my EA encountered an issue during a particularly volatile moment. There was a data gap that caused it to miss out some decent entry (needless to say, left me feeling frustrated).

I'm sure everybody is facing similar hiccups to occur every once in a while, so would love to learn insights from the wise minds here:

1. What techniques help to deal with market data gaps and anomalies?

2. How do you ensure your EA remains stable and responsive when data irregularity occur?

3. Have you built any specific error handling/recovery to mitigate the impact of data issues?

4. What are the best ways to handle midnight “roll-over“ crazy spreads?

5. Do you agree that history data gaps and irregularities help, in a way, to avoid "curve-fitting"?

Looking forward to an insightful and practical discussion!

[Yashar Seyyedin]

What techniques help to deal with market data gaps and anomalies?

==> I think the good approach is to avoid losses... Missing profit is not that important. So if I face an unpredictable incident I prefer to make sure the EA is doing nothing and I do not have a running trade with high exposure.

How do you ensure your EA remains stable and responsive when data irregularity occur?

==> As long you have error and exception handling you always maintain responsiveness and stability within your code. Relying on external sources that you cannot predict their behavior is risky.

Have you built any specific error handling/recovery to mitigate the impact of data issues?

==> I do not know what you mean by recovery. But one has to have a risk measure at every single moment of execution... Capital exposure could be good measure.

What are the best ways to handle midnight “roll-over“ crazy spreads.

==> If you are really a intraday scalper probably you have closed all your positions by rollover time. And if you are a swing trader your stop loss is far enough so it won't bother you that much.

Do you agree that history data gaps and irregularities help, in a way, to avoid "curve-fitting"?

==> I cannot understand how these two can be related... I do not think so.

[Oleksandr Medviediev]

Do you agree that history data gaps and irregularities help, in a way, to avoid "curve-fitting"?

==> I cannot understand how these two can be related... I do not think so.

All fair points, Thank you!

Let me clarify little bit on the last p.5. When backtesting strictly on 'pristine' data (has no gaps or irregularities) there's a risk of overfitting parameters to those 'ideal' data conditions. 

In other words, intentional random gaps/noise can act as a kind of <stress test> to avoid overfit.

So in that sense, I feel some level of 'imperfect' historical data, within reason, could be a BENEFIT rather than a BUG - as it reflects reality better. But I'm curious to hear if others have a different perspective on this.

[Yashar Seyyedin]

Oleksandr Medviediev #:

All fair points, Thank you!

Let me clarify little bit on the last p.5. When backtesting strictly on 'pristine' data (has no gaps or irregularities) there's a risk of overfitting parameters to those 'ideal' data conditions. 

In other words, intentional random gaps/noise can act as a kind of <stress test> to avoid overfit.

What I understand about over fitting is: detecting some kind of behavior by accident that cannot be relied on. If you agree with this even those gaps can cause misunderstanding about market behavior that could lead to over fitting.

[Oleksandr Medviediev]

Yashar Seyyedin #:
What I understand about over fitting is: detecting some kind of behavior by accident that cannot be relied on. If you agree with this even those gaps can cause misunderstanding about market behavior that could lead to over fitting.

Step-1: let's agree on terms.

Overfitting occurs when EA is excessively tuned to historical data - to the extent that it captures (i) noise, (ii) random fluctuations, thereby fitting factual price curve, rather than capturing genuine market patterns.

Agree?

[Yashar Seyyedin]

Oleksandr Medviediev #:

Step-1: let's agree on terms.

Overfitting occurs when EA is excessively tuned to historical data - to the extent that it captures (i) noise, (ii) random fluctuations,  thereby fitting factual price curve, rather than genuine market patterns.

Agree?

Not agree really. 

I call it over fitting when there is too much tunning that cause to detect patterns that are not really there and cannot predict future behavior.

[Oleksandr Medviediev]

“those gaps can cause misunderstanding about market behavior...”

leading to catching wrong patterns (not truly representative) during optimisation - that's a VALID point. 

In turn, once EA starts trading live - it hits sudden data gaps (not accounted during optimisation) that would interfere and prevent from accurately detecting genuine market patterns. The circle is closed.