Seven Checks, Two False Passes: Auditing an EA Against Its Own Backtest

This is the second entry in a public incubation log for a gold breakout EA I am not selling yet. The first entry audited five trading days of live forward testing and found that only one of them had produced a valid measurement. This one goes a layer deeper: it is about the audit I ran to check whether the MQL5 robot actually implements the Python research that produced it.

The short version is that I ran seven checks, recorded five exact matches, and two days later discovered that two of those five matches were false. One of them was inflating the headline expectancy number by a factor of six. The audit did not catch it, and the reason it did not catch it is the part worth writing down.

What was being compared

The strategy is simple enough to state in one paragraph. On XAUUSD M15, take the highest high and lowest low of the last 128 completed bars, place a buy stop at the upper level and a sell stop at the lower one, and only do this inside a fixed session window defined in true UTC. The initial stop is 0.5 ATR. The trail arms once open profit exceeds 0.5 ATR and then follows the maximum favourable excursion minus 0.25 ATR. ATR here is the simple mean of M15 high-minus-low over 96 bars — not Wilder's ATR, which is a different number and a classic silent divergence between a research script and a platform indicator.

The research side is a Python simulator that walks M1 bars. The live side is an MQL5 EA running on a demo account. Both are supposed to be the same strategy. The audit was an attempt to prove that claim rather than assume it.

The seven points, as I scored them at the time

Then there was an eighth item that was not on the checklist at all, which I will come back to, because it is the only thing the audit found that the audit was not designed to find.

The two divergences I knew about and kept

The trail reference point. The Python reference measures the stop and the trail from the break level. The EA measures them from the actual fill price. That is not an oversight; it is the natural implementation on MT5, and I tested it before shipping it. On 662 real-tick trades between 2025-08 and 2026-07, fill-referenced gave +0.2580 R per trade against +0.2437 for level-referenced. Level-referencing has a slightly higher win rate (59.7% vs 58.8%) but only because it widens the stop by the slippage — average risk goes from 0.500 ATR to 0.524 ATR — and it comes out worse on both dollars per trade and risk-normalised return.

The honest caveat is that this test was run on a 64-bar break in a different session window, not on the 128-bar session-A configuration that is actually deployed. It is a decision carried over from a neighbouring configuration rather than re-validated in the one that ships, and I am recording it as such rather than pretending otherwise.

The chase limit. The EA has an input the Python side has no counterpart for: if price is already beyond the level at the moment the order would be placed, chase it with a market order up to 0.10 ATR, otherwise skip. This exists because of a backtest trap I hit earlier — if you let a simulator record "filled at the level" when price is already past it, the trade opens in profit and you get a fictitious R+7.34 with an 88% win rate.

Having added the guard, I then measured whether it was doing anything. It was not. Price is already through the level at order time in 0.14% of cases, and of those, 15% get skipped — 0.02% of all orders. Sweeping the limit from 0.05 to unlimited moves expectancy by ±0.003 R, which is noise. The guard is not conservative; it is inert. That distinction matters and I will return to it.

The thing the checklist could not see

Partway through I stopped comparing line to line and asked a different question: what state can the live EA reach that the reference implementation cannot represent at all?

The answer was immediate. The EA places both a buy stop and a sell stop, and it only refreshes them on the next M15 bar. Inside that 15-minute gap, a whipsaw can fill both. The Python reference is one signal, one direction — that state does not exist in it, so no amount of comparing the two implementations would ever surface it. It is a real exposure: for up to fifteen minutes you can be long and short at once, paying the spread twice, and gold whipsaws are not hypothetical.

The fix was small — OnTradeTransaction catches an own-magic DEAL_ENTRY_IN and deletes the remaining pendings, behind an input so it can be switched off for testing. But the lesson is not the fix. The lesson is that a diff between two implementations is blind to anything only one of them can do, and those are exactly the places where live money gets lost.

The false pass

Two days after I filed the audit as complete, I was measuring something unrelated and noticed that the EA calls DeleteOwnPendings() as the first statement of RefreshPendings() , which runs on every M15 bar. Every pending order is deleted and re-placed every fifteen minutes. The code path is unambiguous, and the live journal shows the cadence: the same buy stop 4371.85 going out at 03:00, 03:15, 03:30, over and over.

So PendingValidMinutes = 240 is dead. It can never expire an order, because nothing survives long enough to reach it. Both sides of my audit said "240", the values matched, and I ticked the box. Meanwhile the Python reference was genuinely keeping each stop order alive for 240 minutes.

That is not a rounding difference. It means the reference counted the same real fill an average of 4.3 times. Worse, the duplication is not random: a level that stands unbroken for a long time generates more duplicate rows, and a level that stands unbroken tends to break well. The duplicates were weighted toward the good trades. It is bias, not variance.

Two independent routes — deduplicating fills, and re-framing the simulator to match the EA — land on +0.049 and +0.060, which is the reassuring part. The last row is a separate bug found in the same pass: the session window was taken as a closed interval, so consecutive rows shared a minute bar, and fills on that shared bar were being counted twice.

The number I now carry forward for the shipped configuration is +0.0068 R per trade, profit factor 1.011, with out-of-sample (2018-02 onward) at −0.1172 and a real-tick year at R −0.0355, PF 0.933. In plain language: after costs, the current configuration has no edge I can demonstrate. That is what the correction produced, and publishing it is the entire point of writing these entries in public.

One more check on that claim, because "maybe you should just leave the orders alive for 240 minutes and get the good number back" is the obvious objection. I measured it. Profit attributable to stale orders — fills on levels that were no longer current — is 3.1% of total R. The 240-minute advantage was the duplicate weighting, not the extra order lifetime. Keeping the orders alive does not recover it. That road is closed.

The second false pass

The same pass turned up an off-by-one bar in the session boundary, which was audit point #1 and also scored as an exact match. The simulator was deciding session membership from the label time of the signal bar, so it traded a bar the live EA does not trade and skipped one the live EA does. The direction is opposite on the two ends of the window. Small, but it is the second item in five where the parameters agreed and the behaviour did not.

Why an audit passes things it should fail

Both false passes have the same shape: I compared values when I should have compared behaviour under the conditions that actually occur. "Both sides use 240" is not a finding. The finding is what fraction of decisions the number changes, and for that parameter the answer was zero.

Once I started asking that question of every input, four of them turned out to be inert in the current configuration:

• PendingValidMinutes = 240 — never reached; orders live 15 minutes.
• MaxHoldMinutes = 480 — zero time-stop exits in 16.5 years of simulation. Mean hold is 3.6 minutes, 99th percentile 27 minutes. Anything from 120 to 1440 produces identical results.
• MaxChaseATR = 0.10 — binds on 0.02% of orders.
• MaxConcurrentPositions = 3 — never binds. At 3.27 fills per day and a 3.6-minute mean hold, the third slot is not reached. A cap of 2 would bind 0.4% of the time; a cap of 1, 11.4%.

I kept all four and annotated them as inert rather than deleting them, because a different configuration will wake them up. But an input that changes nothing is not a safety margin. It is a comment that looks like a control, and while it sits there it makes the system feel more guarded than it is.

What the live account has said so far

The forward test has now produced its first fills. Six closed trades, all exiting through the stop — which includes trailed exits, since the trail is implemented by moving the stop.

Six trades is not evidence of anything about profitability, and I am not presenting it as such. Counting non-overlapping breaks honestly, this strategy needs on the order of 200 trades — roughly seven months — before the P&L carries any statistical weight. What six trades can do is corroborate mechanics, and on that front they are already useful.

The holding times are 0, 1, 0, 2, 6 and 2 minutes. The 480-minute limit is 80 times the longest one observed. That is the "dead input" finding from the audit showing up in live data within a week.

The 19th is also a live picture of the re-entry clustering that the broken counting then multiplied. Five long entries in under an hour at 4407, 4421, 4442, 4461 and 4464 — one trend, re-entered each time the 128-bar high moved up and a new stop order was placed above it. These are real, separate trades; the old framing then counted each of them several times over, which is how it arrived at an average of 10.6 rows per break. Live, the entries are sequential rather than concurrent, because the holds are minutes long — which is exactly why the concurrency cap never binds. Two of those five paid for the other three, and that is the shape the research describes: in the backtest, the top 5% of trades carry the total.

One more correction worth recording. The first fill closed at R = −1.1243, which my monitoring flagged as an anomalous overshoot of the initial stop. It is not. A short is opened at the bid and closed at the ask, and the stop width does not include the spread, so a stop-out is structurally required to land at −(1 + spread ÷ stop width) = −(1 + 0.410 ÷ 3.362) = −1.1220. Measured −1.1243, the remainder being rounding. My detector was flagging 100% of stop exits as anomalies, which is the same failure mode as an alarm that cries wolf: it was worse than having no detector. It now tests against the spread-adjusted floor for stop exits and against −1.0 only for trailed and time-stop exits, where price should never reach the initial stop.

Sitting under this is a pre-registered observation I am not allowed to conclude yet. Cost as a fraction of stop width, measured from bar samples in this session, was 7.20%. The first fill measured 12.20%. Across the six fills the median is 6.38%. The hypothesis is that breakout stop orders are preferentially filled in the moments when the spread is widest — which, if true, means sampling cost from bars understates it structurally. I wrote the failure criterion before looking: if the median across 20 or more stop exits lands inside 5.8–8.6%, the hypothesis is dead. n is 6. No conclusion.

If you are porting research to MQL5

Four things I would do differently, in the order they would have saved me the most time:

1. Diff behaviour, not parameter values. For every input, state what fraction of decisions it changes. If the answer is zero, that is a finding about the code, not a clean bill of health.
2. Ask what the platform can do that your reference cannot represent. Two simultaneous fills, partial fills, requotes, a stop order surviving a reload. Your simulator has no vocabulary for these, so a diff will never mention them.
3. Read the live journal against the reference, not the reference against itself. The 15-minute order lifetime was sitting in plain text in the journal for days before I read it as a discrepancy.
4. Expect the correction to go the wrong way. Both of mine made the strategy look worse. If your reconciliations keep improving the numbers, that is worth a second look on its own.

The audit's real output was not the seven ticks. It was a research number cut from +0.318 to +0.007 R per trade, a dead safety input, and a two-sided exposure that no comparison could have found. I would rather publish that than a curve.

Next entry: why adding a take profit makes this system worse, and what the exit matrix actually looked like. If you want to see the free tools I have published while this EA incubates, they are on my MQL5 profile.