Selection Bias in MT5 EA Backtests: Rank 1 of 28 Combinations, Then a Bottom-5% Out-of-Sample Quarter

Most of this diary has been about one gold EA. This entry is about a second line of research I have been running beside it, and about a mistake that is easy to make and hard to see from the inside: picking the best-looking result out of many and then measuring it as if it had been the only one. This week I measured how large that effect was in my own work, and then I got three months of data that arrived after the choice was made. The three months did not go well, and I think the way they did not go well is worth writing down.

The question I was trying to answer

The usual EA statistics answer the developer's question: what did the strategy earn over the whole history? A buyer has a different question. They install the EA on some ordinary day, starting with no open positions, and they want to know what the first two weeks, the first month and the first three months are likely to look like. Will it trade at all? Is it more likely to be up or down after ninety days?

So I built a ruler for that. Every weekday from January 2013 to late July 2026 is treated as a possible install date. From each date the strategy runs from flat, and I record what happened after 14, 30, 90 and 180 days: how many trades, whether the account was up, how deep the drawdown went. Risk per trade is scaled so that the worst drawdown over the test window is 29.0% for every candidate, which makes the percentages comparable. The ruler does not care about the total return; it cares about the distribution of first experiences.

Before running anything I wrote down three pass lines and fixed the document with a SHA-256 hash (it was locked at 16:52 JST on 24 October, before the first number was computed):

• median trades per month of at least 12,
• at least 70% of install dates in profit after 90 days,
• no more than 10% of install dates with zero trades in the first 14 days.

The strategy itself is a long-only breakout rule with a fixed 2R target and a time exit. Its details are not the subject here. What matters is that the same rule was applied to three candidates.

Three candidates, three failures

Gold alone trades too rarely and is in profit after 90 days only a little more often than not. Nine markets trade constantly but spread the risk over markets where the rule loses money after costs, so the result is close to a coin flip. The three-market version came closest: enough trades, no empty first fortnight, and 69.7% of install dates in profit after 90 days, 0.3 percentage points short of the line.

A miss of 0.3 points sounds like it might be noise, and it is. I also ran four pre-registered sensitivity variants: three that change how costs and swaps are counted, and one that uses the older research engine. In three of them the three-market version passes (74.2%, 71.5%, 70.5%), in one it fails more clearly (67.6%). The honest summary is "on the boundary". The ninety-day windows also overlap heavily: there are about 3,500 install dates, but only about 54 non-overlapping ninety-day windows in 13.5 years, so the uncertainty in that 69.7% is much larger than the 0.3-point gap.

If that were the whole story, the obvious next step would be to put the three-market version on a demo account and see whether it lands above or below the line. It is not the whole story.

Where the three markets came from

I did not choose USDJPY and DAX in advance. In August I ran the same rule across nine markets and looked at the results. USDJPY and DAX looked good, so they became the companions for gold. That is a selection made after seeing the outcome, and it means the three-market numbers above are not a fair estimate of anything. They are the numbers of the winner.

I already knew this in the abstract. What I had not done was measure it. So the pre-registered plan included, among its extra outputs, one step: take gold, add every possible pair from the other eight markets, and run the same ruler on all 28 combinations.

The combination I had picked "because it looked good" was the single best of 28, on both measures. None of the 28 passed all three lines. The typical way to add two markets to gold, with no hindsight at all, gives roughly a coin flip at 90 days and a yearly result close to zero.

Something else was visible in the ranking: six of the top seven combinations in the 90-day ranking included USDJPY. Most of the uplift that made the top of the table look good came from one market.

The August numbers had also shrunk once the backtest was rebuilt to match the EA. In August, the DAX leg showed +0.158R per trade. That came from a research engine with narrower data-vendor spreads and no swap. Run through the same logic as the EA, with spreads floored at what my broker actually quotes and with measured swap rates, it is +0.070R. Each of those steps was a correction toward reality, and each one moved the number down.

Three months the backtest had never seen

The historical data I had used ended on 24 July 2026. The combination was chosen in August, so everything after 24 July is, for this purpose, out of sample: no decision about the combination had been made with it. I downloaded the missing three months for gold, USDJPY and DAX from the same free data source.

Before using the new data I checked that it agreed with the old. The two downloads overlap from 1 to 24 July, and every row in that overlap was identical in all three markets, bid and ask. Re-running the simulation on the joined data reproduced every trade that closed on or before 24 July exactly (1,316 gold trades, 1,185 USDJPY, 1,062 DAX). In gold, the four simulated trades since the end of August could also be checked against the real fills of the same rule on a demo account that started then: same days, entry prices within about a dollar, same exit types.

The pre-registration was not changed. I recomputed its hash and it still matched. Then I ran a single ninety-day window: an imaginary user installs on 25 July 2026 and the clock stops on 23 October.

The historical median for a ninety-day window of the three-market version is +5.35%, and its 10th percentile is -6.74%. The first ninety days after I picked it came in at -9.43%, worse than 95% of the windows in 13.5 years. The drawdown inside the window, 14.86%, was larger than 95% of historical windows too. Almost all of the damage came from USDJPY, the market that had pushed the combination to the top of the ranking: 19 of its 25 trades hit the stop, and its result was worse than 99% of its own history. Gold, the original single-market candidate, had an ordinary quarter and finished slightly up, although with only two trades in September.

What this does and does not show

It would be easy to write this up as "selection bias caught in the act". That would overstate it. One ninety-day window is one observation. A 5th-percentile outcome is something that happens once in twenty tries even when the historical distribution is exactly right. I cannot tell from this whether I was unlucky or whether the historical numbers were inflated by the choice. The shorter horizons are no better: inside these three months there are only about six non-overlapping fourteen-day windows and three thirty-day ones, so the fact that just 34.2% of thirty-day windows were in profit (against 61.6% historically) cannot be separated from chance either.

What I can say is narrower. The concern raised by the 28-combination test, that the picked combination was the most flattering one available, predicted that out-of-sample results would come in below the backtest. They did. A good quarter would have been a weak argument against the concern; a bad quarter is a weak argument for it. It is not a reason for confidence in either direction, and it is certainly not good news.

The pre-registered verdict stays exactly where it was: all three candidates fail, the three-market version on the boundary. The new quarter is recorded as an observation and not added to the test, because the plan did not include it. Changing the rules of the test after seeing a result is the same mistake in a different place.

The tempting fix, and why I am not making it

Having seen that USDJPY did the damage, the natural move is to drop it and try gold plus DAX, or gold plus something else. That would be choosing again, after looking again, from the same table. The new combination would carry the same kind of hindsight as the old one, only with three more months of it. If I want to test a different combination, it has to start from a new pre-registration, with the expected range written down before any new data is seen, and with the 28-combination median, not the winner, as the realistic expectation.

There is also a structural reason to be careful with USDJPY specifically. Across the yearly results in the test window, the gold leg and the USDJPY leg had a correlation of -0.49. Gold's bad years were often good years for the yen trade, and that is the main reason the three-market version had a smaller drawdown per unit of risk than gold alone, which is what allowed it to run at higher risk per trade after normalisation. Most of that period was a long stretch of yen weakness. A diversification benefit that rests on one macro regime continuing is not something I would want to assume.

The trial count

This experiment counted 42 configurations: three candidates, twelve sensitivity runs, and the 27 other combinations in the selection test. The running ledger of every configuration I have tested in this strategy family went from 1,015 to 1,057. I keep that number because it is the denominator any honest significance claim for this family has to use. Reporting the best of 1,057 tries as if it were the only one would be the same error this entry is about, applied at a larger scale.

What I take from it

The specific lesson is that "it looked good when I ran it across several markets" is not a selection criterion. It is the selection. Once the comparison set is known, ranking the pick inside it takes a few minutes and tells you how much of the result is the act of choosing. In my case the answer was: most of it. A median combination would have offered a coin flip at ninety days; the backtest of my pick offered just under seventy percent; and the first real quarter after the pick was a loss in the bottom 5% of its own history.

The general lesson is about where the out-of-sample period starts. It does not start when the data ends. It starts when the last decision was made. My data ended in July; my decision was made in August, using data up to July. Everything after that was clean, and it was the first thing that had been allowed to disagree with me.

The multi-market version is not going to a live account on the strength of these numbers. If it is tested further, it will be on a demo account, against a range written down in advance, and judged only on data that arrives after that range is fixed.