Monte Carlo Permutation Test for Trading Strategies: Why My Placebo Was Blind to Long-Only Systems

19 November 2026, 01:26
Yuki Nakayama
0
5

On Monday I found a hole in the test I use to decide whether a strategy has an edge. The test had passed its own validation the same morning. It matched three earlier studies, and it still could not measure five of the nine strategy legs I currently run on a demo account. It did not mis-measure them. It could not measure them at all, and it reported that as a clear failure. This entry is about how that happened, how I fixed it, what the fixed test says about those nine legs, and the part I find most useful: two different placebo tests can give the same strategy very different scores, because they are asking different questions.

Everything here is research on historical data plus a demo account. Nothing is a product and nothing runs on live money.

What the test does

Every candidate I test goes through the same Python bench. The data is split by entry time into a build half (January 2015 to December 2020), which I may look at as often as I like, and a score half (January 2021 to October 2026), which counts as one trial every time I read it. Data before September 2014 is held back for a final check and is discarded as soon as it is loaded.

The main line is a placebo, or Monte Carlo permutation, test. The question is: would a version of this strategy with no skill do as well? You build 2,000 versions of the strategy that keep everything except the part you think is skill, and see where the real result falls among them. My line is that the real average trade in the score half, before costs, must be above the top 1.67% of the 2,000 placebos (5% divided by three, and tighter once more than three trials have been read). Two other lines follow: still positive after my broker's measured costs, and above the placebo median in the build half.

The bench was rebuilt into one standard form on 16 November, replacing a collection of older scripts. The placebo it used was a direction shuffle. Each trade keeps its exact entry and exit time. For each placebo, each trade is randomly made a buy or a sell, with the probability of a buy set to that instrument's actual share of buys. If a strategy has skill at choosing direction, the real result should beat most of these shuffles.

Before using the rebuilt bench I checked it against three earlier rounds whose results I knew: one that had passed and two that had failed. Trade counts, averages, ranks and pass/fail results all matched. I took that as validation.

The hole

Here is the line that draws the placebo, simplified:

q = share_of_buys(instrument)          # e.g. 0.57
is_buy_in_placebo = random() < q        # per trade, 2,000 times
result = where(is_buy_in_placebo, R_if_bought, R_if_sold).mean()
percentile = share of placebos >= real result

Now feed it a strategy that only ever buys. Its share of buys is 1.0, so every placebo trade is a buy, so every one of the 2,000 placebos is identical to the real strategy. The real result is never above the placebos. It ties all of them, and with my tie rule (greater than or equal counts against the strategy) it landed at exactly 100%, a clear fail, in all ten cases (five legs, two halves), whatever the strategy does.

If I had written "greater than" instead of "greater than or equal", the same strategy would have landed at 0%, a perfect pass. And because the tie depends on two averages being equal to the last bit, a rounding difference could flip either version between 0% and 100%. One character, or one rounding error, decides the direction of the error, and neither direction measures anything.

Five of the nine legs on my demo account only buy (the first item below is two legs):

  • A gold breakout that holds for up to 72 hours, and a second version of the same rule on USDJPY and the German index.
  • The public RSI(2) pullback buy, on five index CFDs.
  • A pre-holiday hold on four US index CFDs.
  • A hold over the day before FOMC announcements, on two US index CFDs.

All five came out at exactly 100.00% in both halves under the direction shuffle. The other four legs buy between 31% and 57% of the time. The three validation rounds did not show the problem because their buy shares were between 41% and 57%. The validation only covered cases where the hole could not show.

The older scripts had not had this problem everywhere. The earlier studies behind the pre-holiday and RSI(2) legs, for example, had used placebos built from random days, not random directions. The rebuild into one standard bench kept the simpler placebo and dropped the one that could test these legs. I found it later that morning, while planning a round of long-only candidates and reading the placebo code to see how they would be scored.

It is a bigger blind spot than it first sounds. A strategy that buys 90% of the time is not exactly at 100%, but its placebos differ from it only in the few trades that get flipped, so the test is almost blind there too. And a strategy that buys a rising index on certain days can make money only because it is long. A direction shuffle cannot separate "this rule picks good days to be long" from "being long was good".

The fix: shift the day, keep the rule

For a long-only strategy the real question is about timing: are the days or hours this rule chooses better than other days on which the same rule could have been applied? So the new placebo keeps the direction and the trade rules and moves the entry to a different day:

  • Same instrument, same half, same year, same weekday, an ordinary day chosen at random.
  • The same entry time of day, the same stop, target and time exit as the real trade. For the RSI(2) leg the placebo buys at the close of a random day and uses the same exit rule. For the month-end leg it applies the same formula on a day that is not at month-end.
  • If two instruments had a real trade on the same day, the placebo moves both to the same random day.

The last point matters more than I expected. I tested it on synthetic data before using it on anything real:

  • No edge, a rising market, one instrument, 1,500 runs: the placebo test flagged the strategy above its top 5% in 5.3% of runs and above its top 1.67% in 2.0% of runs. That is close to what it should be by construction.
  • No edge, four instruments correlated at 0.7, the same random day for all four: 4.2% and 1.7%. Also close.
  • The same four instruments, with each one given its own random day: 15.9% and 9.7%. That is about three and six times too many false passes. Correlated instruments that move together on the real days do not move together on independent random days, so the placebo spread is too narrow and the real result looks unusual when it is not. I did not use this version.
  • A synthetic calendar effect of +0.2R per trade: detected at my main line in 55.2% of runs. The fixed test can see an edge of that size about half the time, which is a useful reminder of its power.

Two more checks. With the placebo set back to direction shuffle, the bench reproduced the old outputs exactly: every one of 55 result columns and four trade lists unchanged. And against three studies from the older scripts, which had used random-day placebos, the new version gave 11.05% against 12.0% for the pre-holiday rule, 0.60% against 1.1% for RSI(2), and 14.60% against 10.2% for a turn-of-month rule. The last gap is the largest, and it comes from where the random days are drawn from: the old study drew from the same month, the new one from the same year and weekday.

Finally a gate, so the hole cannot come back unnoticed. Any row where an instrument buys 90% of the time or more, or 10% or less, is stopped before any profit or loss is read if it is set to direction shuffle. It is not measured and not counted as a trial. I tested the gate with a deliberately always-long test row, and the ledger copy was unchanged afterwards.

The nine legs, measured both ways

Percentiles below are where the real result falls among 2,000 placebos, counted from the top, so smaller is stronger. The main line is 1.67% in the score half (2021 to 2026). The build half is 2015 to 2020, except where the data starts later.

LegBuysDirection shuffle
score / build
Day shift
score / build
Trades
score / build
Intraday US100 noise-band leg51%0.05% / 2.00%0.00% / 0.70%672 / 276
RSI(2) pullback, five indices100%100% / 100%0.05% / 43.65%270 / 197
Bollinger leg, USDJPY51%0.00% / 16.15%0.25% / 12.15%1,045 / 1,067
72-hour breakout, USDJPY and German index100%100% / 100%1.65% / 1.10%923 / 717
NR7 breakout, USDJPY57%3.35% / 52.40%2.25% / 33.55%222 / 248
72-hour breakout, gold100%100% / 100%3.10% / 0.10%466 / 420
Pre-holiday hold, four US indices100%100% / 100%15.75% / 42.95%224 / 196
Month-end rebalancing reversal, two indices31%4.95% / 0.25%15.85% / 18.90%70 / 92
Pre-FOMC hold, two US indices100%100% / 100%25.50% / 24.75%94 / 46

Three things stand out.

The five long-only legs now get a number, and they split. RSI(2) is strong in the score half (0.05%) but sits in the middle of its placebos in the build half (43.65%), so its timing edge is visible in one period and not the other. The second breakout version is at 1.65% against a line of 1.67%: inside by two hundredths of a point, which I read as on the line, not as a clear pass. The gold breakout is just short at 3.10% in the score half and very strong in the build half. The pre-holiday and pre-FOMC legs are visible but far from the line. Their average trades are well above their placebos (the pre-holiday leg made +0.105R per trade before costs in the score half against a placebo median of +0.021R), but with this few trades the placebo spread is wide, and a random-day long position earns part of the same return (about a fifth for the pre-holiday leg, about 60% for the pre-FOMC leg).

The day-shift placebo does not just add numbers to the long-only rows. It changes one of the others. The month-end reversal leg looked good under the direction shuffle: 4.95% in the score half and 0.25% in the build half. Under the day shift it is 15.85% and 18.90%. Both are correct answers to different questions. The direction shuffle asks whether the rule picks the right direction on the days it trades. The answer is mostly yes: strong in the build half, short of my line in the score half. The day shift asks whether month-end is a special date for this rule, and the answer is: not clearly. With only 70 and 92 trades, the same formula applied on ordinary days matches or beats the real result in about 16% to 19% of random draws, so I cannot separate month-end from an ordinary day at my line. The reason I added this leg was month-end flows. The day-shift test does not support that reason as a separate effect, even though the rule's direction choice looks better than chance.

A high win rate means little until you compare it with the placebo's. The real RSI(2) leg won 7 percentage points more often than its random-day placebo, and the pre-holiday leg 8 points more. A high win rate on a long-only index strategy mostly reflects rising indices and the exit rule, not the entry. Most of the high win rate is there on random days too, and only the gap above the placebo is about timing. The Bollinger leg is the opposite case: its win rate is low, but 16 points above its placebo's, and it is one of the strongest rows in the table.

A note on units: the Bollinger leg's day-shift row is measured in multiples of the 20-bar standard deviation rather than R, the same unit as the study it came from. The percentile is unaffected, because the real result and the placebos are in the same unit.

What I changed and what I did not

The direction shuffle is still the default. For rows that trade in both directions it is the right question, and for those rows its results were reproduced exactly. The day shift is a second placebo that a row can name, and the gate forces the choice for rows that are mostly one-directional. The bench's fingerprint changed, so any test table that was frozen before the fix has to be frozen again before it runs. For new candidates, only one signal type, the day after a narrow-range day, has the day-shift placebo built so far. Others need one more hook each.

I did not remove or add any leg because of this check. The number of legs is nine before and after. The legs reached the demo account through earlier, different tests, and this check answers one narrower question: do their entries have timing skill compared with other days? Whether the pre-holiday, pre-FOMC and month-end legs should stay is an open decision. Those three are the ones this check leaves weakest, and I am not going to make the decision in the same week as the measurement that prompted it.

I also did not count this as a trial in the ledger of score-half reads. It was a check of existing legs, not a search for new ones, and I did not tune anything after reading it. If I change a leg because of it, that change gets its own trial.

The checklist I am keeping

  • Before trusting a placebo test, feed it a strategy that cannot have skill under that placebo, such as an always-long rule for a direction shuffle. If the answer is fixed whatever the strategy does, the test is blind there.
  • Check your tie rule. If many placebos can equal the real result, "greater than" and "greater than or equal" give opposite answers.
  • Validate a rebuilt test on cases that could break it, not only on cases you already know. My three reference rounds all traded both directions, so they could not expose a long-only hole.
  • When instruments are correlated, give them the same random day. Independent random days made my test about three times too lenient on synthetic data.
  • Write down which question the placebo asks. "Right direction on these days" and "these days are special" are different claims, and one leg passed one and not the other.

For readers who evaluate EAs or signals: when a seller shows a Monte Carlo or permutation test, ask what was shuffled. Shuffling trade order shows a range of drawdowns; it says nothing about whether there is an edge. Shuffling direction cannot test a strategy that only buys. Shuffling the day can, but only if correlated markets are moved together.

Previous entries in this journal are on my blog page. The cost line in this bench uses my broker's measured costs rather than a flat assumption; if you want to measure your own broker's costs per symbol, my free Execution Caliper is a cost meter that runs inside MT5.