Scalping system shows zero directional edge across 31 features — is there anything left to test? - page 2

New comment
Mohamed Trading #:
There is a model in the loop making
the entry decision, and whether it also closes positions itself, or
whether every exit goes through stop and target, is exactly what I am
asking the codebase.

You might not need the codebase for that. MT5 stamps every closing deal with a reason, DEAL_REASON in the history: SL, TP, or something else when a program closed it. Split the 60 by that and put the average R beside each group. If the closes that weren't SL or TP average well under 1.3 R, that's where your ratio went. And if your tests assumed every exit was a stop or a target, this also tells you whether the system you tested is the one that traded.

Timon-pascal Krueger #:
Arnold's point about exits matches what I measured. I ran 18,900 trade management policies on a surrogate market, identical code on a series built so that no pattern can exist, and all of them ended at minus the costs. Exits change the shape of the results, win rate against average win, but on entries without information they do not create expectancy.

On your question 4, I rebuilt the Zarattini opening range breakout on Nasdaq rule for rule and got the paper back gross, +0.13 R per trade at t 2.9. With CFD costs it came out at +0.002 R, and negative on the four other indices I tried.

On question 2, I recently scanned 15 markets and took 16 candidates that survived discovery into a separate holdout from 2023 on. None held after a correction across the 16, and every discovery t between 2.2 and 4.9 shrank. My best real finding was t 3.65 while the best finding on random data had a median of 3.81. So finding nothing is the normal result, and your nulls are the reason you noticed.
Your point held on our data, and it closed a hypothesis I opened that
same morning.

We had found our realised ratio was 0.906 against a designed 1:1.3, with
the average win at 0.882 R below a 1.3 R target. The obvious read was
that something was cutting winners short. So we audited every exit path.

All 60 exits were the broker's own: 44 stops, 16 targets. The model can
close positions and never did, not once. The twelve winners below 1 R
were locked at exactly +0.40 R by a giveback rule.

Then we measured what those twelve would have done if held: 5 would have
reached target, 7 would have hit the stop. Realised with the lock,
+4.82 R. Held, +0.43 R. The early exits GAINED 4.38 R.

So the exit suppressed the average win and improved the result at the
same time. Your point exactly. I had it backwards.

Two questions, if you don't mind.

Your surrogate market: how do you build a series where no pattern CAN
exist, rather than one where none happens to? We use random-entry nulls
and a volatility-clustering positive control, but a provably signal-free
series is a stronger instrument than anything we have.

And the harder one: you scanned 15 markets, carried 16 candidates into a
clean holdout, none survived, best real t 3.65 against a random median
of 3.81. You have done this more carefully than we have and reached the
same place. So where do you look next — or have you concluded something
about where a retail-accessible edge can and cannot live?

I'm not asking for anything you trade. I've built the measuring side of
this properly — pre-declared rules, holdouts, nulls, positive controls,
6,497 tests — and I have no edge to put through it. If you ever want a
second pair of hands on a test, or an independent replication of
something, I'd be glad to. Either way, your replies have already saved
me weeks.
Arnold Holm #:
Your null result is a useful result. I would not try to rescue the same trade list by searching more exits and position sizes. First check the test plumbing with one timestamped fixture: signal time, entry and exit prices, spread, commission, broker time, and the log line showing when the signal existed. You already found pinned inputs and look-ahead, so that audit matters.

Then freeze one new hypothesis before opening another holdout: signal, costs, maximum number of variants, and the condition that ends the search. Compare it with a matched random-entry null before choosing stops or sizing. An exit rule may change the distribution, but that is a new hypothesis to test, not evidence that the original entry carried direction.

If the corrected pipeline still shows no edge after costs, I would pause this scalper. Another venue or longer timeframe would be a fresh test with its own untouched period. The 60 demo trades help check execution; they cannot settle whether the strategy has an edge. There is no universal minimum trade count. Define the smallest improvement that would clear your real costs first, then size the test around that.
Ran the plumbing audit you suggested — one trade, end to end.

Signal timestamp from the log line, order, fill, prices against the
broker's own record, tick spread, commission, broker clock, and every
feature value at decision time. All three sources agree on the prices.
R reconciles to 0.0033, which is a rounded stop level, not a defect.
Nothing in the chain postdates the signal.

But it turned up eleven discrepancies, and two mattered:
- our database's open time was the row-write time, a median 36 s before
  the fill, on 59 of 60 trades. Two studies that used it as the entry
  time are being recomputed.
- a spread cap had silently fallen back to a static default for weeks
  because of an import failure. Two layers of silent fallback, so the
  measured cap was never enforced anywhere. It now refuses the entry
  instead of failing quietly, with a test that flags any safety limit
  that falls back instead of halting.

Neither would have been found by testing another hypothesis. You were
right that the plumbing comes first.
Ayaz #:

You might not need the codebase for that. MT5 stamps every closing deal with a reason, DEAL_REASON in the history: SL, TP, or something else when a program closed it. Split the 60 by that and put the average R beside each group. If the closes that weren't SL or TP average well under 1.3 R, that's where your ratio went. And if your tests assumed every exit was a stop or a target, this also tells you whether the system you tested is the one that traded.

@Ayaz :
The audit came back, and it went the other way.

All 60 exits were the broker's own: 44 stops, 16 targets. The model can
close positions and never did. The twelve winners below 1 R were locked
at +0.40 R by a giveback rule.

Then we measured the counterfactual: of those twelve, 5 would have
reached target and 7 would have hit the stop. Realised +4.82 R, held
+0.43 R. The early exits gained 4.38 R.

So your arithmetic was right and my inference from it was wrong. The
ratio really was 0.906, and the exit really was suppressing the average
win — but it was adding expectancy, not giving it away.

It also found something I would not have looked for: a spread cap that
had silently fallen back to a default for weeks, so a safety limit we
believed was enforced never was. That came out of tracing one trade end
to end, which only happened because of this thread.

Still owe you the answer on your earlier question about what separates
the systems that survive. If you ever feel like answering it, I'd read
it carefully.
Ayaz #:

You might not need the codebase for that. MT5 stamps every closing deal with a reason, DEAL_REASON in the history: SL, TP, or something else when a program closed it. Split the 60 by that and put the average R beside each group. If the closes that weren't SL or TP average well under 1.3 R, that's where your ratio went. And if your tests assumed every exit was a stop or a target, this also tells you whether the system you tested is the one that traded.

Did exactly that, from the broker's own deal records: 44 stops,
16 targets, zero program closes. The model can issue closes and never
did once.

But the split needs a second column. Of the 44 stops, 14 were at levels
our governor had MOVED after entry — the deal reason still reads SL,
because a stop is what filled. Twelve of those sat at +0.40 R from a
giveback rule, and that is precisely where the average win went.
DEAL_REASON alone says the program never touched those trades; it
touched them constantly, by moving the stop rather than by closing.

So: 30 original stops, 14 moved stops, 16 targets.

Then the counterfactual on the twelve, measured on M1 bars with spread:
held to the original levels, 5 reach target and 7 hit the stop, +0.43 R.
Realised with the lock, +4.82 R. The suppression was worth 4.38 R.

Your last line was the one that mattered. One of our studies modelled
the original geometry and never included the giveback rule — so the
system we tested was not the system that traded. It is being recomputed
with the rule in it, against the corrected fill times rather than our
row-write times, which ran a median 36 seconds early on 59 of 60 trades.

Which brings me to the question I should have asked you at the start:
of all the systems you've written reports on, what separates the ones
that survived from the ones that died?
Mohamed Trading #:
of all the systems you've written reports on, what separates the ones
that survived from the ones that died?

From the reports, two things separated them.

One you've already seen: the win rate against the break-even rate its own average win and loss set. Two configurations tested over sixteen years needed about 80% and won 74%, and both went to zero. The account that was making money needed 88.50% and won 94.13%.

The other is what's still open at the end. The best run of a grid EA I tested finished at +262%, holding ten positions $21,765 under water that had been open 540 days. Before that, four baskets the same size had sat 550 to 789 days each and closed green for $43 to $366.

Your set-up (time split, random-entry null, BH correction, a positive control) is what I would use too, so I would not suspect the method. On the entry side my numbers agree with yours.

I tested the two most popular M1 gold scalping ideas on XAUUSD data since 2018, about 1,000 trades each: (1) the M1 close through a session high or low, with the H1 trend; (2) the sweep of a session high or low (wick through, close back inside), with the H1 trend. Before cost both came out at a profit factor of about 1.00. After the real spread both lose, because an M1 stop is only a few times the spread. That answers your question 1 for me: at that stop size it is structurally hard, whatever the signal.

What held up in my tests was slower: breakouts on M15 in the direction of the H1 trend, during London hours. It is positive in some years and weak in others (range-bound years are the problem), so I treat it as a filter, not as a proven edge. For your question 6 I would stop searching for M1-M5 direction and move to a timeframe where the cost is a smaller share of the stop.

On question 3: the one thing that clearly helped was not the entry. It was sizing the lot from the stop distance and a daily loss lock. It creates no edge, but it stops one bad day from erasing the sample you are trying to measure.

I agree with Ayaz on sample size: 60 trades cannot tell you anything about direction.

Mohamed, it starts from the real minute bars of the same market. Each bar is mirrored around its open with probability one half and every gap between two bars gets a random sign, all in log space so prices stay positive. The sign is a fresh draw for each minute and has nothing to do with what came before, so the next bar's direction is a coin flip by construction and no rule built on past prices can predict it, however many variants you search. What stays real is the size of every bar, so volatility clustering, the intraday volatility profile and the distribution of ranges are the ones your stops and trails meet in the real market. That is why management can be tested on it at all. As a check I ran 379 tests on the series and 15 came out beyond t 2, where chance alone gives about 17. One limit: because the magnitudes are kept, it only rules out directional patterns, so a rule that trades volatility itself is not tested by it.

Ayaz #:

From the reports, two things separated them.

One you've already seen: the win rate against the break-even rate its own average win and loss set. Two configurations tested over sixteen years needed about 80% and won 74%, and both went to zero. The account that was making money needed 88.50% and won 94.13%.

The other is what's still open at the end. The best run of a grid EA I tested finished at +262%, holding ten positions $21,765 under water that had been open 540 days. Before that, four baskets the same size had sat 550 to 789 days each and closed green for $43 to $366.

Hi Ayaz — your three points changed how we test (per-reviewer seed from the trade-list hash, the grid-fit check, and judging a system by its own break-even rather than the raw hit rate). We've used all three since.

One question, if you have a minute.

We measured a public, verified gold account: 94% of its net profit comes from a 5-hour entry window, and the concentration survives removing the ten largest winners (permutation p ≈ 0.002).

But it's that trader's own entries — not a sample of the hour. So the result could just as easily say "he chooses to trade then" as "that hour carries an effect."

How would you separate the two? Specifically: what test would convince you a time-of-day effect is real and not selection, when your only sample is one trader's own fills?

Happy to send the numbers if useful.

Andrea Michelon #:

Your set-up (time split, random-entry null, BH correction, a positive control) is what I would use too, so I would not suspect the method. On the entry side my numbers agree with yours.

I tested the two most popular M1 gold scalping ideas on XAUUSD data since 2018, about 1,000 trades each: (1) the M1 close through a session high or low, with the H1 trend; (2) the sweep of a session high or low (wick through, close back inside), with the H1 trend. Before cost both came out at a profit factor of about 1.00. After the real spread both lose, because an M1 stop is only a few times the spread. That answers your question 1 for me: at that stop size it is structurally hard, whatever the signal.

What held up in my tests was slower: breakouts on M15 in the direction of the H1 trend, during London hours. It is positive in some years and weak in others (range-bound years are the problem), so I treat it as a filter, not as a proven edge. For your question 6 I would stop searching for M1-M5 direction and move to a timeframe where the cost is a smaller share of the stop.

On question 3: the one thing that clearly helped was not the entry. It was sizing the lot from the stop distance and a daily loss lock. It creates no edge, but it stops one bad day from erasing the sample you are trying to measure.

I agree with Ayaz on sample size: 60 trades cannot tell you anything about direction.

Andrea — you are the only person who reproduced our result on another broker's data, and you did it for free. Saying that publicly is the least we owe you.
Your test settled something our own work did not. Our measured round-trip costs are 0.38 bp on gold, 0.40 on US30, 0.55 on NAS100, and our live book is 60 trades at -7.08R. We had been reading that as a signal problem. You read it as a stop-size problem, and your 8 years at ~1,000 trades per idea say you are right.
So we are not switching timeframe on advice, including yours. We will measure, on our own data, for M1 / M5 / M15 / H1: cost as a share of the stop distance, and trades per year. The rule is fixed before the measurement — the winner is the lowest cost-to-stop ratio that still clears 104 events per year, our floor for telling an effect from noise. If nothing clears both, the problem is the instrument or the broker, and we will say so.

Two questions.
1. Those same two numbers from your side, for the M15 breakout with the H1 trend during London: trades per year, and spread as a share of the stop. Your numbers against ours is a stronger test than either alone, and it is the one thing we cannot compute for you.
2. You call it a filter rather than an edge because of the range-bound years. Did you ever fix the parameters on one half of the sample and test them untouched on the other half? We killed one of our own findings that way last week — p 0.0018 on five months, p 0.86 out of sample on 2.6 years. If you have not, we will run it on our data and post the result either way.
123
New comment