My EA's kill switch tripped after 15 cycles — should the backtest stop too?

New comment
 
Ran a frozen first version of a system on BTCUSDT perpetual. Still a Development run, not Validation. The halt happened during the first development segment, 1 Oct to 16 Nov 2020 UTC — later periods haven't been touched yet.

Before running it I'd set two rules: halt if cumulative P&L is negative after 15 completed cycles, and don't evaluate anything below 30 cycles. First rule fired before the second one could be satisfied.
  • Cycles completed: 15
  • Net result: -43.02 USDT
  • Return on 300 USDT: -14.34%
  • Profit factor: 0.473
  • Max drawdown: 23.38%
  • Win rate: 46.67%
Not just fees eating a small edge either — before modeled fees it was already down 34.96 USDT. The seven cycles closed by the trailing stop averaged +5.52 USDT. The eight closed by Layer 2 (a conditional exit that closes a losing position when at least two of three adverse conditions agree) averaged -10.20 USDT.

Here's the annoying part: as a capital-protection mechanism, the kill switch did exactly what it was supposed to. As an experiment, it cut things off before the 30-cycle minimum I'd set. So I can't honestly call this a pass or a fail — official result is NOT_EVALUABLE.

Questions for anyone who's dealt with this:
  1. Should a kill-switch rule be respected during backtesting too, even if it means never reaching an evaluable sample?
  2. Would it make sense to stop the official run but keep logging signals and counterfactual fills separately, in a shadow ledger?
  3. If I do that, does the shadow result count toward evaluating the strategy, or is it diagnostic only?
  4. How do you separate a rule meant to protect real capital from a rule meant to reach a predeclared minimum sample?
Leaning toward keeping the kill switch as official behavior and running the shadow continuation purely as a diagnostic — clearly labeled, not used to rewrite the original result.

 

I run the same kind of rule on a live account, so my answer is that the kill switch belongs in the backtest. Switch it off and you are measuring a system you would never trade, and the number comes back for a different strategy.

Fifteen cycles says nothing either way. My own book has 66,511 trades over 11.4 years and only 46 percent of days are green, so a run of 15 sits well inside normal noise. The reason I keep the rule in the test at all is that the same six setups made +5,243 R over eleven years and still spent 2015 to 2017 inside a -449 R drawdown, and what a halt rule does in the bad regime is most of what it does.

The shadow ledger is worth building, I would just keep its numbers out of the evaluation and use them for diagnosis. A capital rule is a trading rule and has to be simulated. A minimum sample size is a reporting rule about when you are allowed to draw a conclusion. If the switch trips before the sample fills, the honest result is that the system did not survive long enough to be judged, and that is also a result.

 
The problem with a full-off kill switch is that it's rather retroactive in nature─most of the damage is already done by the time that the switch activates. I have developed a mitigating compromise of sorts. I use the average consecutive loss count in my Tester Report to minimize my position size when my "real-time" consecutive loss count exceeds the average consecutive loss count; and to increase my position size back to "normal" when one of those minimal trades closes in profit. In this way, the EA is never killed. The complete plain language logic is posted on my Profile wall.
 
Nice summary of the testing workflow. Following this thread with interest.
 
Ryan L Johnson #:
The problem with a full-off kill switch is that it's rather retroactive in nature─most of the damage is already done by the time that the switch activates. I have developed a mitigating compromise of sorts. I use the average consecutive loss count in my Tester Report to minimize my position size when my "real-time" consecutive loss count exceeds the average consecutive loss count; and to increase my position size back to "normal" when one of those minimal trades closes in profit. In this way, the EA is never killed. The complete plain language logic is posted on my Profile wall.
The retroactive part is fair and I would not defend a hard stop as a forecast. What I would question is the trigger. I checked the autocorrelation of my own daily series and it is -0.02, and the probability of a chop day given a chop day before it is 31 percent, which is exactly the base rate. A loss streak is not a state that predicts the next trade, so sizing down on it buys variance rather than expectancy. There is also the detail that your threshold comes out of the same Tester report you are evaluating, which makes it one more parameter fitted on the run it then judges.

The one regime filter that did carry information in my data still cost expectancy. Half size in the low volatility tertile cut my drawdown by about 30 percent and the total return by 9 percent. I run something like it, so this is not an objection, I just file it under insurance rather than edge. The reason I keep the hard stop as well is that on a funded account the limit is not mine to optimise. It is an external rule, and being right on average does not help once the account is closed.
 
During backtesting/dev, you shouldn't actually halt the run. 15 cycles is way too small and basically just noise, so killing the test early guarantees you'll never get a valid sample size. Let the run finish its full course without stopping—your "shadow ledger" is actually your real baseline test. Just flag the exact cycle where the kill switch would have triggered, then see what happens after: did cutting it off actually protect capital, or did it just lock in a normal drawdown right before a recovery? Also, keep an eye on that Layer 2 exit—losing -10 USDT on average while trailing wins only bring in +5.5 USDT means your risk/reward is heavily skewed against you right from the start.
New comment