ForumSections

How do you check if a live EA still matches its backtest? - page 3

New comment
Christian Kotschote #:
Hi Eduardo,

I like the windows with the same number of trades as the live sample. That gets around the "one draw" problem without assuming independent trades, like the reshuffle does.

Losing bigger instead of more often i didnt have on my list. A streak counter stays quiet there, so the streak needs the size next to it, like you do with R.

How long does the backtest need to be for this? With 100 live trades and 1,000 backtest trades i'd only get 10 windows that don't overlap, and the worst 5% of 10 is basically the worst one again. Do you let the windows overlap?

Best regards,
Chris

Thanks Christian.

It depends on the size of the live sample. If I want to judge the EA after 100 live trades, I’d like at least 20 to 30 separate windows of 100, so 2,000 to 3,000 trades in the backtest, and they should cover different market conditions, not just a lot of trades from one quiet year.

With less history I let the windows overlap and move them forward 10 or 20 trades at a time. That gives many more windows, but they share most of their trades, so the worst 5% looks more stable than it really is. In that case I read the line as a rough guide and leave some extra room rather than switching off the moment it’s touched.

Eduardo


that's more history than i expected. An EA with about 2 trades a week makes round about 100 trades a year, so 2,000 trades would be 20 years of backtest. For those EAs the overlapping windows are the only option, and like you say the line gets softer.

Maybe thats a case for two lines instead of one. A first one where I only look closer or cut the lot size, and a second, wider one where the EA gets switched off.

Chris :)

Hi Chris,

On the window overlap question: instead of fixed non-overlapping windows, you can resample the backtest trade list a few thousand times (a block bootstrap, with blocks of a few trades so losing streaks stay together), always drawing as many trades as your live record has. That gives you a drawdown distribution for exactly your live sample size, and overlap stops being a problem because every sample is a fresh draw. With 100 live trades you can then read the 95th and 99th percentile directly.

It also fits your two-line idea well: the 95th percentile as the "look closer or cut the lot size" line, and the 99th percentile (or the worst backtest drawdown plus a margin) as the "switch it off" line. Writing both numbers down before going live helps avoid deciding emotionally in the middle of a drawdown.

One caveat: the bootstrap only tells you whether the result is plausible bad luck. It cannot tell you if the market changed, so I would still compare the trade lists as you described.

Best regards, César 

Cesar De Los Reyes Sanchez #:

Hi Chris,

On the window overlap question: instead of fixed non-overlapping windows, you can resample the backtest trade list a few thousand times (a block bootstrap, with blocks of a few trades so losing streaks stay together), always drawing as many trades as your live record has. That gives you a drawdown distribution for exactly your live sample size, and overlap stops being a problem because every sample is a fresh draw. With 100 live trades you can then read the 95th and 99th percentile directly.

It also fits your two-line idea well: the 95th percentile as the "look closer or cut the lot size" line, and the 99th percentile (or the worst backtest drawdown plus a margin) as the "switch it off" line. Writing both numbers down before going live helps avoid deciding emotionally in the middle of a drawdown.

One caveat: the bootstrap only tells you whether the result is plausible bad luck. It cannot tell you if the market changed, so I would still compare the trade lists as you described.

Best regards, César 

Good afternoon César,

the block bootstrap is a nice way around the overlap, i didn't think of drawing exactly as many trades as the live record has. And the 95th and 99th percentile as the two lines is more honest than my "a bit past the backtest max".

How long do you make the blocks? With a grid EA one losing streak can be 10 positions closed in the same hour, so I guess I'd have to count the whole basket as one trade first, otherwise the blocks cut it apart.

Best regards,
Chris
123
New comment