How do you check if a live EA still matches its backtest?

New comment
Hi everyone,

there were a few threads here lately about backtest vs live. Most ask why the two differ. I'd like to ask something a bit different: how do you notice it early enough?

What I do with my EAs is pretty simple. I run the backtest over the same weeks as the live trades, on my broker's data, and put both trade lists next to each other by entry time. Same trades with slightly worse prices is costs, thats normal. Trades that show up only on one side mean the EA live is not the EA I tested.

I also only count from the last update. A new version is basically a new EA, so the clock starts again.
Example from an EA I looked at, not one of mine. Same EA, backtest vs the seller's own live signal.

backtest-vs-live

What I havent really solved for myself is the line. When is a difference still noise, and when do you switch the EA off? Drawdown above the backtest, a few weeks below it, something else?

Curious how you handle it.

Best regards
Chris
Chris K.:
[T]here were a few threads here lately about backtest vs live. Most ask why the two differ. I'd like to ask something a bit different: how do you notice it early enough?

It's basically impossible to discern the specific differences between 2 black box utilities. At the end of the day, you don't have the source code of the Product, and you don't even have the EA of the Signal. Additionally, a 72 week bactest versus 7 weeks of signals is an apples to oranges comparison.

What we can discern is the fact that there are about 10 x the number of signal weeks in the backtest, but only about 4 x the number of signal trades in the backtest.

The wide array of different OTC broker-dealer feeds, and the wide array of potential input parameter settings, only add to the futility of attempting to discern differences.

The moral of the story is... Don't trade black box utilities.

Ryan L Johnson #:

It's basically impossible to discern the specific differences between 2 black box utilities. At the end of the day, you don't have the source code of the Product, and you don't even have the EA of the Signal. Additionally, a 72 week bactest versus 7 weeks of signals is an apples to oranges comparison.

What we can discern is the fact that there are about 10 x the number of signal weeks in the backtest, but only about 4 x the number of signal trades in the backtest.

The wide array of different OTC broker-dealer feeds, and the wide array of potential input parameter settings, only add to the futility of attempting to discern differences.

The moral of the story is... Don't trade black box utilities.

Hi Ryan,

you're right, 72 weeks against 7 is not a fair comparison. Thats exactly why I don't do it like that with my own EAs, I run the backtest only over the same weeks as live. The picture is how it usually looks when you buy one, the seller puts the whole backtest next to a short signal.

But the number you found is the interesting part. 10x the weeks and only 4x the trades means the live signal trades about 2.5 times more often per week than the backtest. Without any source code, thats already a reason to ask the seller which settings the signal runs on.

On black boxes I agree in theory. In practice most people here buy them anyway, so I'd rather check what comes out than pretend they won't.

Thanks for looking at it so closely.
Chris
New comment