How do you check if a live EA still matches its backtest?
It's basically impossible to discern the specific differences between 2 black box utilities. At the end of the day, you don't have the source code of the Product, and you don't even have the EA of the Signal. Additionally, a 72 week bactest versus 7 weeks of signals is an apples to oranges comparison.
What we can discern is the fact that there are about 10 x the number of signal weeks in the backtest, but only about 4 x the number of signal trades in the backtest.
The wide array of different OTC broker-dealer feeds, and the wide array of potential input parameter settings, only add to the futility of attempting to discern differences.
The moral of the story is... Don't trade black box utilities.
It's basically impossible to discern the specific differences between 2 black box utilities. At the end of the day, you don't have the source code of the Product, and you don't even have the EA of the Signal. Additionally, a 72 week bactest versus 7 weeks of signals is an apples to oranges comparison.
What we can discern is the fact that there are about 10 x the number of signal weeks in the backtest, but only about 4 x the number of signal trades in the backtest.
The wide array of different OTC broker-dealer feeds, and the wide array of potential input parameter settings, only add to the futility of attempting to discern differences.
The moral of the story is... Don't trade black box utilities.
Hi Chris,
I'd separate two checks: whether the EA is behaving as intended, and whether its performance remains acceptable.
For the first, compare the same dates, version and inputs, then investigate the first unmatched trade. A missing trade alone doesn't prove different logic: execution differences can change which positions remain open and affect later entries. Where logs are available, check signal time, spread filters, trade requests and rejection codes.
For performance, I'd define the review criteria and account risk limit before running live. Exceeding historical drawdown deserves investigation, but the backtest maximum isn't a guaranteed ceiling. A short losing period alone also doesn't establish that the strategy has stopped working.
-
Very few EAs run with their default settings. Read the comments, you’ll see that most buyers ask for the settings after purchasing.
-
Not all signals presented for an EA are identified by MetaQuotes as 100% algorithmic trading.
-
Many EAs have trading-hour filters or news filters. The latter do not work in backtests because historical price data does not include economic calendar data.
At most, you can compare entries and exits and only if the live trades also appear in the backtest. Drawdown, lot size, etc., are irrelevant.
You’re comparing a train to a ship.
Hi Chris,
Good question, I've been working on the same thing. Same weeks, same broker's data, trade lists side by side is how I do it too.
For the line, I set it before the EA goes live and write it down. I pull three numbers from the backtest: the worst drawdown, the longest time it stayed below its last equity high, and the biggest single loss. If live goes past any one of those on the same broker's data, I pause it and dig in. Inside those limits I treat it as normal noise.
One thing I check first now is the broker's data. I ran the same EA with the same settings on a few brokers' data and one pair's worst drawdown was more than 20x bigger on one feed than on another. Swap was way different too. So I always rerun the test on the broker I'm actually trading before I decide the EA changed.
How are you setting yours?
@Mark Ayoub
the three numbers are a simple way to write the line down. The longest time below the last equity high is one I havent used so far, and I think its the one most people ignore. An EA can look fine on drawdown and still be under water for months. The broker's data I already use, but 20x on one pair is more than I would have guessed.
this is the part that was missing for me. The worst drawdown of a long backtest is one extreme, and a few live weeks can't really be held against it. Cutting the backtest into windows as long as the live part and looking at the spread makes much more sense. Live inside the spread is noise, outside I take a closer look.
@Mark Ayoub
the three numbers are a simple way to write the line down. The longest time below the last equity high is one I havent used so far, and I think its the one most people ignore. An EA can look fine on drawdown and still be under water for months. The broker's data I already use, but 20x on one pair is more than I would have guessed.
this is the part that was missing for me. The worst drawdown of a long backtest is one extreme, and a few live weeks can't really be held against it. Cutting the backtest into windows as long as the live part and looking at the spread makes much more sense. Live inside the spread is noise, outside I take a closer look.
Good question. In my experience the matching step decides how early you see it.
I do roughly what you describe, with a few additions that made it catch problems sooner:
- Match on more than entry time. Use symbol, direction and entry time within a small window, and convert both sides to the same server time first. Tester and broker clocks are often an hour or two apart, and that alone produces fake "one-sided" trades. When several orders open in the same second, pair them in order.
- Split the gap per matched trade into its parts: entry price difference, exit price difference, commission and swap. Then keep a separate count of trades that exist on only one side. Costs drift slowly. Unmatched trades are the real alarm, and even one is worth a look.
- Look at entries and exits by order type. Market entries, pending entries, and SL/TP exits behave differently. A gap that only shows on market exits points to a different cause than one on stop entries.
- Group by magic number, not by account. An EA that fires 20 orders at once gets filled differently from one that sends a single order. Averaging them together hides both.
- Don't judge costs before about 30 matched trades. One-sided trades you can act on immediately. Slippage needs a sample.
Agreed on versions: a new build is a new EA. I also restart the count when the broker, account type or VPS changes, because fills change with them.
Good question. In my experience the matching step decides how early you see it.
I do roughly what you describe, with a few additions that made it catch problems sooner:
- Match on more than entry time. Use symbol, direction and entry time within a small window, and convert both sides to the same server time first. Tester and broker clocks are often an hour or two apart, and that alone produces fake "one-sided" trades. When several orders open in the same second, pair them in order.
- Split the gap per matched trade into its parts: entry price difference, exit price difference, commission and swap. Then keep a separate count of trades that exist on only one side. Costs drift slowly. Unmatched trades are the real alarm, and even one is worth a look.
- Look at entries and exits by order type. Market entries, pending entries, and SL/TP exits behave differently. A gap that only shows on market exits points to a different cause than one on stop entries.
- Group by magic number, not by account. An EA that fires 20 orders at once gets filled differently from one that sends a single order. Averaging them together hides both.
- Don't judge costs before about 30 matched trades. One-sided trades you can act on immediately. Slippage needs a sample.
Agreed on versions: a new build is a new EA. I also restart the count when the broker, account type or VPS changes, because fills change with them.
Example from an EA I looked at, not one of mine. Same EA, backtest vs the seller's own live signal.