What auditing 10 best-selling scalping/martingale EAs on the MQL5 Market taught me - page 2

New comment
Chris K. #:
Great thread, and very close to how I ended up looking at EAs, after losing roughly $30,000 on bought EAs and prop challenges.

Habedere! Griaß di aus München!

I think many people could benefit from hearing about your experience, as there’s often a lot to learn from other people’s experiences and mistakes. As the saying goes, a wise person learns from the mistakes of others. If you’re comfortable sharing, would you mind telling us a little about your experience with EAs, what led to the losses you experienced, and what lessons you’ve taken away from it? I think your insights could be genuinely valuable to others and might help them avoid making similar mistakes. Ty.

Ghenadie Tumco #:

Habedere! Griaß di aus München!

I think many people could benefit from hearing about your experience, as there’s often a lot to learn from other people’s experiences and mistakes. As the saying goes, a wise person learns from the mistakes of others. If you’re comfortable sharing, would you mind telling us a little about your experience with EAs, what led to the losses you experienced, and what lessons you’ve taken away from it? I think your insights could be genuinely valuable to others and might help them avoid making similar mistakes. Ty.

Servus Ghenadie, thanks, happy to share.

The money didn't go in one big loss. It went bit by bit since 2021: prop challenges (manual first, later with EAs),
bought EAs, coaching for manual trading, and a grid system from an online course that I coded myself and that
blew a real account. So "lost" is a bit loose, a lot of it was fees and products.

The thing that hurt most: I did test. I ran lots of backtests and they always looked great. I just checked the wrong thing. A backtest doesn't lie, it leaves things out. The EAs with the best-looking backtests were usually the ones built to look good in the backtest. That's why I now compare backtest and live signal over the same period first.

Second lesson came from a prop challenge. EA running, phase 1 passed after a long time. In phase 2 the balance was already at the target, but open trades were still in drawdown. Then one bad day broke the daily loss limit and it was over. Balance said "done", equity said "not yet". Since then I watch equity and open drawdown, not the balance curve.

Third: an EA that's fine on a normal account isn't automatically fine under prop rules. I had to cut the risk
so far that the challenge took ages, which also means more time for that one bad day to show up.

What finally worked: I quit prop firms and went back to my own money. Started again with a small account and
more patience. And once I saw something actually working, I didn't buy the next EA. I asked myself how to
scale that one edge instead.

Hope this helps :)
Chris K. #:

bought EAs, coaching for manual trading, and a grid system from an online course that I coded myself and that
blew a real account. So "lost" is a bit loose, a lot of it was fees and products.

The thing that hurt most: I did test. I ran lots of backtests and they always looked great. I just checked the wrong thing. A backtest doesn't lie, it leaves things out. The EAs with the best-looking backtests were usually the ones built to look good in the backtest. That's why I now compare backtest and live signal over the same period first.

I’m not sure if I understood correctly. Was the mistake that you were trading without a reasonable Stop Loss?

Ghenadie Tumco #:

I’m not sure if I understood correctly. Was the mistake that you were trading without a reasonable Stop Loss?

Hi Ghenadie,

No, the EAs had stop-losses that were set according to the prop firm’s rules. Those were two separate issues.

The backtest involved EAs that simply didn’t work in live trading. They were optimized to look good on historical data, but on a real-money account, the picture was completely different.

The Prop Challenge ended differently. The market opened with a huge price gap, far beyond the stop, so the order couldn’t
be executed there. The positions were closed immediately at the opening, but at a significantly worse price than the stop, and that single loss
exceeded the daily limit.

That showed me that a stop-loss is a plan, not a guarantee, and that you never have control at prop firms. Now I also look at what’s held beyond the closing price and what
impact such a price gap would have on the account.

Best regards,
Chris
Chris K. #:

I understand now. You probably should have known that stop losses can be executed differently after gaps between trading sessions. And, in general, it’s better not to leave positions open at the end of a trading session.

That brings me to another important issue: backtesting and how the results are analyzed. Apart from being out-of-sample, backtests should also be performed using the highest possible quality of historical data — meaning 100% tick quality. I can guarantee that the difference between a 100% quality backtest and a 99% quality backtest can be enormous.

As for the MT4 tester, I don’t think it’s even worth discussing. That tester is simply not suitable for anything serious or realistic. I’m talking specifically about MT5 here.

Regarding the EAs you selected, based on the backtests, I honestly didn’t see a single one that I would consider particularly impressive. And I’m not even talking about out-of-sample performance here.

What I mean is that even if you deliberately try to make an EA look as good as possible over a specific historical period through optimization essentially allowing significant curve fitting it is still surprisingly difficult to achieve truly impressive results.

I would actually like to see some really impressive MT5 backtests using real ticks with 100% history quality, even if they are intentionally curve-fitted. That could even make for an interesting competition: let’s see just how far we can push curve fitting. I have a feeling we would still struggle to find anything truly spectacular. :)

Ghenadie Tumco #:

I understand now. You probably should have known that stop losses can be executed differently after gaps between trading sessions. And, in general, it’s better not to leave positions open at the end of a trading session.

That brings me to another important issue: backtesting and how the results are analyzed. Apart from being out-of-sample, backtests should also be performed using the highest possible quality of historical data — meaning 100% tick quality. I can guarantee that the difference between a 100% quality backtest and a 99% quality backtest can be enormous.

As for the MT4 tester, I don’t think it’s even worth discussing. That tester is simply not suitable for anything serious or realistic. I’m talking specifically about MT5 here.

Regarding the EAs you selected, based on the backtests, I honestly didn’t see a single one that I would consider particularly impressive. And I’m not even talking about out-of-sample performance here.

What I mean is that even if you deliberately try to make an EA look as good as possible over a specific historical period through optimization essentially allowing significant curve fitting it is still surprisingly difficult to achieve truly impressive results.

I would actually like to see some really impressive MT5 backtests using real ticks with 100% history quality, even if they are intentionally curve-fitted. That could even make for an interesting competition: let’s see just how far we can push curve fitting. I have a feeling we would still struggle to find anything truly spectacular. :)

Hi Ghenadie,

fair point, I should have known. I learned it the expensive way.

On tick quality I fully agree, and from the buyer side it's one of the first things I´d check now: does the seller's report show "every tick based on real ticks" and the history quality?
If that part is cut off the screenshot, I ask for it.


Not sure which EAs you mean, I haven't named any here. But on your last point: in my experience the backtests that sell aren't the spectacular ones, they're the smooth ones.
A straight line with a tiny drawdown looks safer than a big number, and that's exactly what fitting is good at.


Count me in for the competition, as a spectator :)

Best regards
Chris

Chris K. #:


The responsibility for properly evaluating an EA and running high quality backtests probably falls on the buyer. For their own benefit, buyers need to learn how to verify what they are purchasing. Otherwise, they may end up paying for it through financial losses.

I haven’t mentioned any specific EA, nor do I intend to. What I said is that, out of all the EAs I’ve tested, I haven’t found anything that I consider satisfactory or truly impressive according to my own criteria, and I want to emphasize that point.

My criteria aren’t based on individual parameters considered separately. They are all components of a formula that produces a single index, with significant weight assigned to the average number of trades per day.

Everyone has their own criteria. You have yours, and smoothness is one of them.

And this is where the discussion becomes practical: what does a satisfactory backtest actually look like for each of us?

What does yours look like?

To check a backtest against live results, compare distributions rather than the headline numbers. Export both trade lists and compare, for each, the average win and loss, win rate, trades per week and the loss streak distribution. Then re-run the backtest on real ticks over the exact live period and compare trade by trade, matching on entry time. Trades that exist in only one list point to a data or logic difference, and a systematic gap in entry or exit price is execution cost. For a scalper, spread and slippage are often the whole edge, so log each live fill's price versus the requested price and add that to the tester with a spread and delay model. To judge whether results are statistically real, bootstrap the per-trade results: resample with replacement a few thousand times and look at the spread of profit factor and maximum drawdown. If the live result sits far outside that range, either the strategy has changed or the backtest was not representative. Be wary of martingale-type equity curves, where many small wins hide a single large tail loss that a short sample will not contain.
Ghenadie Tumco #:

The responsibility for properly evaluating an EA and running high quality backtests probably falls on the buyer. For their own benefit, buyers need to learn how to verify what they are purchasing. Otherwise, they may end up paying for it through financial losses.

I haven’t mentioned any specific EA, nor do I intend to. What I said is that, out of all the EAs I’ve tested, I haven’t found anything that I consider satisfactory or truly impressive according to my own criteria, and I want to emphasize that point.

My criteria aren’t based on individual parameters considered separately. They are all components of a formula that produces a single index, with significant weight assigned to the average number of trades per day.

Everyone has their own criteria. You have yours, and smoothness is one of them.

And this is where the discussion becomes practical: what does a satisfactory backtest actually look like for each of us?

What does yours look like?


Good morning Ghenadie,

agreed, in the end it's on the buyer. That's pretty much why I started writing these things down.

One small correction: smoothness isn't a plus for me. A very smooth curve is what makes me suspicious, because that's what fitting is best at.
What I want to see is real ticks over many years, with different market phases in it, including the bad ones. Enough trades that a handful of lucky ones don't carry the result.
Some ugly months, so it looks like trading and not like a ruler. And a drawdown and flat stretch I could actually sit through
But for me the backtest isn't the decision, it's the expectation. Then the EA runs small on a live account and has to stay inside that range.


Your index sounds interesting. Why so much weight on trades per day? Is it about getting enough trades to trust the stats, or something else?

Best regards
Chris

Chris K. #:

Your index sounds interesting. Why so much weight on trades per day? Is it about getting enough trades to trust the stats, or something else?

Yes. And in general, if an EA shows very few trades with a high profit factor in the backtest, that usually suggests the EA has been heavily optimized and only trades when the market conditions closely match the settings it was optimized for.

When an EA is able to open trades under a wide range of market conditions, meaning it trades almost every day, and still remains profitable, that is a much more interesting sign to me. It suggests that the EA is adapting to the market rather than waiting for the market to match its predefined settings.

123
New comment