Discussing the article: "MetaTrader 5 Machine Learning Blueprint (Part 12): Probability Calibration for Financial Machine Learning"
You are missing trading opportunities:
- Free trading apps
- Over 8,000 signals for copying
- Economic news for exploring financial markets
Registration
Log in
You agree to website policy and terms of use
If you do not have an account, please register
Check out the new article: MetaTrader 5 Machine Learning Blueprint (Part 12): Probability Calibration for Financial Machine Learning.
Tree-based classifiers are typically overconfident: true win rates near 0.55 appear as 0.65–0.80 and inflate position sizes and Kelly fractions. This article presents afml.calibration and CalibratorCV, which generate out-of-fold predictions via PurgedKFold and fit isotonic regression or Platt scaling. We define Brier score, ECE, and MCE, and show diagnostics that trace miscalibration into position sizes, realized P&L, and CPCV path Sharpe distributions to support leakage-free, correctly sized trading.
A Random Forest that predicts a 0.68 probability when the true win rate is actually 0.55 is not just inaccurate. It is systematically overconfident, and that overconfidence flows straight into every sizing calculation that follows. The get_signal function maps the inflated probability to a larger z-score and therefore a larger position. The Kelly fraction grows with the predicted probability. Overbetting Kelly even slightly on every trade will eventually ruin a bankroll that would have survived the same trades with correct sizing.
The symptoms only show up in the equity curve. A strategy that looks well-sized in backtesting with raw probabilities will systematically take positions that are larger than the evidence actually supports. This produces deeper drawdowns and lower geometric growth than a correctly calibrated version of the same strategy.
In this article, we cover the afml.calibration module and its role in the pipeline. You will learn (1) why tree-based classifiers tend to be overconfident and how this appears in a reliability diagram; (2) what Brier score, ECE, and MCE measure; (3) when to use isotonic regression versus Platt scaling; (4) how to calibrate without temporal leakage using out-of-fold predictions from PurgedKFold; and (5) how miscalibration propagates from probabilities to position sizes, P&L, and CPCV path Sharpe distributions.
Author: Patrick Murimi Njoroge