Discussing the article: "MetaTrader 5 Machine Learning Blueprint (Part 20): Denoising, Detoning, and Clustering the Feature Correlation Matrix"
You are missing trading opportunities:
- Free trading apps
- Over 8,000 signals for copying
- Economic news for exploring financial markets
Registration
Log in
You agree to website policy and terms of use
If you do not have an account, please register
Check out the new article: MetaTrader 5 Machine Learning Blueprint (Part 20): Denoising, Detoning, and Clustering the Feature Correlation Matrix.
Raw feature correlations contain estimation noise and a shared market-mode component that distort clustering. We fit the Marcenko–Pastur noise ceiling (with an effective sample size correction), apply constant-residual denoising and market detonation, and run the Optimal Number of Clusters routine. The result is a cleaned correlation matrix and stable cluster labels that avoid substitution effects and feed clustered MDI/MDA in the next article.
A sample correlation matrix estimated from a finite window is never the true correlation matrix. Two distinct sources of contamination sit inside it, and clustering is sensitive to both.
The first is estimation noise. With N features and T observations, a matrix built from purely random, uncorrelated data still produces a spread of nonzero sample correlations and a spread of sample eigenvalues away from 1, purely from finite-sample variation. Random matrix theory gives the exact shape of that spread under the null of no correlation; anything inside it is indistinguishable from noise, and anything outside it is evidence of real structure. Section 7 below fits that shape to the feature panel used in this article.
The second is a shared market-mode component. Features derived from the same instrument's price path tend to share exposure to the prevailing regime: a trending, high-volatility stretch inflates trend, volatility, and several microstructure estimators simultaneously, independent of whatever family-specific signal each feature is meant to isolate. That shared component shows up as an unusually large first eigenvalue with roughly uniform loadings across the affected features. MLAM Section 2.6 makes the mechanical consequence explicit: clustering struggles to separate feature families when the correlation matrix carries a strong common tone, because every feature looks somewhat similar to every other feature through that shared channel.
Denoising addresses the first problem. Detoning addresses the second. Both are prerequisites for clustering that this article treats as a single pipeline, applied before any importance computation runs.
Author: Patrick Murimi Njoroge