Discussion of article "Statistical Distributions in MQL5 - taking the best of R"

[MetaQuotes]

New article Statistical Distributions in MQL5 - taking the best of R has been published:

The R language is one of the best tools of statistical processing and analysis of data. Thanks to availability and support of multiple statistical distributions, it had become widespread in the analysis and processing of various data. Using the apparatus of probability theory and mathematical statistics allows for a fresh look at the financial market data and provides new opportunities to create trading strategies. With the statistical library, all these features are now available in the MQL5.

Let us consider the functions for working with the basic statistical distributions implemented in the R language.

Those include the Cauchy, Weibull, normal, log-normal, logistic, exponential, uniform, gamma distributions, the central and noncentral beta, chi-squared, Fisher's F-distribution, Student's t-distribution, as well as the discrete binomial and negative binomial distributions, geometric, hypergeometric and Poisson distributions. In addition, there are functions for calculating theoretical moments of distributions, which allow to evaluate the degree of conformity of the real distribution to the modeled one.

The MQL5 standard library has been supplemented with numerous mathematical functions from R. Moreover, an increase in operation speed of 3 to 7 times has been achieved, compared to the initial versions in the R language. At the same time, errors in implementation of certain functions in R have been found.

Fig. 2. Distribution histogram of random numbers, generated according to the normal distribution with the parameters mu=5 and sigma=1

Author: MetaQuotes Software Corp.

[fxsaber]

Great guide, Thank you!

I would like to ask you to make the meta-editor in the meta-editor, despite the fact that they are connected via an includnik (written in MQL5), to make the sub-lighting in its own colour.

Now the article in the source code does not have this sub-lighting, so it's a bit hard to read/receive.

We are waiting for "Visualisation in MQL5 - taking the best from R".

[Maxim Kuznetsov]

MetaQuotes Software Corp.:

Published article Statistical Distributions in MQL5 - Taking the Best from R:

Author: MetaQuotes Software Corp.

The work deserves respect for its volume, but

- Testing statistical hypotheses is not the most speed-critical component in MQL products.

- The issues of accuracy loss remain open (it is not for nothing that mat.libraries grow strong for a long time and are valued like cognac, in terms of age).

[Vasiliy Sokolov]

Respect and respect to MQ!

[Реter Konow]

I can imagine the research opportunities that will open up when all these statistics and quality graphical visualisations are combined.

[ivanivan_11]

When the question of integrating R and MT https://www.mql5.com/ru/forum/73266/page10#comment_2283757 was raised half a year ago, for some reason it seemed that a full-fledged data exchange would be implemented. not a separate library for a narrow range of tasks.

and what is the inherent advantage of this library over the existing 4 years old version of alglib https://www.mql5.com/en/code/1146? and specifically the library

specialfunctions.mqh

Classes of distribution functions, integrals, polynomials: . CGammaFunc - Gamma function. CIncGammaF - incomplete Gamma-function. CBetaF - Beta function. CIncBetaF - incomplete Beta function. CPsiF - psi function. CAiryF - Airy functions. CBessel - Bessel functions of integer order. CJacobianElliptic - elliptic Jacobi functions. CDawson - Dawson integral. CTrigIntegrals - trigonometric integrals. CElliptic - elliptic integrals of the first and second kind. CExpIntegrals - exponential integrals. CFresnel - Fresnel integrals. CHermite - Hermite polynomials. CChebyshev - Chebyshev polynomials. CLaguerre - Laguerre polynomials. CLegendre - Lejandre polynomials. CChiSquareDistr - chi-square distribution. CBinomialDistr - binomial distribution. CNormalDistr - normal distribution. CPoissonDistr - Poisson distribution. CStudentDistr - Student's t-distribution. CFDistr - F-distribution.

[Quantum]

Maxim Kuznetsov:

labour deserves respect for its volume, but

- Stat hypothesis testing is not the most speed-critical component in MQL products.

- The issues of accuracy loss remain open (it is not for nothing that mat.libraries are strong for a long, long time and are valued like cognac, by age).

To check complex calculations, there are unit tests (scripts in the /Scripts/Unittests folder).

To assess the accuracy of the calculation of functions of the statistical library, you can compare them with the values obtained in Wolfram Alpha.

The TestStatPrecision.mql5 script calculates probability density functions (PDF) and cumulative distribution functions (CDF) for each of the library distributions.

The obtained results are compared with the values from Wolfram Alpha (presented to the nearest 30 digits) and the number of matching digits after the decimal point is displayed.

The results of the script are displayed in the "Experts" tab:

Testing precision for distribution:Beta
Distribution: Beta,  Wolfram PDF=1.250000000000000000000000000000,  PDF_calculated=1.249999999999998223643160599750,  deltaPDF=0.000000000000001776356839400250
Distribution: Beta,  Wolfram CDF=0.812500000000000000000000000000,  CDF_calculated=0.812500000000000222044604925031,  deltaCDF=-0.000000000000000222044604925031
Distribution: Beta PDF correct digits=14
Distribution: Beta CDF correct digits=15
   
Testing precision for distribution:Binomial
Distribution: Binomial,  Wolfram PDF=0.178863050569879750151258690494,  PDF_calculated=0.178863050569879888929136768638,  deltaPDF=-0.000000000000000138777878078145
Distribution: Binomial,  Wolfram CDF=0.416370829447481383134288535075,  CDF_calculated=0.416370829447481938245800847653,  deltaCDF=-0.000000000000000555111512312578
Distribution: Binomial PDF correct digits=15
Distribution: Binomial CDF correct digits=15
   
Testing precision for distribution:Cauchy
Distribution: Cauchy,  Wolfram PDF=0.078353202752933087671394218887,  PDF_calculated=0.078353202752933101549182026702,  deltaPDF=-0.000000000000000013877787807814
Distribution: Cauchy,  Wolfram CDF=0.165249340538567907055167438557,  CDF_calculated=0.165249340538567907055167438557,  deltaCDF=0.000000000000000000000000000000
Distribution: Cauchy PDF correct digits=16
Distribution: Cauchy CDF correct digits=30
   
Testing precision for distribution:ChiSquare
Distribution: ChiSquare,  Wolfram PDF=0.389400391535702439238519900755,  PDF_calculated=0.389400391535702439238519900755,  deltaPDF=0.000000000000000000000000000000
Distribution: ChiSquare,  Wolfram CDF=0.221199216928595121522960198490,  CDF_calculated=0.221199216928595121522960198490,  deltaCDF=0.000000000000000000000000000000
Distribution: ChiSquare PDF correct digits=30
Distribution: ChiSquare CDF correct digits=30
   
Testing precision for distribution:Exponential
Distribution: Exponential,  Wolfram PDF=0.441248451292297727555080655293,  PDF_calculated=0.441248451292297727555080655293,  deltaPDF=0.000000000000000000000000000000
Distribution: Exponential,  Wolfram CDF=0.117503097415404600400989920672,  CDF_calculated=0.117503097415404544889838689414,  deltaCDF=0.000000000000000055511151231258
Distribution: Exponential PDF correct digits=30
Distribution: Exponential CDF correct digits=16
   
Testing precision for distribution:F
Distribution: F,  Wolfram PDF=0.702331961591220799157042620209,  PDF_calculated=0.702331961591220910179345082724,  deltaPDF=-0.000000000000000111022302462516
Distribution: F,  Wolfram CDF=0.209876543209876531559388013193,  CDF_calculated=0.209876543209876587070539244451,  deltaCDF=-0.000000000000000055511151231258
Distribution: F PDF correct digits=15
Distribution: F CDF correct digits=16
   
Testing precision for distribution:Gamma
Distribution: Gamma,  Wolfram PDF=0.606530659712633424263117376540,  PDF_calculated=0.606530659712633424263117376540,  deltaPDF=0.000000000000000000000000000000
Distribution: Gamma,  Wolfram CDF=0.393469340287366575736882623460,  CDF_calculated=0.393469340287366575736882623460,  deltaCDF=0.000000000000000000000000000000
Distribution: Gamma PDF correct digits=30
Distribution: Gamma CDF correct digits=30
   
Testing precision for distribution:Geometric
Distribution: Geometric,  Wolfram PDF=0.050421000000000000540456568388,  PDF_calculated=0.050420999999999979723774856666,  deltaPDF=0.000000000000000020816681711722
Distribution: Geometric,  Wolfram CDF=0.882350999999999996425970039127,  CDF_calculated=0.882350999999999996425970039127,  deltaCDF=0.000000000000000000000000000000
Distribution: Geometric PDF correct digits=16
Distribution: Geometric CDF correct digits=30
   
Testing precision for distribution:Hypergeometric
Distribution: Hypergeometric,  Wolfram PDF=0.036675398904501069208272667765,  PDF_calculated=0.036675398904501069208272667765,  deltaPDF=0.000000000000000000000000000000
Distribution: Hypergeometric,  Wolfram CDF=0.996784948797332703840368139936,  CDF_calculated=0.996784948797332703840368139936,  deltaCDF=0.000000000000000000000000000000
Distribution: Hypergeometric PDF correct digits=30
Distribution: Hypergeometric CDF correct digits=30
   
Testing precision for distribution:Logistic
Distribution: Logistic,  Wolfram PDF=0.235003712201594494590750628049,  PDF_calculated=0.235003712201594494590750628049,  deltaPDF=0.000000000000000000000000000000
Distribution: Logistic,  Wolfram CDF=0.377540668798145462314863607389,  CDF_calculated=0.377540668798145406803712376131,  deltaCDF=0.000000000000000055511151231258
Distribution: Logistic PDF correct digits=30
Distribution: Logistic CDF correct digits=16
   
Testing precision for distribution:Lognormal
Distribution: Lognormal,  Wolfram PDF=0.000000247498055546993546655130,  PDF_calculated=0.000000247498055546993546655130,  deltaPDF=0.000000000000000000000000000000
Distribution: Lognormal,  Wolfram CDF=0.000000044817423501713188227213,  CDF_calculated=0.000000044817423501713168374878,  deltaCDF=0.000000000000000000000019852335
Distribution: Lognormal PDF correct digits=30
Distribution: Lognormal CDF correct digits=22
   
Testing precision for distribution:NegativeBinomial
Distribution: NegativeBinomial,  Wolfram PDF=0.046875000000000000000000000000,  PDF_calculated=0.046875000000000000000000000000,  deltaPDF=0.000000000000000000000000000000
Distribution: NegativeBinomial,  Wolfram CDF=0.937500000000000000000000000000,  CDF_calculated=0.937500000000000000000000000000,  deltaCDF=0.000000000000000000000000000000
Distribution: NegativeBinomial PDF correct digits=30
Distribution: NegativeBinomial CDF correct digits=30

Testing precision for distribution:NoncentralBeta
Distribution: NoncentralBeta,  Wolfram PDF=1.835315758284358889085297050769,  PDF_calculated=1.835315758284356890683852725488,  deltaPDF=0.000000000000001998401444325282
Distribution: NoncentralBeta,  Wolfram CDF=0.279804451879309967754494437031,  CDF_calculated=0.279804451879309523665284586968,  deltaCDF=0.000000000000000444089209850063
Distribution: NoncentralBeta PDF correct digits=14
Distribution: NoncentralBeta CDF correct digits=15
   
Testing precision for distribution:NoncentralChiSquare
Distribution: NoncentralChiSquare,  Wolfram PDF=0.266641691212769094132539748898,  PDF_calculated=0.266641691212769094132539748898,  deltaPDF=0.000000000000000000000000000000
Distribution: NoncentralChiSquare,  Wolfram CDF=0.142365913869366367272562001745,  CDF_calculated=0.142365913869366339516986386116,  deltaCDF=0.000000000000000027755575615629
Distribution: NoncentralChiSquare PDF correct digits=30
Distribution: NoncentralChiSquare CDF correct digits=16
   
Testing precision for distribution:NoncentralF
Distribution: NoncentralF,  Wolfram PDF=0.354683475208693754776589912581,  PDF_calculated=0.354683475208693865798892375096,  deltaPDF=-0.000000000000000111022302462516
Distribution: NoncentralF,  Wolfram CDF=0.090794346737526995805289686814,  CDF_calculated=0.090794346737526995805289686814,  deltaCDF=0.000000000000000000000000000000
Distribution: NoncentralF PDF correct digits=15
Distribution: NoncentralF CDF correct digits=30
   
Testing precision for distribution:Normal
Distribution: Normal,  Wolfram PDF=0.000013365598267338118769627896,  PDF_calculated=0.000013365598267338122157759685,  deltaPDF=-0.000000000000000000003388131789
Distribution: Normal,  Wolfram CDF=0.000015229981947977879768092203,  CDF_calculated=0.000015229981947977883156223992,  deltaCDF=-0.000000000000000000003388131789
Distribution: Normal PDF correct digits=20
Distribution: Normal CDF correct digits=20
   
Testing precision for distribution:Poisson
Distribution: Poisson,  Wolfram PDF=0.000000000000281323432020839554,  PDF_calculated=0.000000000000281323432020839908,  deltaPDF=-0.000000000000000000000000000353
Distribution: Poisson,  Wolfram CDF=0.999999999999981348253186297370,  CDF_calculated=0.999999999999981237230883834854,  deltaCDF=0.000000000000000111022302462516
Distribution: Poisson PDF correct digits=27
Distribution: Poisson CDF correct digits=15
   
Testing precision for distribution:Uniform
Distribution: Uniform,  Wolfram PDF=0.004000000000000000083266726847,  PDF_calculated=0.004000000000000000083266726847,  deltaPDF=0.000000000000000000000000000000
Distribution: Uniform,  Wolfram CDF=0.000500000000000000010408340856,  CDF_calculated=0.000500000000000000010408340856,  deltaCDF=0.000000000000000000000000000000
Distribution: Uniform PDF correct digits=30
Distribution: Uniform CDF correct digits=30
   
Testing precision for distribution:Weibull
Distribution: Weibull,  Wolfram PDF=0.019512185823866712297558478895,  PDF_calculated=0.019512185823866712297558478895,  deltaPDF=0.000000000000000000000000000000
Distribution: Weibull,  Wolfram CDF=0.000976085818024337737580653496,  CDF_calculated=0.000976085818024330365005880594,  deltaCDF=0.000000000000000007372574772901
Distribution: Weibull PDF correct digits=30
Distribution: Weibull CDF correct digits=17
   
Testing precision for distribution:T
Distribution: T,  Wolfram PDF=0.319904796224811438509760819215,  PDF_calculated=0.319904796224811494020912050473,  deltaPDF=-0.000000000000000055511151231258
Distribution: T,  Wolfram CDF=0.682299044355095474223560358951,  CDF_calculated=0.682299044355095474223560358951,  deltaCDF=0.000000000000000000000000000000
Distribution: T PDF correct digits=16
Distribution: T CDF correct digits=30
   
Testing precision for distribution:NoncentralT
Distribution: NoncentralT,  Wolfram PDF=0.000000000000040650786864501445,  PDF_calculated=0.000000000000040650786864501173,  deltaPDF=0.000000000000000000000000000271
Distribution: NoncentralT,  Wolfram CDF=0.000000000000004816980000000000,  CDF_calculated=0.000000000000004818163532209154,  deltaCDF=-0.000000000000000001183532209154
Distribution: NoncentralT PDF correct digits=27
Distribution: NoncentralT CDF correct digits=17   

The functions are calculated with good accuracy, which allows them to be used in statistical calculations.

[Quantum]

ivanivan_11:

and what is the inherent advantage of this library over the existing 4 years old version of alglib https://www.mql5.com/en/code/1146? and specifically the library.

It is closer to R in functionality. More distributions (21) + random number generators. There are functions for calculating theoretical moments of distributions.

[Maxim Kuznetsov]

Quantum:

To check complex calculations, there are unit tests (scripts in the /Scripts/Unittests folder).

The functions are calculated with good accuracy, which allows them to be used in statistical calculations.

I have had the "good fortune" to support a kind of software closely related to mathematics, so I am sceptical about any "new" implementation of known methods..Unittests are not a panacea, and errors will (I guarantee it) pop up at the most inopportune times.

PS/ if you develop the dialogue, then due to insufficiently enthusiastic attitude I will obviously go to the ban :-)

[Renat Fatkhullin]

The comment above shows accuracy comparisons with a benchmark, which was Wolfram Alpha with 30 decimal places detail.

We understand very well that such complex matters should be covered with tests as much as possible. That's why we have a special section /Scripts/Unittests, where we have collected several extensive tests of mathematical libraries functionality.

Please upgrade to the latest MT5 beta that we released yesterday and run these unit tests yourself.

[Реter Konow]

In my opinion, the potential of what has been available in MQL for a long time is far from being realised by traders in strategies and Expert Advisors, but many people demand more and more. Why are they constantly criticised when they get the desired innovations? Who needs such precision (up to 30 decimal places)? Strange whims... Statistics is a necessary thing, although I think that even without this functionality, you could solve all your trading tasks with MQL, just by having the desire to work. However, thanks for the addition to the developers.