Discussion of article "Statistical Distributions in MQL5 - taking the best of R" - page 4

New comment
 
fxsaber:

What the fuck?! The goals of the math bible are clear.

Find a robust mathematical model in R. They found it, so they investigated it in R. And then you can easily port a ready-made researched solution (a trading algorithm in fact) from R to MQL at the expense of bibla and this article. All kinds of machine learning is always a fit for financial markets.
Writing in MQl is a pain in the arse.)) It's easier (and more reliable)) to call the required functionality directly from R (and so on)). Renat mentioned these methods - they are not the simplest and most modern, but they exist. For example, how to attach COM objects to MT? Partially you can (not all) - (pardon me) through gaskets))).
 
СанСаныч Фоменко:

I don't understand something

What is the result of calling the specified function in MQL? A scalar? A vector?

Here is what we have on R

n <- 2000

>

> k <- seq(0, n, by = 20)

>

> dbinom(k, n, pi/10, log = TRUE)

[1] -754.219687 -660.247472 -592.126636 -534.532344 -483.881605

[6] -438.460449 -397.256529 -359.600217 -325.015561 -293.146935

[11] -263.718651 -236.510862 -211.344286 -188.070044 -166.562645

[16] -146.714976 -128.434635 -111.641185 -96.264050 -82.240889

[21] -69.516303 -58.040813 -47.770020 -38.663934 -30.686405

[26] -23.804662 -17.988917 -13.212041 -9.449276 -6.678001

[31] -4.877524 -4.028903 -4.114796 -5.119322 -7.027950

> a<-dbinom(k, n, pi/10, log = TRUE)

> str(a)

num [1:101] -754 -660 -592 -535 -484 ...

I.e., calling the function in R yields a vector that can be drawn using the universal method plot

> plot(a)

Put the updated file in /Include/Math/Stat, please.

It adds an overloaded function for the vector + an additional log_mode parameter.

Here is an analogue of your calculations, completely matching the results with R:

#include <Math/Stat/Binomial.mqh>

void OnStart(void)
  {
   double    vars[101];
   double    results[101];
   const int N=2000;
//--- 
   for(int i=0;i<=N;i+=20)
      vars[i/20]=i;

   MathProbabilityDensityBinomial(vars,N,M_PI/10,true,results);
      
   for(int i=0;i<ArraySize(results);i++)
      Print(results[i]);
//---
  }

Conclusion:

-754.2196869142003
-660.2474717208812
-592.1266359491199
-534.5323437511825 
...

We are now working on adding vector operations to be as close as possible in syntax to R.

Of course, we will add a lot of new unit tests to the delivery.

Files:
Binomial.mqh  19 kb
 

By the way, statements about full speed optimisation of mathematical libraries in R are greatly exaggerated.

This is the result of writing code head-on and wild system overhead due to work with dynamic objects/entities. Unlike R, MQL5 operates with clearly typed arrays and compiles code in native x64. This gives a huge advantage and allows you to win even against DLL implementations of functions in R.


Here is a test based on the example above: R 3.3.1 x64 

install.packages("microbenchmark")

library(microbenchmark)
n <- 2000
k <- seq(0,n,by=20)
res <- microbenchmark(a<-dbinom(k, n, pi/10, log = TRUE))
print(res)



Unit: microseconds
                                 expr    min     lq     mean median    uq    max neval
 a <- dbinom(k, n, pi/10, log = TRUE) 20.167 20.168 20.88726 20.469 20.77 35.819   100

triple run showed a minimum time of 20 microseconds.

The same test in MQL5:

#include <Math/Stat/Binomial.mqh>

void OnStart(void)
  {
   double    vars[101];
   double    results[101];
   const int N=2000;
//--- 
   for(int i=0;i<=N;i+=20)
      vars[i/20]=i;

   ulong msc=GetMicrosecondCount();
   MathProbabilityDensityBinomial(vars,N,M_PI/10,true,results);
   msc=GetMicrosecondCount()-msc;
      
   Print("Time: ",msc," msc");
//---
  }


2016.10.10 21:21:12.156 new (Si-12.16,H1)       Time: 10 msc

MQL5 counts twice as fast: 10 microseconds versus 20 microseconds in R.

Calculated on i7-4930k, Windows 10 x64, MetaTrader 5 x64, no virtualisations and no external load.

 
Renat Fatkhullin:

By the way, statements about full speed optimisation of mathematical libraries in R are greatly exaggerated.

This is the result of writing code head-on and wild system overhead due to work with dynamic objects/entities. Unlike R, MQL5 operates with clearly typed arrays and compiles code in native x64. This gives a huge advantage and allows you to win even against DLL implementations of functions in R.


Here is a test based on the example above: R 3.3.1 x64

triple run showed a minimum time of 20 microseconds.

The same test in MQL5:

MQL5 counts twice as fast: 10 microseconds versus 20 microseconds in R.

Calculated on i7-4930k, Windows 10 x64, MetaTrader 5 x64, no virtualisations and no external load.

Congratulations!!! A clean win, on both blades.
 
Renat Fatkhullin:

By the way, statements about full speed optimisation of mathematical libraries in R are greatly exaggerated.

This is the result of writing code head-on and wild system overhead due to work with dynamic objects/entities. Unlike R, MQL5 operates with clearly typed arrays and compiles code in native x64. This gives a huge advantage and allows you to win even against DLL implementations of functions in R.


Here is a test based on the example above: R 3.3.1 x64

triple run showed a minimum time of 20 microseconds.

The same test in MQL5:

MQL5 counts twice as fast: 10 microseconds versus 20 microseconds in R.

Calculation on i7-4930k, Windows 10 x64, MetaTrader 5 x64, no virtualisations and no external load.

and I got an error((

'vars' - parameter conversion not allowed

 
ivanivan_11:

and I got an error((

'vars' - parameter conversion not allowed

You need to take the updated file from https://www.mql5.com/ru/forum/97153/page4#comment_2882502.

Overridden function for vectors has been added there.

 
Renat Fatkhullin:

You need to take the updated file from https://www.mql5.com/ru/forum/97153/page4#comment_2882502

There is a redefined function for vectors.

Yes, thank you. it works like that. it worked for me like this. i have to tell you right away that i run MT from under vyne, 32bit. that's why i guess what the answer will be - 64bit is faster))) to avoid any confusion - R is also 32bit.



 
ivanivan_11:

Yes, thank you. it works like that. it worked for me like this. I have to say that I am running MT from under wyne, 32bit. so what will be the answer, I can guess - 64bit is faster))))

That's not serious.

Under wyne, and even 32 bits. For 32 bits MT5 uses an old compiler similar to MT4. It is a couple of tens of times slower than 64-bit code which is created by a new compiler specially for the 64-bit version of MT5. We have published a report comparing the speed of 32 and 64 bit MQL5 programmes.

Each MQL5 programme actually contains two copies of the compiled code: 32-bit for old systems for the sake of compatibility and 64-bit for new ones.

We have completely abandoned the development of 32-bit compilers as it has no sense and no future.

That's why you should use only 64-bit versions of MetaTrader 5 if you want to get the maximum performance.


By the way, information for those who sit on MT4 - there too the speed of MQL4 code execution is a couple of dozens of times slower than in 64-bit MetaTrader 5.

 
Andrey Dik:
Congratulations! A clean win, on both blades.

calculation is twice as fast as similar R calculation with compilation (first pass of R is byte/or jit compilation, so it has max>1.5 avg and min~=avg) ?

well, achievement... :-)

 
Maxim Kuznetsov:

calculation twice as fast as the same R calculation with compilation (the first pass of R is byte/or jit compilation, so it has max>1.5 avg and min~=avg) ?

well, achievement... :-)

R took the minimum time over many runs, not the first run and the maximum time.

So no, "cold start" doesn't explain it here. And even in an environment where all the library code is in a DLL. There is practically nothing to warm up.

But what does explain it:

  1. Absence of overhead on dynamic objects as in R due to work with purely typed double data
  2. A terribly efficient compiler in MQL5. It is really scary to look at the resulting asm code.
  3. The presence of MQL5 libraries in the source code, which allows you to natively inline their code directly into the place of call and maximise the use of const environment of the called place.

    This is where the compiler has a lot of room compared to actually unoptimised implementations in DLLs, which have no chance to learn the environment and adjust to it.

1234567891011...20
New comment