A Machine Learning in Trading: The Tideverse and the book on R - General

СанСаныч Фоменко 2023.10.01 09:38 #32671

Maxim Kuznetsov #:

R is remarkable for its hodgepodge. At any given moment it has everything, any package for any occasion.

But after a year or two, it's inimitable - it will be impossible to execute the examples in the book.

Awell-structured system with an excellent reference apparatus created by a professional teamcannot be a mishmash. System R is oriented to trading, without distractions to everything and anything for the sake of universality and populism.

This is exactly what the book demonstrates.

And another thing that the book based on R does is to break the illusion that you can get a model based on MO in a hurry, without owning a large set of tools, and most importantly, without understanding why you need this or that tool (package, function), why you need this or that stage of model building, without understanding that you can't throw something away - everything will fall apart.

FOREX - Trends, forecasts Out of sporting interest, Is any prediction doomed?

СанСаныч Фоменко 2023.10.01 09:45 #32672

A perfect example of a complete lack of understanding of WHAT is being done is dozens of pages of correlations of something with something.

Before calculating correlation, one should answer the question: does correlation exist for the series used? For financial series - this is the first and the most important question, because for financial series correlation does not exist, because for them there is no mathematical expectation and it is necessary to prove that the used average CAN be used as a mathematical expectation.

By the way, in R it is to check once a sneeze, routine.

What am I doing Thoughts on some of Signs of a REAL

Maxim Dmitrievsky 2023.10.01 09:59 #32673

СанСаныч Фоменко #:

It's a wonderful book!

It must cover all the problems of the MoD.

for beginners, which you are for now.

for boys in general. The Tideverse.

fxsaber 2023.10.01 10:00 #32674

СанСаныч Фоменко #:

A perfect example of a complete lack of understanding of WHAT is being done is dozens of pages of correlations of something to something.

Since you do not understand, I will explain.

Forum on trading, automated trading systems and testing trading strategies

Machine Learning in Trading: Theory, Models, Practice and Algorithm Trading

fxsaber, 2023.10.01 09:38

void FillArray( double &Array[], const int Amount )
{
  for (uint i = ArrayResize(Array, Amount); (bool)i--;)
    Array[i] = MathRand();
}

MathRand is a random number. I.e. correlation is calculated on random matrices. The purpose is to make sure that different variants of algorithm implementations give the same result. The speed of algorithm execution and memory consumption are compared. If you are incompetent, stay out of it.

How to Start with MetaTrader 5 Python User Is there a pattern

fxsaber 2023.10.01 10:07 #32675

fxsaber #:

I have not found a standard one for line-by-line calculation. Alglib seemed slow. I am trying my own variant.

#include <fxsaber\Math\Math.mqh> // https://www.mql5.com/ru/code/17982

matrix<double> CorrMatrix( const double &Array[], const int PatternLen, const bool CreateMatrix = true )
{
  matrix<double> Res = {};
  const int Size = ArraySize(Array) / PatternLen - 1;
  
  if (CreateMatrix)
  {
    Res.Init(Size + 1, Size + 1);
    Res[Size][Size] = 1;
  }
  
  double Corr[];

  DEJAVU Dejavu;
  Dejavu.SetSignal(Array);
  
  if (CreateMatrix)
    for (int i = 0, j; i < Size; i++)
    {
      Res[i][i] = 1;
      j = i + 1;

      Dejavu.SetPattern(PatternLen, i * PatternLen);
      Dejavu.GetCorrelation(Corr, PatternLen, j * PatternLen);
      
      for (; j <= Size; j++)
      {
        const double Tmp = Corr[(j + 1) * PatternLen - 1];
        
        Res[i][j] = Tmp;
        Res[j][i] = Tmp;
      }    
    }
  else
    for (int i = 0; i < Size; i++)
    {
      Dejavu.SetPattern(PatternLen, i * PatternLen);
      Dejavu.GetCorrelation(Corr, PatternLen, (i + 1) * PatternLen);
    }
        
  return(Res);
}

#property script_show_inputs

input int inRows = 100; // Длина строки
input int inCols = 15000; // Количество строк

#define  TOSTRING(A) #A + " = " + (string)(A) + " "

// NumPy: https://www.mql5.com/ru/forum/86386/page3257#comment_49545300
void OnStart()
{  
  PrintCPU(); // https://www.mql5.com/ru/forum/86386/page3256#comment_49538685

  double Array[];
  FillArray(Array, inRows * inCols); // https://www.mql5.com/ru/forum/86386/page3267#comment_49671602

  matrix<double> Matrix;  
  Matrix.Assign(Array);
  Matrix.Init(inCols, inRows);
  Matrix = Matrix.Transpose();
  
  ulong StartTime, StartMemory;
  
  Print(TOSTRING(inRows) + TOSTRING(inCols));
  
  BENCH(matrix<double> Matrix1 = CorrMatrix(Matrix)) // https://www.mql5.com/ru/forum/86386/page3267#comment_49671602
  BENCH(matrix<double> Matrix2 = Matrix.CorrCoef(false)); // https://www.mql5.com/ru/docs/basis/types/matrix_vector
  BENCH(matrix<double> Matrix3 = CorrMatrix(Array, inRows)); // https://www.mql5.com/ru/code/17982

  BENCH(CorrMatrix(Array, inRows, false)); // https://www.mql5.com/ru/code/17982

  Print(TOSTRING(IsEqual(Matrix1, Matrix2)));
  Print(TOSTRING(IsEqual(Matrix1, Matrix3)));
}

Result.

EX5: 4000 AVX Release.
TerminalInfoString(TERMINAL_CPU_NAME) = Intel Core i7-2700 K  @ 3.50 GHz 
TerminalInfoInteger(TERMINAL_CPU_CORES) = 8 
TerminalInfoString(TERMINAL_CPU_ARCHITECTURE) = AVX 
inRows = 100 inCols = 15000 
matrix<double> Matrix1 = CorrMatrix(Matrix) - 13795998 mcs, 1717 MB
matrix<double> Matrix2 = Matrix.CorrCoef(false) - 38346008 mcs, 1717 MB
matrix<double> Matrix3 = CorrMatrix(Array, inRows) - 26632760 mcs, 1716 MB
CorrMatrix(Array, inRows, false) - 19412824 mcs, 0 MB
IsEqual(Matrix1, Matrix2) = true 
IsEqual(Matrix1, Matrix3) = true

The self-design turned out to be faster than the standard variant, but slower than Alglib (I could not understand the algorithm). At the same time, the homemade version can count matrices of any size, unlike other variants.

Declaring variables behind the Get the number of Questions from a "dummy"

Maxim Dmitrievsky 2023.10.01 10:12 #32676

fxsaber #:

Result.

The self-design was faster than the standard variant, but slower than Alglib (I could not understand the algorithm). At the same time, it can read matrices of any size, unlike other variants.

Why do you like mql so much for calculations? You can write dlls in C and it will be as fast as possible.

To me, mql is still a language for opening trades, mostly. And what is right, in fact.

Any questions from newcomers Errors, bugs, questions Discussion of article "How

mytarmailS 2023.10.01 10:23 #32677

fxsaber #:
If you don't understand, let me explain.

MathRand is a random number. I.e. correlation is calculated on random matrices. The goal is to make sure that different algorithm implementations give the same result. The speed of algorithm execution and memory consumption are compared. If you're incompetent, stay out of it.

So what is the ultimate goal of this whole epic with correlation?

Well, you have found out which implementation is faster, which one gives identical results, which one does not.... What are you going to do with this knowledge?

Interesting and Humour Any rookie question, so Matstat Econometrics Matan

fxsaber 2023.10.01 10:25 #32678

Maxim Dmitrievsky #:

And why do you like mql so much for calculations? You can write dlls in c and it will be as fast as possible.

I am limited by my incompetence.

mytarmailS 2023.10.01 10:25 #32679

Maxim Dmitrievsky #:

to me, mql is still the language for opening trades, mostly. And rightly so, in fact.

Also testing and visualisation, and fast action I guess

fxsaber 2023.10.01 10:27 #32680

mytarmailS #:
So what's the end goal of this whole correlation thing?

Well, you've found out which implementation is faster, which one gives identical results, which one doesn't... What are you going to do with this knowledge?

I'll be able to calculate what others can't because of technical limitations.

Machine learning in trading: theory, models, practice and algo-trading - page 3268