ForumSections

Machine learning in trading: theory, models, practice and algo-trading - page 3271

New comment
fxsaber #:

Compare to the old version of Alglib. I have no data that it has become slower.

You yourself wrote that the standard one is slower than the current Alglib one. I have the old one in the form of code, but not in the terminal.
Forester #:

Maxim's CPU is 2 times faster than mine. I don't remember if he gave timings for Algliba, I think not.

Timings.

Forum on trading, automated trading systems and testing trading strategies

Machine learning in trading: theory, models, practice and algo-trading.

Aleksey Vyazmikin, 2023.09.26 05:19 AM

Array size: 1716.61376953125 MB
Time taken: 4.784467697143555 seconds

This is on an old FX-8350


Forum on trading, automated trading systems and testing trading strategies.

Machine Learning in Trading: Theory, Models, Practice and Algorithm Trading

Aleksey Vyazmikin, 2023.09.26 05:37 AM

For statistics, this is my result

2023.09.26 06:28:23.304 Test_Corr (USDJPY,H1)   EX5: 3981 AVX Release.
2023.09.26 06:28:23.304 Test_Corr (USDJPY,H1)   TerminalInfoString(TERMINAL_CPU_NAME) = AMD FX-8350 Eight-Core 
2023.09.26 06:28:23.304 Test_Corr (USDJPY,H1)   TerminalInfoInteger(TERMINAL_CPU_CORES) = 8 
2023.09.26 06:28:23.304 Test_Corr (USDJPY,H1)   TerminalInfoString(TERMINAL_CPU_ARCHITECTURE) = AVX 
2023.09.26 06:28:23.332 Test_Corr (USDJPY,H1)   inRows = 100 inCols = 15000 
2023.09.26 06:28:45.032 Test_Corr (USDJPY,H1)   matrix<double> Matrix1 = CorrMatrix(Matrix) - 21700095 mcs, 1717 MB
2023.09.26 06:29:48.495 Test_Corr (USDJPY,H1)   matrix<double> Matrix2 = Matrix.CorrCoef(false) - 63460976 mcs, 1717 MB
2023.09.26 06:29:50.225 Test_Corr (USDJPY,H1)   IsEqual(Matrix1, Matrix2) = true
Clearly Python is much faster. And so it will be faster in C, so it turns out that MQ is not telling us something, promising comparable performance....

I should note that Python has a small parallelisation when running code - for half a second for about two cores, the rest is counted on one core.

Forester #:
You wrote yourself that the standard one is slower than the current alglibov one. I have the old one in the form of code, but not the terminal.

The Alglib source itself was rewritten by MQ for its matrices. I don't even want to discuss the standard CorrCoef, there are obvious problems there.

I.e. there are two sources of Alglib.

  • The original one on dynamic arrays BEFORE MQ-transformation.
  • Modified MQ for its matrix/vector types. That's what I tested.
The sources of both versions of Alglib should be available, so you can compare them with each other.
fxsaber #:

Timings.


Well, yes... people devote years to algorithms. I saw an article about fast sorting in C++ - the code seems to be more complicated and does unnecessary actions, but it works faster due to pre-calculation of the processor. Looking ahead...
fxsaber #:

The Alglib source itself was rewritten by MQ for its matrices. I don't even want to discuss the standard CorrCoef, there are obvious problems there.

I.e. there are two sources of Alglib.

  • The original one on dynamic arrays BEFORE MQ-transformation.
  • Modified MQ for its matrix/vector types. That's what I tested.
The sources of both versions of Alglib should be available, so you can compare them with each other.
I'm lazy. I don't use correlation. I just noticed a difference in speed a year ago and shared it when the topic came up.
fxsaber #:
NumPy seems to have a different algorithm from ALglib

In AlgLib, the original documentation says why different, which ones and what they are for. With regressions (I was mostly digging into AlgLib there) it is quite original.

Again, everything compares strangely, as you can't. Build graphs of speed=f(dimensionality,special_matrix_properties) dependencies for different libraries/realisations and look at them. You're taking edge cases, taken from the ceiling.

and there you look not at the absolute value, but at the symptomatology and the presence of a "plateau". From there you choose a tool to work with specific data.

[Deleted]
Forester #:

Maxim's CPU is 2 times faster than mine. I don't remember if he gave timings for Algliba, I think not.

I have mt through virtualisation there, the tests will not be very plausible.

Plus I chose to compute something in python and then transfer it to any platform. For example, for crypto you don't need terminals at all.

It's a total crap shoot in terms of speed
Maxim Kuznetsov #:

Again, it's a strange way of comparing things, in a way you can't.

I don't do comparisons, I provide code that everyone can measure for their own case.

The string length of 100 is the length of the pattern. You probably don't need more than that.

15000 samples is memory limited because of the quadratic size of the correlation matrix. The more samples, the better. That's why I wrote a homemade one, where you can have a million of them.

I have neither desire nor time to spend on objective comparison. I made it for my own tasks and shared the working code. Whoever needs it will see it.

Ehhh... If even 5% of all our efforts were directed to a common task, we would conquer this market in months.

And so for years, decades everyone is boiling in his own soup and 90% of useful time is spent on this useless forum.
fxsaber #:

I made it for my own tasks and shared the working code. Anyone who needs it will have a look.

Where can I see the final code?

1...326432653266326732683269327032713272327332743275327632773278...3749
New comment