Machine learning in trading: theory, models, practice and algo-trading - page 408

 

Good day to all.


Let me clarify, which ML packages are used by the participants of the discussion, or everything is done through MQL?

 

Surprisingly the file counted, the quality of the model is weak, who wants to check the validation....

Files:
 
Mihail Marchukajtes:

Surprisingly the file counted, the quality of the model is weak, who wants to check the validation....

In a file it is written
* TruePositives: 853
* TrueNegatives: 1098
* FalsePositives: 732
* FalseNegatives: 985

So it's almost 50/50.
I got about the same result for a slightly simpler logic problem, which is why I assumed that the noise inputs are very much spoiling the RNN result.

 
elibrarius:

The file says
* TruePositives: 853
* TrueNegatives: 1098
* FalsePositives: 732
* FalseNegatives: 985

So it's almost 50/50.
I got about the same result for a slightly simpler logic problem, which is why I assumed that the noise inputs were really screwing up the RNN result.


It does, the generalizability of the model is 52 percent, so...
 
Aleksey Terentev:

Good day to all.


Let me clarify, which ML packages are used by the participants of the discussion, or everything is done through MQL?

As a rule, everything is done in R, with the exception of a few enthusiasts who are trying to rewrite some models in MQL. Regarding packages - it depends on which model one uses - trees, neural networks, deep neural networks, etc.

Which model are you interested in?

Good luck

 
Vladimir Perervenko:

As a rule, everything is done in R, except for a few enthusiasts who try to rewrite some models in MCL. As for packages - it depends on what model one uses - trees, neural networks, deep neural networks, etc.

Which model are you interested in?

Good luck


You forgot about Reshetov's optimizer, it's used here too ....
 
elibrarius:

The file says
* TruePositives: 853
* TrueNegatives: 1098
* FalsePositives: 732
* FalseNegatives: 985

So it's almost 50/50.
I got about the same result for a slightly simpler logic problem, which is why I assumed that the noise inputs were really screwing up the RNN result.


An ordinary MLP 20-5-1 can do that to the same problem:

Average error on training (60.0%) plot =0.116 (11.6%) nLearns=2 NGrad=2297 NHess=0 NCholesky=0 codResp=2
Mean error on validation (20.0%) section =0.111 (11.1%) nLearns=2 NGrad=2297 NHess=0 NCholesky=0 codResp=2
Mean error on the test (20.0%) section =0.129 (12.9%) nLearns=2 NGrad=2297 NHess=0 NCholesky=0 codResp=2
Calculation time=0.77 min.

Although sometimes it gives 50% too, probably it's necessary to set more than 2 retraining cycles.
---------------------------------

Network nnet 20-1-1 of R :
Predicted
Actual 0 1 Error
0 0.41 0.09 0.18
1 0.24 0.26 0.48

Slightly worse, but only solves it in a couple of seconds.

But the RNN has an error = 0, on the training section. And that's cool, too. I'll probably blog my experiment

 
elibrarius:

An ordinary MLP 20-5-1 can produce this for the same problem:

Average error in training (60.0%) plot =0.116 (11.6%) nLearns=2 NGrad=2297 NHess=0 NCholesky=0 codResp=2
Mean error on validation (20.0%) section =0.111 (11.1%) nLearns=2 NGrad=2297 NHess=0 NCholesky=0 codResp=2
Mean error on the test (20.0%) section =0.129 (12.9%) nLearns=2 NGrad=2297 NHess=0 NCholesky=0 codResp=2
Calculation time=0.77 min
---------------------------------

Network nnet 20-1-1 from R :
Predicted
Actual 0 1 Error
0 0.41 0.09 0.18
1 0.24 0.26 0.48

Slightly worse, but only solves it in a couple of seconds.

The thing to understand here is that solving the problem is easy, but how the model will behave next is the question. We can learn quickly and without errors. And with new data we have zero. Predikshin has one very good feature, on new data it works with the same error rate as on training. Because it doesn't retrain and blah blah blah. And the other networks can learn this example very quickly, but show much worse results on new data... IMHO
 
Mihail Marchukajtes:

We need to understand something else: it's easy to solve the problem, but how the model will behave afterwards, that's the question. We can learn quickly and without errors. And with new data, we have zero. Predikshin has one very good feature, on new data it works with the same error rate as on training. Because it doesn't retrain and blah blah blah. And the other networks can learn this example very quickly, but show much worse results on new data... IMHO

So here's the new data:

Average error on validation (20.0%) plot =0.111 (11.1%) nLearns=2 NGrad=2297 NHess=0 NCholesky=0 codResp=2
Average error in the test (20.0%) plot =0.129 (12.9%) nLearns=2 NGrad=2297 NHess=0 NCholesky=0 codResp=2

 
elibrarius:

So here's the new data:

Average error on validation (20.0%) plot =0.111 (11.1%) nLearns=2 NGrad=2297 NHess=0 NCholesky=0 codResp=2
Average error on test (20.0%) plot =0.129 (12.9%) nLearns=2 NGrad=2297 NHess=0 NCholesky=0 codResp=2


I'm not arguing, it could be... :-)
Reason: