Machine learning in trading: theory, models, practice and algo-trading - page 1173

New comment
 
Yuriy Asaulenko:

How do you read books? Do you put them under your ass?

It's very popular. A problem either has a solution or it doesn't. If there isn't one, you can't solve it. If there is, you won't be able to solve it. And NS has nothing to do with it.

So read the book suggested above. There is always an optimal solution in a quantity of one, whether it is satisfactory is a second question.

 

As promised, I report the results of the test of different indicators for selecting a model of binary classification. The test includes 17 indicators: Logloss; BalancedErrorRate; LogLikelihoodOfPrediction; WKappa; Kappa; BrierScore; ZeroOneLoss; Accuracy; BalancedAccuracy; MCC; Precision; Recall; F1; AUC; HingeLoss; HammingLoss; CrossEntropy, the default is Logloss. The methodology is simple, we run CatBoost with the same settings on the same samples for training, test and final check, the numerical indicators are taken from the file for the final check, which does not take part in the training. For each indicator, 8 trees of different sizes from 1 to 8 are formed, which averages as the ability to select good values from the set, and also used to make sense of changing the size of the trees. Used 4 targets for training, whose strategies differed to a greater extent on the target for the trend movement. The results of each indicator were summarized in the following table:



It turned out that some of the indicators are fully consistent with each other, they were gathered into groups - the indicator group identifier is shown in green


Below is a summary table for each indicator, where the maximum and minimum values of the index are marked, as well as peak values of the whole sampling are colored.

Then there is a summary table for the average profit of each indicator, it is a variant of identifying a universal indicator for all variants of the target - by the maximum average amount.



As I wrote earlier, I was testing my formula on that data in parallel in order to select the best indicator out of the obtained data - "%All*1%", the results were combined into such a table for each indicator participating in the tests.

Such an approach made it possible to rank by the sum of the Choice Factor scores for each indicator, as shown in the table below.

Interestingly, the favorite in the first ranking, Precision, was at the very bottom, which suggests that it is not universal, but on the contrary, its performance was very good for the first target.

We can also consider what options this factor selected from the table of each indicator and look at the rating on the sum of these indicators.

To summarize, you can gather the results in a summary table and summing up the points for the places - who has less that metric can be better.

We can conclude that Logloss is rather an average index, not the best index for model selection, while different TS may require different methods of selection of trained models. So for example for more balanced sample by target (columns_100) Precision is perfect, but for unbalanced sample, where for example 20% (columns_200) and less than target "1", it goes very badly and here it is reasonable to use HingeLoss, which coped with this task perfectly. The most balanced metric was Recall, which can be used to probe the ground, I think. Such an indicator as Accuracy turned out to be extremely unreliable and unbalanced on the one hand it was able to find the best option with the target most balanced, but on the other hand the average of all options trees it did not select well, as a result to find unbalanced samples or adjust the model it is not very suitable.

 
Alexander_K2:


Now the trend is easily and gently flushed down the toilet with the coveted key, which I found literally "on my knees.


I wonder what I came up with there.

 
Aleksey Vyazmikin:

Great tests, thanks

Is there any information on the differential of errors of the train\test? Just take some Accuracy or logloss there, the most popular

for example, something like this

2018.11.27 02:26:37.780 Core 2  2018.11.23 23:59:59   RlMonteCarlo TRAIN LOSS
2018.11.27 02:26:37.780 Core 2  2018.11.23 23:59:59   0.19023
2018.11.27 02:26:37.780 Core 2  2018.11.23 23:59:59   RlMonteCarlo TEST LOSS
2018.11.27 02:26:37.780 Core 2  2018.11.23 23:59:59   0.43959

right trace left test:

I'm interested in the model's ability to generalize, and what kind of stuff is there to fight overfits. I see you've quickly mastered the device. Finally, substantive conversations :))

 
Maxim Dmitrievsky:

Great tests, thanks

Is there any information on the differential of errors of the train\test? Just take any one Accuracy or logloss there, the most common

for example, something like this

right trace left test:

I'm interested in the model's ability to generalize, and what kind of stuff is there to fight overfits. I see you've quickly mastered the device. Finally, substantive talks :))

Catbust has three samples used - one is a test sample, on which it is trained; the second is to check training of each model and select the best model, which was obtained in the process of training and proved to be stable on test data; the third is an exam sample, which is marked after the training of the selected model. Since it can actually learn very well on the training sample, it makes no sense to compare the results on the test sample with the training sample, but the test with the exam sample does. But, since I processed all this by hand in Excel, my strength was not enough to check the results of the test sample as well. Data like this are of course needed to evaluate the model, and should be taken into account and analyzed, but until the process is not automated (just started writing a script to prepare the set-up for processing - far from analysis) and I can not talk about any figures.

I haven't noticed any special methods to combat overtraining, except for methods of stopping the training in case of extra good readings... In the algorithm itself there is some randomness, which can be turned off - it also allows to choose not the best predictors for branching the tree, which should help prevent overlearning.

Now I want to feed predictors in portions and see what happens, because the model has a greedy tree principle anyway and that sometimes prevents it from getting better results.

 
Aleksey Vyazmikin:

Catbust has three samples used - one is the test sample, on which it is trained, the second is to check the training of each model and select the best model, which was obtained in the process of training and proved stable on the test data, and the third is an exam sample, which is marked by the fact of finishing training by the selected model. Since it can actually learn very well on the training sample, it makes no sense to compare the results on the test sample with the training sample, but the test with the exam sample does. But, since I processed all this by hand in Excel, my strength was not enough to check the results of the test sample as well. Data like this are of course needed to evaluate the model, and should be taken into account and analyzed, but until the process is not automated (just started writing a script to prepare the set-up for processing - far from analysis) and I can't talk about any numbers.

I haven't noticed any special methods to combat overtraining, except for methods of stopping the training in case of extra good readings... In the algorithm itself there is some randomness, which can be turned off - it also allows to choose not the best predictors for branching the tree, which should help prevent overlearning.

Now I want to feed in portions of predictors, and see what happens, because the model still has the greedy principle of building a tree, and this sometimes prevents it from getting better results.

That's exactly what makes it annoying, the eternal integrations that take more time than TC development. I'd also like to use python, but I have no time yet.

 
Maxim Dmitrievsky:

This is exactly what annoys me, the eternal integration, which takes more time than TC development. I'd like to use python as well, but I don't have time yet.

I like ketbust because it has console version and I want to use it with MT5.

 
Aleksey Vyazmikin:

I am not with python, I liked that ketbust has console version, I want to make this version compatible with MT5.

I have almost all console versions, they are uncomfortable to use

 
Maxim Dmitrievsky:

Almost everyone has console ones, it's inconvenient to work with them.

I didn't know, who else has them?

Why not convenient - there will be a script on MT5, where it will be possible to implement the settings and enumeration of parameters - much better than with pure Python or R code.

If I want to use Catbust I should mention that it is able to work with grafical farms and that also attracted my attention.

 
Aleksey Vyazmikin:

I didn't know, who else exactly has it?

Why not convenient - there will be a script on MT5, where it will be possible to implement settings and enumeration of parameters - any better than with pure Python or R code.

Catbust knows how to work with grafic farms, which also caught my attention.

all the libraries have the mbo, because this one is just the libs on the pros.

it is not convenient and long to work with MO in low-level languages, especially research
1...116611671168116911701171117211731174117511761177117811791180...3496
New comment
Reason: