Is there a pattern to the chaos? Let's try to find it! Machine learning on the example of a specific sample. - page 7

 
Aleksey Vyazmikin #:
I can tell you what predictors one of the models used - check if you successfully trained (I have almost no doubt) - do you need to?

You can put the column numbers in commas. It would be interesting to compare.

 
elibrarius #:

You can put the column numbers in commas. It's interesting to compare.

Is the first column a zero or a "1"? :)

Here's the zero count

1,225,345,625,702,725,779,798,841,1104,1118,1268,1337,1357,1639,1737,1922,1937,2027,2043,2125,2239,2354,2474,2475,2507,2796,2816,2953,3012,3207,3228,3378,3671,3717,3726,3734,3864,3927,3982,4522,4560,5007,5029,5129,5313,5359,5548
 

Here's another variant - I like it even better, as it's a stable result on all samples.

606,1048,1060,1083,1095,1103,1108,1110,1137,1198,1347,1353,1511,1525,1526,2055,2581,2582,3078,3153,3273,3341,3676,3690,3695,3839,3919,3967,4397,4433,5052,5364,5579



Balance

 
Aleksey Vyazmikin #:

Is the first column zero or "1"? :)

That's zero.

I have 0. I'll try to teach it tomorrow.
 
elibrarius #:
I'm 0. I'll try training it tomorrow.

It should work - there are few predictors. Although, for screening I used my own method, creating forced quantum tables, and then the algorithm made a selection from them.

The question here is how you can define the predictor's attributes to learn how to select them before training....

 
Aleksey Vyazmikin:

Есть ли закономерность в хаосе?


The very statement of the question is wrong! The very definition of "CHAOS" is a state where regularities are OUTSIDE....

It is necessary to start not with CHAOS, but with the search for a state similar to the MARKET!

And it will not be chaos, but some intermediate state between chaos and ordered data.....

And to look for patterns in chaos is foolishness.... ( read the definition of "chaos" )....

 
The market is complete chaos, but you need to be able to see a system in it. Or, to be more precise, to drive the market into the procrustean bed of the system.
 
webgopnik #:
The market is complete chaos, but you need to be able to see a system in it. Or, to be more precise, to drive the market into the procrustean bed of the system.

It's a bit like the expression: "Make shit into candy!" - folk wisdom...


In fact, everything is exactly the opposite: the market is a SYSTEM! And you need to cure your brain of chaos to understand this system....

 
Aleksey Vyazmikin #:

It should work - there are few predictors. Although, for screening I used my own method, creating forced quantum tables, and then the algorithm made a selection from them.

The question here is how one can define the predictor's attributes to learn how to select them before training....

By the way, your data can really be referred to Big Data...

250 million cells in a table.

How long does catbust take to train 1 model on a full dataset?

 
elibrarius #:

By the way, your data - really can be attributed to Big Data.....

250 million cells in a spreadsheet.

Yes, there is a lot of data, and I plan to add more - so development of a pre-training screening methodology is required.

elibrarius #:

How long does catbust take to train 1 model on a full dataset?

I'm just now training it, and I will say that a lot depends on the settings, especially the number of splits in the quantum tables.

Just started an experiment, where training goes with default settings on the video card - one pass, without taking into account model evaluation and test on exam sample, takes 2-3 minutes - depending on the resulting number of trees in the model. On my already quite outdated FX-8350 processor, it is about 60% slower.

I think that the speed is quite acceptable, I usually train 100 models with a fixed seed, which would average the effectiveness of the method.

If you train to the "end", the programme estimates the time up to 2 hours.

Reason: