Machine learning in trading: theory, models, practice and algo-trading - page 1248

 

I'm thinking, if overtraining is a consequence of memorizing the market due to noise, then memorizing the market requires a certain amount of memory in the form of a forest of decision trees and boosting, and models like single tree and neural network with a small number of neurons should be less overtrained. Then it turns out that there is a critical amount of data that can not be described by, say, a single sheet - what is the volume of 1% or 10% - of the entire sample (training, test, control) is the question. Then maybe we should evaluate the data in terms of the amount of memory needed to remember this data and try to make the model an order of magnitude less of this critical volume? How to do this - I do not know, maybe there is something similar to the archiving should be - if the archiver compresses a sample of 10mb in 1mb, the model should not be more than 102.4 kb. And, then, knowing that the model not only memorized the sample, but found regularities in it, we can be less critical when testing on an independent sample and conclude not about overtraining, but about the lack of data on the sample for training, as there were no situations describing the current market state, and therefore there was simply no possibility to find such regularity when using the available predictors.

 
Now making a hand model based on the sheets received, and it turns out that about 20 sheets to buy and 5 sheets as a filter can generate income every year from 2014 to 2018, and so I think, it can not be overtraining and to fit a little leaves somehow ... so it's a pattern that will lose its validity in 2019?
 
Maxim Dmitrievsky:

A pattern must have some fundamental prerequisites, for example, some cycles in the market are fundamental, or the reaction to news, intraday volatility... things like that.

And if the regularities are not clear from where they come from, then it is unclear what will happen next.

Models on trees and deals with the issue of identifying patterns, the predictor narrates the event, and a set of certain events and gives a pattern. It's just that this regularity is not from the field of physical phenomena and it can not be constant, because it can be affected by unknown factors (phenomena not described by available predictors).

In general the point is that statistically recurring event is detected by MO methods of classification and it's kind of better than just fitting the market with optimizer indicators, or not?
 
Maxim Dmitrievsky:

The correlation between the number of short skirts in summer and the financial well-being of citizens may be 90 percent, but it does not mean a correlation, much less a pattern

Okay, let's replace the word "pattern" with "omen" as an unknowable correlation to the event.

Maxim Dmitrievsky:

The optimizer is also an MO.

So you equate these two methods?

 
Maxim Dmitrievsky:

It's not me who's betting, it's how it is... optimizer minimizes any f-function, neural network optimizer optimizes the weights f-function

If we're talking about optimization of TC filters only, then we can partially agree about the same actions, but if the input/output point is also optimized, then the situation is somewhat different from the usual MO.

 
Maxim Dmitrievsky:

In this regard, it is necessary to study data mining and application to markets, if it is possible at all, because data mining is also the top of dumbness, but some information can be extracted)

To study in order to apply in another field?

 
Maxim Dmitrievsky:

in order to understand how to extract something useful from the data

So I noted earlier that I extracted something useful - maybe it's a grain of gold, maybe it's an accident... who knows... Who knows ... and there's no way to know for sure.

 
Assuming that the markets are all the same, the price behavior has similar patterns, why not combine a dozen instruments in one sample and look for common "signs" for all markets?
 
Vizard_:

That's how it's done - it's generalized. The "patterns" are patterns found from different samples... That is, you just matched in a trend sample, and now you're giving us a false sense of humor.) But all the same, the data should be prepared thoughtfully, for the idea... ...although it may work, but I doubt it...

I had training for 2016-2017, and then just checking the sheets for 2014-2018 and selecting the ones that were profitable every year and met a number of criteria (overall growth/not a lot of drawdown). That's what I think, is it possible to use such a model.

As for combining different instruments, so many predictor here is the gain in pips over different time intervals, and it won't work then with different instruments...

 
Maxim Dmitrievsky:

All markets are different, of course, the patterns are also different, and what works for one is a loss for the other.

To assume something you have to assume something as a basis for such a rough assumption

So I assume that the subject is the same everywhere - the trader and why would he change his behavior depending on the instrument? If he uses technical analysis or any other method, but this method he uses everywhere, it is different that he may use a set of methods at different times and therefore it is easier to fit one method on one instrument and when the trader (collective image) switches to another method, then the model will break.

Reason: