Machine learning in trading: theory, models, practice and algo-trading - page 98

[СанСаныч Фоменко]

Dr.Trader:

The second part of the experiment is.

I had 14 previously selected predictors, added another 14 with random values. The maximum allowed number of ForeCA components is now 28.

Prediction accuracy with all 28 components on training data in both cases (with and without foreCA) 76%, accuracy on new data in both cases 57%.

In my opinion foreCA did not cope with garbage in predictors, I did not see the expected miracle.

Wonderful!

Thank you for reducing my workload.

[Dr. Trader]

I noticed that on other data my ForeCA example did not work, because the predictors have too high covariance.
This can be solved with this code:

#это  строки кода из примера:
targetName <- colnames(trainData)[ncol(trainData)]
predictorColnames <- colnames(trainData)[-ncol(trainData)]
#...
#это  нужно добавить сразу после них:
while(TRUE){
        covMatrix <- cov(as.matrix(trainData[,predictorColnames]))
        covMatrixRank <- qr(covMatrix)$rank
        if(covMatrixRank == ncol(covMatrix)){
                break
        }else{
                eigenValues <- abs(eigen(covMatrix)$values)
                #  eigenValuesLow <- order(eigenValues, decreasing=FALSE)[1:(ncol(covMatrix)-covMatrixRank)]
                #  predictorColnames <- predictorColnames[-eigenValuesLow]
                eigenValuesHigh <- order(eigenValues, decreasing=TRUE)[1]
                predictorColnames <- predictorColnames[-eigenValuesHigh]
        }
}

Just remove the predictors with high eigenvalue from the covariance matrix one by one. Maybe we should remove the predictors with low eigenvalue the other way around. But it is still long and inefficient, but how to make it better - I do not know yet.

[mytarmailS]

Why have the indicators never worked and never will, my opinion... And how you can try to fix it....

a little bit about being stoned... :)

We all know that the markets are not stationary, but we mostly use the tools for the analysis of the market for stationary series, including me, I do not know why, what is the reason?

There are some well-established algorithms for non-stationary data analysis, and there are not many of them - hidden Markov models "HMM" , the method of group accounting of arguments "MGUA", and especially for BP there are recurrent neural networks, why don't we use these algorithms? But we all use scaffolding, conventional neuronics, the stupidest classifiers which have nothing to do with the problem

Why is that? Why is it that even though I understand all of this, I continue to train RF... What's wrong with us... Write what you think about it ...

Now about the indicators and a little bit about geekiness again...;)

Knowing that the markets are unsteady, we're trying to find some indicators parameters that would work in the future, how is it possible if the market is constantly changing? The answer is no!!! Or it's not so simple ...?

I think there is a way to be more objective about the market using indicators, to illustrate our idea we can use spectrum analysis, in this case the wavelet

in the picture on the left there is a scale with periods marked on it, i.e. the periods that are in the market, the redder is the area in the graph the stronger the period prevails, the strongest period is outlined with black outline like in the picture...

Please do not read any further until you run the code

after the code is executed click on the "back arrow" above the picture in the studio to see all the pictures that came out after the code execution as if they were slides

install.packages("dplR")

hankel <- function(data, r=10) {
  do.call(cbind,
          lapply(0:(r-1),function(i) { data[(i+1):(length(data)-(r-1-i))]}))} #hankel  matrx

price <- cumsum(rnorm(200))+1000    ;    plot(price,t="l")
H.price <- hankel(price,100)

library(dplR)
for(i in nrow(H.price):1 ){
  m <- morlet(y1 = H.price[i,] )
  wavelet.plot(m)
}

You see that the periods in the market are constantly floating like clouds in the sky, and now it's clear that there's nothing to do in the market with fixed period, but you can adapt. What if you reveal the current strong period in the market and constantly adjust the indicator for that real - objective period, which is exactly on the market now?

links

https://cran.r-project.org/web/packages/dplR/vignettes/timeseries-dplR.pdf

https://www.r-bloggers.com/wavelet-spectrogram-non-stationary-financial-time-series-analysis-using-r-ttrquantmoddplr-with-usdeur/

[СанСаныч Фоменко]

For lovers of NS here

Deep Learning Part 1: Comparison of Symbolic Deep Learning Frameworks

[Yuri Evseenkov]

mytarmailS:

Why have the indicators never worked and never will, my opinion... And how we can try to fix it....

We all know that the markets are not stationary, but the vast majority of tools we use to analyze the market are stationary series, including me, I do not know why, what is the reason? fashion? stubbornness?

Knowing that the markets are non-stationary, we try to find some indicator parameters that would work in the future, how is it possible if the market is constantly changing?

But the classification, as Sanych wrote about it, will not work?

[mytarmailS]

Yuri Evseenkov:
And the classification, as Sanych wrote about it, will not work?

Let's be more detailed, because I do not understand what classification we are talking about

[Yuri Evseenkov]

mytarmailS:
Let's be more specific, because I don't understand what classification we're talking about at all

L What am I, a doctor? Here is Sanych writing:

"Here we are discussing classification-based predictions, which do not take into account the previous state when predicting the next bar. Predictions (forecasts) based on classification are predictions based on patterns. And if there was news in the past that caused a change that does NOT follow from previous values (not extrapolated), then the classification will catch that change as such and if there is a similar change in the future (not exactly the same, but similar) it will be recognized and a correct prediction will be made. "

That's what I think we should dig in this direction :"classification will catch such a change as such" .

[mytarmailS]

Yuri Evseenkov:

L What am I, a doctor? Here is Sanych writing:

"Here we are discussing classification-based predictions, which do not take into account the previous state when predicting the next bar. Predictions (forecasts) based on classification are predictions based on patterns. And if there was news in the past that caused a change that does NOT follow from previous values (not extrapolated), then the classification will catch that change as such and if there is a similar change in the future (not exactly the same, but similar) it will be recognized and a correct prediction will be made. "

So I think it is worth digging in this direction:"classification will catch this change as such" .

experiment is the criterion of truth - don't think, but do

It seems to me personally that spectral analysis is more promising, but this is for me personally...

[DAFomenko]

Yuri Evseenkov:

So I think we should dig in this direction:"classification will catch such a change as such.

Classification is not a panacea or a grail-making tool.

The first thing the application of classification does is to apply the tools to the problems to which the tools apply. For example, the idea of applying spectral analysis to financial markets has been discussed many times, and everything seems to be a great tool, but for other objects, ah, no, again it is offered.

Second. The classification is quite applicable to financial markets, but there are many troubles, as it was written above. But with the classification we can focus on the main problem - the problem of retraining (overfitting) of the TS. What could be more important? It is not pleasant, of course, to be deprived of illusions about having a favorite grail, but here is the choice: happiness is good, but truth is better?

Third. Classification poses the question quite specifically: What are we predicting. Let us compare it with TA. We take indicators. It is always a bar [1]. The current bar is not used. What does it mean for H1? We use the information of hourly freshness for market entry forecasts! This is in the best case.

This is completely different in the classification. You take the current value of the target variable and match it to yesterday's raw data - shift the target by one or more bars. When you use a model fitted to such data, you always really predict the future when the next bar arrives.

PS.

If you are going to use it to predict abrupt market moves (news), then you will succeed if you can form a target variable, and you have big problems with that in much simpler cases.

[mytarmailS]

DAFomenko:

For example, the idea of applying spectral analysis to financial markets has been discussed many times, and everything seems to be a great tool, but for other objects, ah, no, again it is proposed.

????????????????????????

If the researcher has no idea about spectral analysis ending with the wikipedia article, I agree that it is not applicable to the market:)