Machine learning in trading: theory, models, practice and algo-trading - page 3662

New comment
 

See I don't even have an over-trained graph on my training... aim for the IC and OOS to be similar, this means no pattern drift on new data.

Check out how many data partitioners there are. You have to choose one for each instrument :) Trend strategies are better for the Eurobucks.

If nothing helps, I use markup enhancers.


 
Maxim Dmitrievsky #:

On 10 MAshkas, study period

No tuning, just started, 5 sec.

Maxim Dmitrievsky #:

See, I even have a graph on training without retraining... strive to make IC and OOS similar, it means that there is no drift of patterns on new data.

Check out how many data partitioners there are. You have to choose one for each instrument :) Trend strategies are better suited for the Eurobucks.

If nothing helps, I connect markup enhancers.


Man, step forward

 
Ivan Butko #:

Man, step forward

3 years of periodic ass....w

and it's in python, it takes longer to write in MQL, because there are few ready-made ones.

Even if you ask a cool MQL engineer to do it, he won't solve the problem, because it is very specific. Optimisers are even less so.

That is, they don't know how to approach it at all. They'll start training all sorts of deep neural networks with a known outcome. All kinds of reinforcement learning, optimisation of everything that moves, a million signs, that's a load of rubbish.
 
Maxim Dmitrievsky #:

3 years of periodic ass....w

and this is in python, it takes longer to write in MQL because there are few ready

Even if you ask a cool MOSHnik to do it, he will not solve the problem, because it is very specific. Optimisers even less so.

So they don't know how to approach it at all. They'll start training all sorts of deep neural networks with a known outcome. All kinds of reinforcement learning, optimising everything that moves, a million signs, that's a load of crap.
Yeah, I've tried RL. It all boils down to one thing: the point of the q-table is to memorise history, and putting formulas on top of it looks like an attempt to memorise it "faster". Works great stationary when you need to teach a dummy to walk upright and down stairs.


If you haven't wrapped it up in a trade secret yet: how do you explain the robotability of your model? If even hundreds of neurons in 7 layers can't do the job.
 
Ivan Butko #:
Right, tried parsing RL. It all boils down to one thing: the point of a q-table is to memorise history, and putting formulas on top looks like an attempt to memorise "faster". Works great stationary when you need to teach a dummy to walk upright and down stairs.


If you haven't wrapped it up in a trade secret yet: how do you explain the robotability of your model? If even hundreds of neurons in 7 layers can't do the job.
It comes down to finding patterns. Any number of features in n-dimensional space will contain n number of patterns. Then check them on new data and forbid to trade where the patterns are bad.

So many patterns are classified with the label 0, so many with the label 1. They must differ "seriously" for the model to classify them with low error. And have few contradictions within the class, i.e. few mislabelled ones.
 
Clustering does something. It separates groups of observations that are maximally compactly located in one cluster from other clusters that are maximally far from each other and also have compact representations of other observations. This is good for the model because there is distance between the groups, they overlap little or not at all, so the model does not overtrain. These groups can be treated as separate patterns. Then we need to check how each group predicts the next observations. Select the best ones and filter the rest or do not allow trading.

Suppose we take the average of forecasts for each cluster for n bars ahead and the standard deviation. If the average is shifted towards buying or selling, it is a good cluster. But there will still be some fraction of bad examples. If it's 50/50, it's a bad cluster. You need at least 70/30.

Then you take the second cluster, which is as far away from this one as possible, and look in it. There should be a bias in the other direction.

This is a kind of TS with buy and sell signals.

In practice, you need to make additional efforts, because in general case the boundaries between clusters are usually blurred and predictions are close to 50/50.
 
bestvishes #:
Very innovative ideas。 Can you keep writing articles about this tag clustering method, such as second level clusters, which are not very clear yet？

I'm not ready to fully describe my approach yet. Here is an example of dividing a dataset into groups, where observations with tags 0 and 1 will be maximally far apart in the feature space, e.g. by Euclidean distance.

You can develop this topic and write something interesting based on it.

import numpy as np
from sklearn.cluster import KMeans
from sklearn.metrics import pairwise_distances

def filter_samples_by_clusters(X, y, n_clusters=2, threshold_percentile=20, min_distance_percentile=80):
    # Разделяем данные на два класса
    X_0 = X[y == 0]
    X_1 = X[y == 1]
    
    # Применяем K-means к каждому классу отдельно
    kmeans_0 = KMeans(n_clusters=n_clusters, random_state=42).fit(X_0)
    kmeans_1 = KMeans(n_clusters=n_clusters, random_state=42).fit(X_1)
    
    # Получаем центроиды кластеров
    centroids_0 = kmeans_0.cluster_centers_
    centroids_1 = kmeans_1.cluster_centers_
    
    # Вычисляем расстояния до центроидов своего класса
    distances_0 = pairwise_distances(X_0, centroids_0).min(axis=1)
    distances_1 = pairwise_distances(X_1, centroids_1).min(axis=1)
    
    # Вычисляем расстояния до центроидов противоположного класса
    distances_0_to_1 = pairwise_distances(X_0, centroids_1).min(axis=1)
    distances_1_to_0 = pairwise_distances(X_1, centroids_0).min(axis=1)
    
    # Оставляем только те точки, которые:
    # 1. Близки к своим центроидам
    # 2. Далеки от центроидов другого класса
    threshold_0 = np.percentile(distances_0, threshold_percentile)
    threshold_1 = np.percentile(distances_1, threshold_percentile)
    
    min_dist_0 = np.percentile(distances_0_to_1, min_distance_percentile)
    min_dist_1 = np.percentile(distances_1_to_0, min_distance_percentile)
    
    good_indices_0 = np.where(y == 0)[0][
        (distances_0 <= threshold_0) & 
        (distances_0_to_1 >= min_dist_0)
    ]
    good_indices_1 = np.where(y == 1)[0][
        (distances_1 <= threshold_1) & 
        (distances_1_to_0 >= min_dist_1)
    ]
    
    # Объединяем индексы
    good_indices = np.concatenate([good_indices_0, good_indices_1])
    
    return good_indices
 
Out of sample improvements are a very difficult thing to do, it's hard to find satisfactory ideas all over the net, feel you are sharing, this is great, thank you.
 
Maxim Dmitrievsky #:
Did not look, afk
Average deal is around 30 pips on euro, on flat pairs a little less than that
Can I see a screenshot from the statistics page? It would be interesting to know other estimates: what is the duration of the training and OOS plots, for example. And everything else for comparison.
 
Forester #:
Can I see a screenshot from the statistics page? It would be interesting to know other estimates: what is the duration of training and OOS plots, for example. And everything else for comparison.

Not interesting

Different models, different statistics, different tools. Zero useful information

It is interesting to know how to obtain them.

1...365536563657365836593660366136623663366436653666366736683669...3741
New comment