Discussion of article "Advanced resampling and selection of CatBoost models by brute-force method" - page 10

[Maxim Dmitrievsky]

elibrarius:

Which ones?

F1, MCC seems better.

Here's the full list

https://catboost.ai/docs/concepts/loss-functions-classification.html

[Aleksei Kuznetsov]

Maxim Dmitrievsky:

F1, MCC seems to be better

here's the full list

https://catboost.ai/docs/concepts/loss-functions-classification.html

It's hard to speculate on why they're better.

[iwelimorn]

Maxim Dmitrievsky:

Once again.

Well, yeah, it makes sense sometimes.

A little more staking. Yeah, it makes sense to stack. It's still an open question as to how much.

Great article and titanic work!

Dropped the labels standing after the change of direction in the dataset before feeding them into the mixtures model.

X['labels'].loc[X['labels'].diff(2)!=0] = np.nan
X = X.dropna().reset_index(drop=True)

from observation, more models give positive results.

Iteration:  0 R^2:  0.5260017837414628
Iteration:  1 R^2:  0.6331497253790681
Iteration:  2 R^2:  0.029844101542724655
Iteration:  3 R^2:  0.792134192720112
Iteration:  4 R^2:  -0.5575595243513453
Iteration:  5 R^2:  0.6266442030501214
Iteration:  6 R^2:  0.22435035980633944
Iteration:  7 R^2:  0.38384379495564724
Iteration:  8 R^2:  0.028795249767878173
Iteration:  9 R^2:  0.4976165963899961
Iteration:  10 R^2:  0.38091909530246537
Iteration:  11 R^2:  0.6668471456765452
Iteration:  12 R^2:  0.9173487144971179
Iteration:  13 R^2:  0.31007344321289365
Iteration:  14 R^2:  0.4391449002278133
Iteration:  15 R^2:  0.8643834389201246
Iteration:  16 R^2:  0.1609614914253078
Iteration:  17 R^2:  -0.593998636390909
Iteration:  18 R^2:  0.7253019217609338
Iteration:  19 R^2:  0.11872614816490491
Iteration:  20 R^2:  0.0023909994463473527
Iteration:  21 R^2:  0.6486563192357756
Iteration:  22 R^2:  0.011546691763490213
Iteration:  23 R^2:  0.8085201529889603
Iteration:  24 R^2:  0.9233877589157101
Iteration:  25 R^2:  -0.08743681307491602
Iteration:  26 R^2:  0.6132492793114834
Iteration:  27 R^2:  0.0016322532208268292
Iteration:  28 R^2:  -0.2241780583190326
Iteration:  29 R^2:  0.6739577294046565
Iteration:  30 R^2:  0.43270922254052624
Iteration:  31 R^2:  -0.40571688327667377
Iteration:  32 R^2:  0.8558895655918841
Iteration:  33 R^2:  0.22222950417101828
Iteration:  34 R^2:  0.8718519436072187
Iteration:  35 R^2:  0.3103828388939166
Iteration:  36 R^2:  -0.3274785257455094
Iteration:  37 R^2:  -0.056797883380174774
Iteration:  38 R^2:  0.7372428685425149
Iteration:  39 R^2:  0.46650446544282376
Iteration:  40 R^2:  0.7158974618001703
Iteration:  41 R^2:  -0.08982821839565835
Iteration:  42 R^2:  -0.004541876992860683
Iteration:  43 R^2:  0.39737057218337546
Iteration:  44 R^2:  -0.35131421135378293
Iteration:  45 R^2:  -0.003237278102928842
Iteration:  46 R^2:  0.37272998002935753
Iteration:  47 R^2:  -0.5554473442698793
Iteration:  48 R^2:  0.38766288235168855
Iteration:  49 R^2:  0.8582912360226171

results of the best test in tester and terminal:

Generally, a beautiful work. I used it to test all my targets and threw them in the rubbish)))

[Maxim Dmitrievsky]

welimorn:

Great article and a Herculean effort!

Dropped the labels standing after the change of direction in the dataset before feeding them into the mixtures model.

from observation, more models have a positive result.

results of the best test in the tester and terminal:

In general, beautiful work. With its help I checked all my target ones and threw them in the rubbish)))

yes, you can drop before clustering

thanks for the feedback :)

Z.Ы so you can test all models at once, averaged. Play. Parser for all models has not made yet, still in doubt. But sometimes an ensemble of several models really improves.

def test_models(result):
    pr_tst = get_prices(FULL_DATE, STOP_DATE)
    X = pr_tst[pr_tst.columns[1:]]
    X.columns = [''] * len(X.columns)

    #test the learned model
    p = 0
    for i in range(len(result)):
        p += result[i][2].predict_proba(X)

    p = p/len(result)
    p2 = [x[0]<0.5 for x in p]
    pr2 = pr_tst.iloc[:len(p2)].copy()
    pr2['labels'] = p2
    R2 = tester(pr2, MARKUP, plot=True)

test_models(res)

ZYZY. you can make an enumeration of different combinations of trained models by the same metric R2, as a development of the theme. Then keep the best ensemble. It is even possible through genetics, if there are many models.

[Rasoul Mojtahedzadeh]

Hi Maxim,

Thanks for another nice article! :)

I took EURUSD, H1 data from 2015 to 2020 and did split it into three sets:

1. Training Set: from 2015.01.01 to 2018.01.01

This only be used to train GMM.

2. Validation Set: from 2018.01.01 to 2019.01.01

This set together with Training Set will be used in brute force function to find the best model

3. Test Set: from 2019.01.01 to 2020.01.01

This set only be used to test the best model obtained from brute force algorithm

Below is the result of a backtest on whole data set,

I have double checked my code and yet I might have done something wrong. Anyhow you might have some idea about the results. Best regards, Rasoul

[Maxim Dmitrievsky]

Rasoul Mojtahedzadeh:

Hi Maxim,

Thanks for another nice article! :)

I took EURUSD, H1 data from 2015 to 2020 and did split it into three sets:

1. Training Set: from 2015.01.01 to 2018.01.01

This only be used to train GMM.

2. Validation Set: from 2018.01.01 to 2019.01.01

This set together with Training Set will be used in brute force function to find the best model

3. Test Set: from 2019.01.01 to 2020.01.01

This set only be used to test the best model obtained from brute force algorithm

Below is the result of a backtest on whole data set,

I have double checked my code and yet I might have done something wrong. Anyhow you might have some idea about the results. Best regards, Rasoul

Hi, Rasoul. Try to reduce the training set size. It can depends of different settings, but key trick is that then less train size, better generalisation on new data. In the next article I'll try to explain this effect.

[Evgeni Gavrilovi]

Maxim Dmitrievsky

Can you tell me how I can upload my data via csv file?

I tried it this way, but it didn't load.

The format of the file is:

time,close

2020,11,15,1.3587

2020,11,16,1.3472

def get_prices(START, STOP):
    prices = pd.read_csv('file.csv', columns=['time', 'close']).set_index('time')
    prices.index = pd.to_datetime(prices.index, unit='s')
    prices = prices.dropna()
    pr = prices.copy()
    count = 0
    for i in MA_PERIODS:
        returns = pr - pr.rolling(i).mean()
        for l in range(LOOK_BACK):
            prices[str(count)] = returns.shift(l)
            count += 1
    return prices.dropna()

[Maxim Dmitrievsky]

Evgeni Gavrilovi:

Can you tell me how I can upload my data via a csv file?

def get_prices(look_back = 15):
    prices = pd.DataFrame(mt5.copy_rates_range(SYMBOL, TIMEFRAME, START_DATE, STOP_DATE), 
                            columns=['time', 'close']).set_index('time')
    prices.index = pd.to_datetime(prices.index, unit='s')
    prices = prices.dropna()
    pr = prices.copy()
    count = 0
    for i in MA_PERIODS:
        returns = pr - pr.rolling(i).mean()
        for l in range(look_back):
            prices[str(count)] = returns.shift(l)
            count += 1
    
    prices = prices.dropna()
    prices.to_csv('pr.csv', sep=';')

pr = pd.read_csv('pr.csv', sep=';')

i.e. this is an example of loading data from the terminal, saving it to a file. And then you can use it in colab

[Stanislav Korotky]

Maxim Dmitrievsky:
Hi, Rasoul. Try to reduce the training set size. It can depends of different settings, but key trick is that then less train size, better generalisation on new data. In the next article I'll try to explain this effect.

That's good. It would be nice to see in the article a remark about the scope of applicability of this thesis - in particular, to different IO methods. For some reason they recommend 70/30 for NS. And logically, IMHO, 50/50 should give more stable results.

[Maxim Dmitrievsky]

Stanislav Korotky:

This is good. It would be nice to see a remark in the article about the scope of applicability of this thesis - in particular, to different methods of MO. For some reason 70/30 is recommended for NS. And logically, IMHO, 50/50 should give more stable results.

There is active and passive learning. Passive learning consists in manual markup of data and training on them, in this case there should be a lot of data, but there is a problem of correct markup. That is, the "teacher" has to mark up the data so that it is, conventionally, from the same distribution and generalises well. In this regard, it makes almost no difference what proportion of the trainee\test. It gives you almost nothing, it's just a model check, a check of how well you have labelled the data manually.

In active learning, the model learns to mark up the data in an optimal way. The article is just such a case of partitioning through GMM. That is, both learning with and without a teacher is used. In this case, the model learns to learn from small partitioned data and must partition the remaining data itself in an optimal way. This is a relatively new approach (from around 2017). And I want to look at it in more detail in a follow-up article.

too much "data" in the sentences, I apologise for the tautology )