Machine learning in trading: theory, models, practice and algo-trading - page 3730

[[Deleted]]

Has anyone thought about the effect of "amplifying" the high-frequency components in the time series before partitioning and training? In terms of how it will affect the model itself and its generalisation properties.

Usually they do the opposite - smooth the series.

A simple example:

def amplify_high_freq(df: pd.DataFrame, long_period: int = 100, amplification_factor: float = 2.0) -> pd.DataFrame:
    """ Amplifies the high-frequency components in the 'close' column by adding the amplified residuals from the long term EMA to the price. :param df: Source DataFrame with prices (index is time, column is 'close').
:param long_period: Period for the long term EMA (low frequency component).
:param amplification_factor: Amplification factor for high-frequency residuals (k > 1). :return: DataFrame with new column 'close_amplified'. """"
    if 'close' not in df.columns:
        raise ValueError("The DataFrame must contain a 'close' column.")
        
    # 1. Calculation of the low-frequency component (long-term smoothing)
    # We use the EMA as it tracks the trend better than the SMA.
    df['EMA_long'] = df['close'].ewm(span=long_period, adjust=False).mean()

    # 2. High-frequency component extraction (Residuals)
    # Residual = Price - Trend
    df['High_Freq_Component'] = df['close'] - df['EMA_long']

    # 3. Applying the gain and creating a new row
    # P_amplified = P + k * (P - EMA_long)
    df['close_amplified'] = df['close'] + amplification_factor * df['High_Freq_Component']

    # Remove the temporary columns to keep it clean
    return df.drop(columns=['EMA_long', 'High_Freq_Component']).dropna()

Before and after amplification:

You can amplify different components and see how it affects learning. You can do it without MA.

[Aleksei Kuznetsov]

If we substitute k=2, the formula can be recalculated in
close_amplified = close + 2 * (close - EMA )
= close * 3 - EMA * 2

I.e. price is EMA with some (probably selectable) coefficients. This way the essence of the transformations is clearer.

If then the close_amplified differences are fed to training, the differences of MAs (100) will be very small (1/100 of Close deltas), i.e. there will be almost Close differences.

How the training will be affected by MO is not yet clear. Probably, as well as differences of MA(100). Theoretically. If you compare in practice (with and without) - it will be interesting whether the theory will agree.

[[Deleted]]

Forester #:

If we substitute k=2, the formula can be recalculated in
close_amplified = close + 2 * (close - EMA )
= close * 3 - EMA * 2

I.e. price is EMA with some (probably selectable) coefficients. The essence of the transformations is clearer this way.

If then the close_amplified differences are fed to training, the differences of MAs (100) will be very small (1/100 of Close deltas), i.e. there will be almost Close differences.

How the training will be affected by MO is not yet clear. Probably, as well as differences of MA(100). Theoretically. If you compare in practice (with and without) - it will be interesting whether the theory will agree.

No, I'm training with the original dataset but with labels that are marked on the converted one. So far, I can only see that there is more noise in the trades and the forward is more even.

[[Deleted]]

Adaptive amplification (we change the multiplier depending on the volatility) gives good results.

[Rorschach]

Maxim Dmitrievsky #:

Has anyone thought about the effect of "amplifying" the high-frequency components in the time series before partitioning and training? In terms of how it would affect the model itself and its generalisation properties.

Usually they do the opposite - smooth the series.

A simple example:

Before and after amplification:

You can amplify different components and see how it affects learning. You can do it not through MA.

It's kind of like fractional differentiation, isn't it?

It's probably better to amplify the midrange frequencies.

[[Deleted]]

Rorschach #:

It's kind of like fractional differentiation, isn't it?

It's probably better to amplify the midrange frequencies.

You know, like amplify some frequencies to emphasise label markings on certain strategies. So that there are more examples. Not much improvement, more like variation of final models gives.

It doesn't improve inherently flawed strategies :)

[[Deleted]]

The question was raised here who distinguishes between real and SB rows. You can try to do it with the help of MO, classifying real and SB series (on the basis of real ones). It is called Adversarial validation.

If the accuracy is around 0.5, then the model cannot distinguish real from SB.

I did a test, comparing real to SB:

{'learn': {'Accuracy': 0.6625488511038815, 'Logloss': 0.6129790273501617}, 'validation': {'Accuracy': 0.6570160003273724, 'Logloss': 0.6187963401083126}}

Accuracy > 0.5, model sees the difference.

Comparing 2 random walks:

{'learn': {'Accuracy': 0.5185200343797324, 'Logloss': 0.6920313398646679}, 'validation': {'Accuracy': 0.5128000491128982, 'Logloss': 0.6926835526214326}}

Accuracy ~0.5, no difference, model can't tell the difference.

[[Deleted]]

The problem arises by asking about the nature of the SB. If it is non-stationary, then a difference in the drift of the mean will be found even between two SBs. If increments, the difference may be useless for trading. If higher order increments, what kind of increments? :), etc.

It seems most logical to compare the original series with SBs, but then even different SBs will be different. But each has its own properties (drift), so the model will learn to distinguish the SB of the initial series from just SB, which can be useful as a filter (when the initial series is not similar to itself at the moment, but to another SB, then do not trade, for example) :). But even then you will get a binding to some left SB. Randomness multiplied by randomness will give randomness again.

If in the economic sense, the SB inherent in the original series cannot have certain characteristics due to fundamental reasons, so comparison with any other SB may show that the original series deviated from its SB at the moment 😁 I'm tired of philosophising. In short, you can just teach the model to understand which SB is in front of it now - familiar or not.

[[Deleted]]

I managed to make an augmentation based on yesterday's reasoning, which gives such curves:

And even such (reduction of traine interval by 2 times):

[Aleksei Kuznetsov]

Maxim Dmitrievsky #:

Got an augmentation based on yesterday's reasoning that gives these curves:

And even such (reduction of traine interval by 2 times):

What about the signal? It's beautiful in the tester. I wonder what it'll be like in real life.