Discussion of article "Grokking market "memory" through differentiation and entropy analysis" - page 9

[Maxim Dmitrievsky]

clemmo:

I am getting these errors:

1. It seems EA is attempting to sell but placing stops as though it was a buy? 
2019.07.25 04:35:35.174 Trades '666': failed  market sell 0.10 USDCHF sl: 0.98139 tp: 0.99039 [Invalid stops] 

2. The EA removes itself on the formation of a new bar or candle.

3. The EA can only place sell trades (unless this is by design during the optimization phase?)

4. EA uses a different trade volume for NZDUSD only?

update0: after further testing I can say the problem of auto-removal is related to the custom settings from the hyperparameters screenshot. I will attempt to isolate which one.
update1:It is related to adjusting the history depth to 1500.  I will attempt to see if different values create different results. I am testing on M1 and M15 and have sufficient price history.
update2: EA will remove itself if History_depth  1027 >= ||<= 956 

5. After some time the entropy window will shrink to a small size as in the screenshot.

update3: I can suggest the problem with sell-only is related to too low a timeframe used in testing (M1). This generates very large entropy values which are out of bounds? Testing on M15>= creates both buy and sell orders. 

update4: I can suggest the entropy window size issue is related to the initial size of the chart window. If it is scaled full screen the window appears OK. 

update5: After attempting to modify magic numbers now all charts remove on the next bar. I've tried restarting the terminal, using new charts. I'm not sure what else to try, there are no errors in the log. I cannot test further?

Hi, I think there are some problems with normalization for entropy, that's why it change its values over time. Maybe a little later I fix this an include another indicator About another problems - I think I don't fully understand. Maybe have some errors in terminal log?

[kttongsg]

Hi Maxim,

I tried compiling the fractional_entropy_trader but the following erro message appeared:

'virtual_optimizer' - function already defined and has different type Auto_optimizer.mqh  line 47  col 18

Can you advise how I can fix this problem?

Thanks

Daniel

[Alexander]

Very complicated to understand,

What basic knowledge do you need to deal with this topic?

Does the whole thing have anything to do with integral calculus?

[Carl Schreiber]

Alexander:

Very complicated to understand,

What basic knowledge do you need to deal with this topic?

Does the whole thing have anything to do with integral calculus?

1. There is only summation and multiplication!
2. Instead of mathematics, just try to understand the programme flow.
3. Entropy is simply a measure in physics for disorder - here: high entropy = many price fluctuations.
   

[Rafael Campagnoni Prado Rocchi]

kttongsg:

Hi Maxim,

I tried compiling the fractional_entropy_trader but the following erro message appeared:

'virtual_optimizer' - function already defined and has different type Auto_optimizer.mqh  line 47  col 18

Can you advise how I can fix this problem?

Thanks

Daniel

Hi, I just read the article and I am about to compile, but for a quick help to you: Locate the 2 function definitions, rename the conflicting one, this should be enough for you to compile it. 

[Oleg Mironov]

Good afternoon,

I would like to return to the topic

Listing 5.4. Finding the minimum value of d that successfully passes the ADF statistical test

def plotMinFFD():
from statsmodels.tsa.stattools import adfuller
path,instName='./','ES1_Index_Method12'
out=pd.DataFrame(columns=['adfStat','pVal','lags','nObs','95%
conf','corr'])
df0=pd.read_csv(path+instName+'.csv',index_col=0,parse_dates=True)
for d in np.linspace(0,1,11):
df1=np.log(df0[['Close']]).resample('1 D').last() # понизить до
# суточных наблюдений
df2=fracDiff_FFD(df1,d,thres=.01)
corr=np.corrcoef(df1.loc[df2.index,'Close'],df2['Close'])[0,1]
df2=adfuller(df2['Close'],maxlag=1,regression='c',autolag=None)
out.loc[d]=list(df2[:4])+[df2[4]['5%']]+[corr] # с критическим
# значением
out.to_csv(path+instName+'_testMinFFD.csv')
out[['adfStat','corr']].plot(secondary_y='adfStat')
mpl.axhline(out['95% conf'].mean(),linewidth=1,color=
'r',linestyle='dotted')
mpl.savefig(path+instName+'_testMinFFD.png')
return

Python code from the book - this function determines the minimum degree of fractional differentiation,who is familiar with Python, translate. In my opinion

it is exactly what is missing in the article for practical application of the method ( or send a lazy person to the right address - preferably with the function).

[Maxim Dmitrievsky]

Олег:

Good afternoon,

I would like to return to the topic

Listing 5.4. Finding the minimum value of d that successfully passes the ADF statistical test

Python code from the book - this function determines the minimum degree of fractional differentiation,who is familiar with Python, translate. In my opinion

it is exactly what is missing in the article for practical application of the method ( or send a lazy person to the right address - preferably with the function).

The Dickey-Fuller test may be useful for some statistical studies, but in the article the value of d is searched in the optimiser, plus the proposed version of the algorithm is constantly being retrained, so it would not be of much use here, in my opinion.

https://www.mql5.com/en/code/13072

[Oleg Mironov]

Maxim Dmitrievsky:

The Dickey - Fuller test might be useful for some statistical studies, but in the paper the value of d is searched in the optimiser, plus the proposed version of the algorithm is constantly being retrained, so it wouldn't be of much use here, in my opinion

I wanted to use automatic determination of d at the training stage with subsequent automatic processing of both training sample data and real trade data for a bunch of subsequent algorithms (if we are talking about machine learning methods - well we are not, I mean...).

[Maxim Dmitrievsky]

Олег:
I wanted to use automatic detection of d in the training phase followed by automatic processing of both training sample data and real trading data for a bunch of subsequent algorithms (if we're talking about machine learning methods - well we're not, I mean...).

I gave a link above, it seems to work

[Oleg Mironov]

There is a Dickey-Fuller test in MQL, but it is a test of a series for stationarity. There is an example of preprocessing by the Box-Cox method on Hubre, for which the code of automatic determination of the lambda parameter - which has a similar value to the discussed d - was given in an old article on the site.

The Box-Cox method on forex does not always lead to stationarity of the series (almost always not to the end) and additional processing is required. It is as an attempt to replace the BC method with fractional differentiation (with auto determination of the best value of d) that my question should be looked at.

I have full respect for you and the article you have written, my question is really unnecessary within the framework of the article, it is already as a plan for the future....