Download MetaTrader 5

Fast Fourier Transform - Cycle Extraction - page 4

To add comments, please log in or register
Carl Rodriguez
371
Carl Rodriguez  

wow...

Well, there sure is a lot to know about these cycle methodologies. I have read a lot of stuff over the last week or so. I have seen 3 or 4 different software packages that do what we need it to do as far as projecting cycles forward in time. Unfortunately those software packages are either restricted to a club/group for people with mega $$$ to spend on memberships, or the packages for sale are a couple thousand dollars.

I think I'll learn more if I try to do it myself.

So, I now need to find out the math behind the Bartels test, which is a statistical test to see how well an individual cycle fits with past data. The Bartels test will test the cycle for amplitude, phase and frequency and give a % result for fit.

So, the theory is to find the 5 or 6 highest ranked cycles (above 90% bartel rating) and use those to project forward in time.

If anybody has a college statistics textbook, see if the bartels test is in there. I've searched on-line but don't seem to find any references to the math used for it.

Carl

SIMBA
2078
SIMBA  
crodzilla:
Well, there sure is a lot to know about these cycle methodologies. I have read a lot of stuff over the last week or so. I have seen 3 or 4 different software packages that do what we need it to do as far as projecting cycles forward in time. Unfortunately those software packages are either restricted to a club/group for people with mega $$$ to spend on memberships, or the packages for sale are a couple thousand dollars.

I think I'll learn more if I try to do it myself.

So, I now need to find out the math behind the Bartels test, which is a statistical test to see how well an individual cycle fits with past data. The Bartels test will test the cycle for amplitude, phase and frequency and give a % result for fit.

So, the theory is to find the 5 or 6 highest ranked cycles (above 90% bartel rating) and use those to project forward in time.

If anybody has a college statistics textbook, see if the bartels test is in there. I've searched on-line but don't seem to find any references to the math used for it.

Carl

Carl,

Sorry to dissappoint you,using 5 or 6 cycles with high bartels doesn`t work.

I used them trough a software called cycletrends,and the results were not positive...main reason may be that FFT does not work very well for noisy time series data,so even if you limit yourself to the very high bartels cycles,the end result will be far below your expectations.

Potential solutions:

1-Smoothe the data and apply FFT to a "denoised" set.

2-Use Mesa algorithm,which is better than FFT for noise series...AND you have a digital filters generator in this forum,for free,that uses Mesa to find cycles.

3-Use Goertzel algorithm,which seems to be the best one for extracting cyclical components from noisy time series,then apply FFT to project them into the Future...check the following link,very interesting info on this system and on cycles in general....And you can start experimenting with the attached indicator.

Advanced Financial Engineering Mathematics applied to algorithmic trading of stocks and commodities by Dennis Meyers Ph.D

Regards

Simba

Files:
Carl Rodriguez
371
Carl Rodriguez  

Thanks...

Thanks Simba for the info... no disappointment. At the very least I will learn something, and hopefully everyone who is interested in cycle analysis will. I am conviced to the core of my soul that cycle analysis can be used as a major part of any trading system. But as with anything, there is a right way and a wrong way to do it.

The CycleTrends v4 program is not the only program on the block. I've seen another half-dozen that look just as impressive if not more than CycleTrends.

There is a lot to learn, but I think it will be well worth it.

You and I are not that far off track really.

What I have done is write an indicator in MT4 that will remove the noise with an offset moving average as per Brian Millard, and export that to my external windows program that I wrote to continue with the analysis.

I will call FFT from my external program probably several hundred times in order to extract the cycles from the data. The problem that I saw was, the FFT gives different cycles for different data lengths of time. For example, a set of 5 highest amplitude cycles are different for 1000 bars and 2000 bars. So, what I want to do is step my data history starting at 500 up to 5000 bars, by 100, performing an FFT each time, then analyzing each set, throwing out the excess cycles that are close to being equal. That set is the set I wanted to perform a Bartels test on to verify the fit of 5 or 6 highest amplitude cycles. No, its not going to be perfect, but in theory you should at least be able to forecast a direction, and an approximate time for that direction to end.

All the above, I can do with my external-to-MT4 program. But I cannot find out the math behind the Bartels test, that I can put into a program. And, that is the key really. I'm not a math major, so I need to be able to use math functions that are built into most programming languages.

Yes I have seen the Goertzel indicator. I can easily add that to my external program for automatic analysis of any timeframe. I am contemplating that...

Carl

SIMBA
2078
SIMBA  

Practical solution

crodzilla:
Thanks Simba for the info... no disappointment. At the very least I will learn something, and hopefully everyone who is interested in cycle analysis will. I am conviced to the core of my soul that cycle analysis can be used as a major part of any trading system. But as with anything, there is a right way and a wrong way to do it.

The CycleTrends v4 program is not the only program on the block. I've seen another half-dozen that look just as impressive if not more than CycleTrends.

There is a lot to learn, but I think it will be well worth it.

You and I are not that far off track really.

What I have done is write an indicator in MT4 that will remove the noise with an offset moving average as per Brian Millard, and export that to my external windows program that I wrote to continue with the analysis.

I will call FFT from my external program probably several hundred times in order to extract the cycles from the data. The problem that I saw was, the FFT gives different cycles for different data lengths of time. For example, a set of 5 highest amplitude cycles are different for 1000 bars and 2000 bars. So, what I want to do is step my data history starting at 500 up to 5000 bars, by 100, performing an FFT each time, then analyzing each set, throwing out the excess cycles that are close to being equal. That set is the set I wanted to perform a Bartels test on to verify the fit of 5 or 6 highest amplitude cycles. No, its not going to be perfect, but in theory you should at least be able to forecast a direction, and an approximate time for that direction to end.

All the above, I can do with my external-to-MT4 program. But I cannot find out the math behind the Bartels test, that I can put into a program. And, that is the key really. I'm not a math major, so I need to be able to use math functions that are built into most programming languages.

Yes I have seen the Goertzel indicator. I can easily add that to my external program for automatic analysis of any timeframe. I am contemplating that...

Carl

Hi,

For people following this thread and not yet experienced with cycle terminology...High Bartels test`s results for a specific cycle means that this cycle has a high probability of being "non random" ,so,it is fairly probable that we will find it in Future data too,and consequently we can use it to forecast evolution of future prices with a good probability of success(in theory).

Carl,I will try to look for the Bartels test formula and attach here ,in the meantime I have 2 suggestions for you.

1-Check for those cycles that are repetitive among a walk forward run of x/5000 of your dataset(If I understood you well ,this is more or less what you are trying to do with x=500 and a walk forward step of 100)...the common ones to most/all the steps..will,for sure have a high Bartels result.

2-I am working with Goertzel at the moment since I find it extracts the cyclic component of forex data fairly better than Mesa or FFT,and ,for example,while doing an analysis of GBPJPY...720 sets..for D1, H4,H1,M30,M15,M5,M1...I have found that all the datasets(different timeframes) had,among others,the following cycle periods...11,18,29,39...with good amplitudes at ALL timeframes...so,I don`t need a Bartels test to use JUST those periods as the basis of developing my cyclic indicators since I have found them to be permanent among a very wide span of time at different timeframes....My suggestion is that you could do a similar exercise with FFT and your favourite Forex pair,and see if you find anything similarly interesting.

Yes,there is a lot to learn..and,yes,it is worth every hour we invest on it,if I find the Bartels formula, I will post it in this thread.

Regards

Simba

clahn04
150
clahn04  

Simba,

It's been a long time! For a while i didn't see you on the boards. Glad you are doing well. In response to your remarks, when you say "720 sets" among those time frames, what exactly do you mean? I was assuming it was something along the lines of using different amounts of bars per time frame or different time periods with the same amount of bars among the different time frames. I hate to assume though...:-) You know what they say after all....

All the best,

cl

SIMBA
2078
SIMBA  

Assuming

clahn04:
Simba,

It's been a long time! For a while i didn't see you on the boards. Glad you are doing well. In response to your remarks, when you say "720 sets" among those time frames, what exactly do you mean? I was assuming it was something along the lines of using different amounts of bars per time frame or different time periods with the same amount of bars among the different time frames. I hate to assume though...:-) You know what they say after all....

All the best,

cl

Clahn,

What has 16 legs,14 testicles and 2 tiny breasts?

Yes assuming is probably going to distance you from the right answer..so,just asking..What?..will get you the right one fast and easy..Snow White and the 7 dwarfs

Use the Goetzel indicator I posted with 720 periods,not 500.

Regards

Simba

clahn04
150
clahn04  

haha....glad you still have your wit....glad you are still using cycles....i've been using DF's for a very long time now with a lot of success....i just find i can "feel" the markets better....

best,

cl

codobro
53
codobro  

This is an interesting thread! I have considered using FFT for cycle analysis, however the repainting and constant monitoring of the repainting are 2 things I don't like. I did like michal's point, and I wonderd if detrended FFT analysis would be better, but MESA seems more definite.

SIMBA, thank you for the link and indicator, can you say whether the Goetzel indicator repaints? You also mention it is better than MESA, how do you calculate it? We have the digital filter program from newdigital, is there another program you would use?

If you happen to find it I would like to see how bartels are calculated, but you have very easy to understand explainations

Thank you to everyone contributing to this thread

SIMBA
2078
SIMBA  

Facts

Codobro,

The Trouble with Facts is that there are so many of them...;)

I don`t know exactly how Bartels is calculated I just know it is a test for statistical significance,my own experience using just high Bartels cycles has been very negative,probably because I used them with a software cycletrends that used FFT,so there was continuous repainting of even the high bartels cycles.

Mesa has a problem,it is very bad on noisy data,and has an advantage,it is very good on low noise data,while Goertzel is very good on noisy data(see the papers at meyersanalytics for a detailed explanation)...so,since Forex data is noisy,I prefer to use Goertzel and then see confirmation by Mesa just to have a fuller view.

If you check this link SSA-MTM Toolkit you will see the right way to use mesa...what they do is first use SSA(singular spectrum analysis) to eliminate noise,then reconstruct the time series,without noise,then apply Mesa to it...there is exceptional free software there,basically by the Geophysics dpt of UCLA in cooperation with European CERN and a very detailed explanation about how to use the different cycle extraction models in combination to achieve the best results.

I have been unable to install it on my computers(windows xp and windows vista),so,if anybody tries it and knows how to install it,please be so kind to post it here.

OK,back to Goertzel...

The Goertzel indicator doesn`t repaint,you have to learn how to read it,what it gives you are numbers,forget about the time location of the curves you see...

Just point your cursor to the highest indicator top,normal non squared mode,it will give you back 3 numbers: phase,amplitude and period...when you see that period is,for example 0.0250..what do you do?

P.S:You will have to open the data window for this.

Well,actually, what this is telling you is that the highest amplitude cycle has a period equivalent to 0.0250 of the dataset length...since you used 720 periods..that means a cycle length of 18 periods(720*0.0250=18).

So,if we do it for EURGBP H4 now with squared=false and periods=720 we will get a highest peak with the following data:

amplitude=5.2547

phase=2.0174

period=0.0507

That means that the dominant cycle has a periodicity= 720*0.0507 approx 36/37 periods and that this cycle has a phase 0f 2.174..now go back to your digital filters generator and do the same analysis for 720 datapoints...

What you will see is the following:

1-A confirmed peak at 35/36 periods,which is similar enough to our Goertzel 36/37 periods...so,you can be fairly sure that there is a cycle there with that periodicity(36/37) that is not the byproduct of random noise.

2-Several higher peaks at periods from 5 to 28...What Mesa says are important cycles...Goertzel says are just the result of random noise,with the exception of the MESA peak at 23 that seems confirmed by a good amplitude Goertzel of 0.0356*720=approx 25 periods,so,you can use a 25 periodicity too in addition to the 36/37.

3-A very high peak at 106 periods...well,you are using 720 datapoint,so a 106 cycle period looks like probably random...you can then use Mesa with full history and see if it is still there...No,it isn`t,so,you discard it.

Now you know that to use cycles with EURGBP H4 you have to use 36/37,25 and 49/50(you will discover it if you continue the analysis) periods.

Now..a very good question to ponder...Does the phase being at 2.0174 has some relevance? Don`t be shy and contribute your opinions

Regards

Simba

Files:
mesa.gif 306 kb
barnix
1310
barnix  

Download link:

SSA-MTM Toolkit Downloads

0.Install Ubuntu 64 bit

1.Download Spectra_st_linux64.exe

2.Install Grace

3.Run Spectra_st_linux64.exe

3.Unpack soi.zip

Variations of the Southern Oscillation Index (SOI)

between January 1942 and June 1999. Time on the abscissa is

in calendar years, and SOI on the ordinate is normalized by its

standard deviation.

4.File/Data, Read vector, soi, Read File

5.Plot,Vector

http://www.atmos.ucla.edu/tcd/PREPRINTS/2000RG.pdf

Files:
soi.zip 4 kb
1_1.jpg 50 kb
2_1.jpg 55 kb
To add comments, please log in or register