Adaptive digital filters - page 9

New comment
 
NorthernWind:
Prival:
There was some noise in the residuals, but not Gaussian. Strange noise of +-1 pip and nothing else, a few rare spikes of 2-5 pips plus 1 gap was 40 pips (I was specifically looking for a week with a good gap).

And me and Mathemat and someone else saw this noise on ticks. Moreover, on the ticks it is clear that +-1 points have higher probability of the reverse movement than its continuation. Unfortunately, this regularity is inside the spread. And it is not high.

And the fact that it appeared after the processing is interesting.

You analysed the returns, I've seen everything you've posted. Reread it several times. I did it differently. I took all ticks for the week, removed trend y(x)=a*x+b. I searched for an oscillatory process in the residuals. Calculated the ACF. And using Kalman I was removing this oscillation, and so on, until I got almost similar to returnee (which is almost what I got). I was looking for all the components of the process, I wanted to approximate the dimensionality of the model (how many significant oscillations there are in a week)
 
grasn:

to Piligrimm

About polynomials - you are profoundly mistaken. They can be effectively used for prediction...

I tried it with all available in MathCad and MathLab and I wasn't satisfied with the result.

PS: your avatar isn't the universal "OM" sound by any chance?


I can only repeat, if something doesn't work for you, that doesn't mean it can't be at all. Keep looking, I'm sure from my practical experience that using polynomials in forecasting is just as effective as neural networks, although much more time consuming. About the avatar - you're right.
 
Prival:

to grasn and rsi and all

I want to explain, because you have repeatedly attacked me for the slogan "Number rules the world". I brought it so that you could pay attention to it. You're smiling, but I don't think you fully understand what I'm talking about. I suggest that you make a very simple experiment. Suppose the price changes as a sine wave. Draw a sine on a piece of paper and put two reference points on it. Like this one.

Fig.1

That is, we have taken the Close minutiae and assume that it is a correct digitisation, see fig. 1. (blue marks). Everything looks nice and correct, and now think, if the first tick has not come exactly at the end of the minute, but for example 2 seconds before the end of the minute, + the second tick was not at the end of the minute, but at the beginning. See Fig. 2 for the result (the blue counts stand differently on the time axis). And it turns out that the sine waveform has changed, the frequency is wrong, the phase is wrong, and in general, everything is bad .....

Fig.2

Who can tell me which sine waveform is real? Or can you also give me a prediction, what will be the number at the next Close (even if it's strictly a sine wave)?

How many copies are already broken in the analysis of the Y axis (prices), and the X axis (time) is forgotten. Or they think it is OK. They take Close and go ahead. And as a result .... long and persistent searches and conclusions DSP does not work.

And let's write this acronym differently, so DSP. (DSP !!!) the only thing left to do is to define what the signal is. Do we not know how to process numbers like adding, subtracting, multiplying and dividing, what else do we have? Well, who here does not know DC, these complex operations.

You may still wonder why many DSP methods do not produce the results you expect from them. Maybe proper X-axis processing will improve many digital processing methods, starting with the simplest MA? And for the signal (the useful component that moves the market) too, not much is known, what I read is the same philosophy :-(.

And unfortunately money rules the world, not numbers.

Although I still undertake to prove to anyone (you can buy me a brandy, because I already owe many people :-)) that if between that "true" price, which no one knows, there is someone who can control the sampling rate, then he can do anything he wants. From an ordinary 100 MHz sine wave, you can make any curve you see on the screen. At least remember the movies, where the wheels go backwards and the cart goes forwards :-).

And that's why that beautiful phrase, "a number rules the world and the name of this number is sampling rate". It's not so bad. After all, by controlling this number, you can control the curve on the screen, i.e. the value (price) of money. And if money rules the world, then by controlling it, I will rule the world.

Z.U., what's that cartoon, "beaver-breath" I really want to see :-). And you can not get rid of me so easily, like Prival in the mire, do not wait :-).

And in the light of what I have written above, for me any DC will never be that almighty GOD who can slip me any figure at any time. They will be weak :-) It's hard to take a break from the fighting course :-)

When I started to study Forex market in 2000 I also thought that based on my previous experience in modeling and forecasting of time series I should forecast both price and time to get accurate and objective picture of processes. But after few years of experiments I realized that time forecasting in Forex market is much more difficult than price and computer resources available to me are not enough for adequate forecasting, while for relatively normal trading the only thing you need is to know the price. So, looking at your sinusoidal graphs I can say that in principle it is not a problem that phase and frequency have changed, the amplitude is the same in both, and based on that you can predict the direction of price movement, if nothing more complicated and not worth bothering about it.
 
rsi:

Prival, Mathemat, I'm afraid to get annoyed again, but I have to say it again - there is virtually no noise in the quotes - that is the input signal. You are trying to use the tools of mathematical statistics (filtering is the same). Statistics of what? Statistics, laws of distribution, their moments of different orders refer to random variables (processes). If you get a tick, is that a signal or noise? I argue that it is a signal, because with this data you can give a buy or sell order, and it will be executed (all other general conditions being equal). Yes, it is difficult to predict what the next price value will be, so I like to think there is a random component there and a non-random component that can be identified and then extrapolated-predict. And it is not random, it is just unknown. Or, if you like, all random - without dividing it into additive components. What are you going to separate? The same Kalman filter will filter out a very definite component - defined by your own model in the form of a smooth analytic function. Do you know it? I don't. You're trying to identify the dynamic properties of the market, and applying a physical analogy is, I'm afraid, also futile: you can find minute candles with an amplitude greater than a figure, as well as gaps, which indicates that it is practically inertial-free.

I completely agree, there is neither noise nor random component in quotes; there is only distortions in signals due to DC filters and delays and signal information losses due to connection and prehistoric method of candlestick formation. As for AC distortions, they can be partly solved, if an adequate model is built, it will take them into account. Unfortunately, in this situation we will have to retrain the model for each brokerage company, it will not be universal. As for informativity losses, we can use ticks provided by Reuters and other news agencies, in general, it is not a stumbling block, the main thing is to find an effective strategy, the rest is a technical matter.
 
Prival:
NorthernWind:
Prival:
There was some noise in the residuals, but not Gaussian. Strange noise of +-1 pip and nothing else, a few rare spikes of 2-5 pips plus 1 gap was 40 pips (I was specifically looking for a week with a good gap).

And me and Mathemat and someone else saw this noise on ticks. Moreover, on the ticks it is clear that +-1 points have higher probability of the reverse movement than its continuation. Unfortunately, this regularity is inside the spread. And it is not high.

And the fact that it appeared after the processing is interesting.

You analysed the returns, I've seen everything you've posted. Reread it several times. I did it in a different way. I took all ticks for the week, removed trend y(x)=a*x+b. I searched for an oscillatory process in the residuals. Calculated the ACF. And using Kalman I was removing this oscillation, and so on, until I got almost similar to returnee (which is almost exactly what I got). I was looking for all the components of the process, I wanted to approximate the dimensionality of the model (how many significant oscillations there are in a week)

There is no perfect filter, this "noise" is +-1 pips, it's the distortion that occurs during processing, due to the fact that the resolution of the computer is finite, the filter is not perfect, etc., it's not noise in the original signal.
 
Piligrimm:
There is no perfect filter, this "noise" is +-1 pips, it is distortion that occurs during processing, due to the fact that the resolution of the computer is finite, the filter is not perfect, etc., it is not noise in the original signal.


That's kind of what I was talking about. It's measurement noise (quantization and sampling noise) .

What does the "OM" universal sound symbolise. Enlightenment.

 

Piligrimm, will you allow it?

It's the noise of the universe that our normal senses don't allow through their filters. And at the same time the signal that the practitioner must emit in order to enter into stochastic spiritual resonance with the universe. Shudko :)

 
Piligrimm:
As for the loss of informativeness, you can use the ticks supplied by Reuters and other news agencies, this is not a stumbling block...
:-)
 
Mathemat:
... When I talk about errors, I usually talk about errors of prediction or approximation. Prival talks about errors of observation and measurement. This is quite natural in terms of his speciality. But these are quite different errors. Nevertheless this point of view has a right to life, although in my opinion it is artificial...

Completely agree with you. Regarding measurement errors, I added PS in my previous post. And concerning forecast errors - it must be, in my opinion, the subject of research, and criterion for trade decisions, and that random variable, to which statistical methods and exactly the Bayss approach should be applied. And not to price or returns - that's good for entering and that's after pre-processing. Prediction probabilities have a right to exist and everything that has already happened has probability equal to one.

MTS doesn't have to be implemented with neural networks so disliked by Prival, but we have to understand that it's not about filters (it's unclear what they separate from what), but about DataMining, clustering and other similar modern technologies of multivariate data analysis (I think Piligrimm mentioned MSUA here), that allow to identify latent patterns in time series.

In general, I have the feeling of a Lefty trying to make a point: "The English don't clean their guns with a brick!" :-)

 
Prival:

I would love to help. But unfortunately I can't read MQL code as freely as MathCad where formulas are written the way we are used to seeing them in books. The only thing that seems to me (though I'm not sure) is using one of regression types, to make it clearer

There is a linear regression like y(x)=ax+b. You can calculate coefficients a and b in different ways, you can use ANC (seems not to be used there), and you can use recursion, but to understand it you need to clearly understand the loops (I get confused there, where, what, why is calculated). Most probably there is a non-linear regression, because there are some if() while calculating + type of regression equation itself is not clear, how many coefficients there are.

In general, almost all indicators can be considered as digital filters, the MA is a digital filter. The word adaptation usually means that some parameters (coefficients in the filter gut) have to change depending on the characteristics of the input signal. Therefore first of all I would refer AMA, FRAMA and similar adaptive digital filters (averaging parameter (n) changes depending on input process variance estimation), almost all FFT, wavelet filters that use threshold processing (trying to match TF parameters with a spectrum of input desired signal).

But SATL, FATL are not adaptive, because TF coefficients were calculated once at design stage to match the transient response of the filter with the spectrum of the input signal (AFR and IFR), and during operation these coefficients do not change. These are the so-called matched filters. But there is an ideal, what is called in DSP optimum filter, to build it is difficult, but possible. For this you need to know spectra of useful signal and noise.

I don't know, if I helped you or confused you :-), but in any case good luck.

12345678910111213141516...41
New comment
Reason: