Random Flow Theory and FOREX - page 14

New comment
 
Yurixx:


Well, in my opinion, this is too rigid a statement. In fact, there are no studies that give statistics on the lifespan of models. Moreover, there are no data on the amount of information (= lag time) needed to recognise the model. Even those who introduce and use these models prefer not to conduct such studies or not to publish them. Obviously, it is believed that if the strategy has a positive mo, then the model is recognised before the probabilities are equalised.

And there are such strategies, live ones. Look at Better. His EA actually does what I wanted to implement in mine - it recognises pivot points and enters at the start of a wave. And it moves up as well as down. Here you have a forecast and before the recognition.

...

So, we need to clearly estimate the resources both in terms of memory and computing cycle time. Otherwise we may get one computation cycle of 5-10 hours. What news then? Only left to play for days or weeks. :-)


Of course it is in fact a hypothesis and it is not based on statistics (which cannot be an argument for such claims because one example is enough to disprove them) but on systemic considerations, i.e. equilibrium considerations. ButBetter's advisor isn't an argument either - little observation time. However for us, moneyless dons (and there are few billionaires among us :), there is good news: the fact that our actions do not affect the market in any way by orders of magnitude simplifies the task. Except when our model coincides with the model of one of the whales :)

The problem of the volume of calculations will be solved by creation of a cluster, I think for the work on the ready strategy it will not be a problem to recruit participants. Well, one computer can hopefully handle one filter.

 
lna01:


The good news is that our actions do not affect the market by orders of magnitude, which simplifies the task. Except when our model coincides with that of some whale :)

The problem of the amount of computation can be solved by setting up a cluster, I think it will not be a problem to recruit participants to work on a ready-made strategy. And one computer can hopefully handle one filter.


I wonder how the fact that "our actions have no impact on the market" simplifies the task "by an order of magnitude"? It seems to me to be the opposite. If our actions had an impact on the market (in a desirable way), it would not only simplify but even eliminate the problem. :-) And we would not need any robots then.

As for the cluster, you've upset me. Even my personal involvement I think will not save the situation. There are only 8 billion people on Earth, probably no more computers. How on earth can you make a cluster if you have 10-100 billion filters to compute? :-)))

 
Yurixx:
I think the opposite is true. If our actions influenced the market (and in a desirable way) it would not only simplify it, it would eliminate it as a problem. :-) And we wouldn't need any robots then.

In a desirable way? As soon as our actions start to noticeably affect the market, the market will start working against us personally. Including central banks and intelligence agencies. Soros was successful in his time as a one-time action, and you want to have a model that allows you to pump all the money in the world? Well, let's say it works. And who will need the money after that? :)

Purely technically all the constants will just start turning into variables.

With a billion participants the approximation of no feedback will stop working. And 100 is not an unattainable figure.

 

Калмановская фильтрация

Introduction

At present, adaptive filters are widely used in which incoming new information is used for continuous correction of a previously made estimation of the signal (target tracking in radiolocation, automatic control systems, etc.). Of particular interest are recursive adaptive filters, known as Kalman filters.

These filters are widely used in control loops in automatic regulation and control systems. This is where they originated, which is evidenced by such specific terminology used in describing their operation as state space.

The task of parameter estimation

One of tasks of the statistical decision theory of great practical importance is the task of evaluation of state vectors and parameters of systems which is formulated as follows. Suppose we need to evaluate the value of a vector parameterX that is not directly measurable. Instead, another parameter Z, which depends on X, is measured. The task of estimation consists in answering the question: what can we say about X knowing Z. In general, the procedure for optimal estimation of the vector X depends on the quality criterion adopted for the estimation. For example, a Bayesian approach to the parameter estimation task requires complete a priori information about the probabilistic properties of the estimated parameter, which is often impossible. In these cases, the Least Squares Method (LSM) is used, which requires much less a priori information. Let us consider an application of LOC for the case when the observation vector Z is connected with the estimation vector X by a linear model and there is an interference V, uncorrelated with the estimated parameter:

(1)

where H is a transformation matrix describing the relationship between the observed values and the estimated parameters.

The estimate X minimising the square of the error is written as follows:

(2)

Let the disturbance V be uncorrelated, in which case the RV matrix is simply a unit matrix, and the equation for the estimate becomes simpler:

(3)

Filtering problem

Unlike in the parameter estimation problem that have fixed values, in the filtering problem we need to evaluate processes, i.e. find current estimates of time-varying signal distorted by interference, and thus unavailable for direct measurement. In general, the type of filtering algorithms depends on the statistical properties of the signal and the interference. Let us assume that the desired signal is a slowly varying function of time and the interference is uncorrelated noise. We will use the least squares method, again due to the lack of a priori information about the probability characteristics of the signal and the interference.

Essentially, we have implemented a parametric estimation process based on an autoregressive model of the signal generation process.

Formula 3 is easily implemented programmatically by filling in the matrix H and the observation column vector Z. Such filters are called finite-memory filters because they use the last k observations to obtain the current estimate Xpoc. At each new observation cycle a new observation is added to the current set of observations and the old one is discarded. This process of obtaining estimations is called a sliding window.

Filters with growing memory

Filters with finite memory have one major disadvantage - after each new observation a full recalculation of all stored data must be performed again. Besides, calculation of estimates can be started only after the results of the first k observations have been accumulated. That is, these filters have a long time of transient process. To solve this drawback we must change from a filter with permanent memory to a filter with growing memory. In such a filter the number of observed values used for estimation must coincide with the number n of the current observation. This allows obtaining estimations starting from the number of observations equal to the number of components of the estimated vector X. And this is determined by the order of the model adopted, i.e. how many terms from the Taylor series are used in the model. At the same time with the increase of n the smoothing properties of the filter are improved, i.e. the accuracy of estimations is increased. However, direct implementation of this approach is connected with increasing computational costs. Therefore, filters with growing memory are implemented as recurrent filters.

A filter with growing memory has one important feature. If we look at formula 6, the final estimate is the sum of the predicted estimation vector and the correction term. This correction is large for small n and decreases as n increases, tending to zero for . That is as n increases the smoothing properties of the filter grows and the model inherent in it starts to dominate. But real signal can fit to the model only in separate areas, so the prediction accuracy worsens. In order to struggle against it starting from some n a prohibition on further reduction of the correction term is introduced. This is equivalent to change of filter's bandwidth, i.e. at small n the filter is wider (less inertial), at greater n it becomes more inertial.



Compare Figure 1 and Figure 2. In the first figure, the filter has a large memory, with good smoothing, but due to the narrow bandwidth, the estimated trajectory lags behind the real trajectory. In the second figure, the filter has a smaller memory and is less smooth, but tracks the real trajectory better.

Having read the branch from the beginning and having glanced through the content of the article excerpted above, it becomes clear that the method of processing of BPs using Kalman filters is based on an a priori statement about presence of deterministic trends in the analyzed BPs.
Let us define:
1. Let there be a normally distributed random variable with zero expectation and zero or sign-variable correlogram. By integrating it, we obtain an analog of a time series. If the series is long enough, we can mark out any lengthy stretches of directional price movement. Let us call such a trend a stochastic one. Based on the principal impossibility to build the TS, that could make the profit on such series on the long time interval, we conclude that it is impossible to detect the stochastic trends with the help of casual systems.
2. Suppose there is a normally distributed random variable with zero expectation and positive correlogram.
If the series is long enough, we can mark out any lengthy periods of the directed price movement. Let us call such a trend a deterministic one. Deterministic trend can in principle be detected using lowpass digital filters or their derivatives. For example, crossing of two moving averages with different periods is nothing else but an approximation to the derivative of a smoothed time series. It is clear that this works as mathematics requires: a derivative greater than zero means that the function is increasing, less than zero means that the function is decreasing. But only few people know that it works only for series with a positive FAC, and ALL currency series in all TFs have a negative FAC! And, consequently, the method does not work in the market or it works, but by accident...
Main Problems of Time Series Analysis
 The principal differences between a time series and a sequence of observations that form a random sample are as follows:
- first, unlike the elements of a random sample, the members of a time series are not independent;
- Second, the members of a time series are not necessarily equally distributed.

Generally, 4 factors are distinguished under the influence of which values of time series are formed.
1. The long-term ones that form the overall (long-term) trend in the examined attribute. Usually this trend is described by a non-random function (whose argument is time), usually monotonous. This function is called a trend function or simply a trend.
2. Seasonal, which forms periodic fluctuations in the analyzed attribute at certain times of the year. Since this function is to be periodic (with periods multiple of "seasons"), its analytical expression involves harmonics (trigonometric functions) whose periodicity is usually determined by the essence of the task.
3. Cyclical (conjunctural) forming changes of the analysed attribute caused by long-term cycles of economic or demographic nature (Kondratieff waves, demographic "pits", etc.).
4. Random (irregular), which cannot be counted and recorded. Their impact on the formation of time series values is precisely what determines the stochastic nature of the elements of the series, and hence the need to interpret the members as observations made on random variables. Let us denote the result of the impact of random factors by means of random variables ("residuals", "errors").

Of course, it is not necessary that all four types of factors are involved simultaneously in the process of forming the values of any time series. Conclusions as to whether the factors of a given type take part or not in the formation of the values of a particular time series can be based both on the analysis of the substantial part of the problem, and on a special statistical analysis of the investigated time series. However, in all cases, the participation of random factors is assumed. So, in general terms, a data model (with an additive structural scheme of the influence of factors) looks like a sum of all or some of the factors.
There are criteria according to which BPs are referred to one or another type. According to them, FX type BPs do not contain deterministic trends, do not have a cyclic component, do not have a seasonal component... shall I go on?
Now a question to you, Prival: what are we going to detect with a multivariate, super-duper sophisticated Kalman filter?



 

The filter can be set to any process not necessarily deterministic. You can set 1 filter to (1. Long-term, ....) 2nd filter to (2. Seasonal,...) 3rd filter to (3. Cyclic...) 4th filter to (4. Random (irregular),...) etc.

I do not see a fundamental difference in analyzing and deciding where the quotes flow and the plane movement moves. An aircraft can perform a dive, a caber, a snake, a cobra and . ... any type of manoeuvre at any time (as the pilot pleases). And when he knows that the missile is launched, believe me, he is spinning like hell in a frying pan.

The task of Kalman filters (not one !!! filter for all cases) but several. 1 Determine - type of manoeuvre and time of its start. 2 Predict where the plane will be in some time, to attack it as effectively as possible (take profit :-)).

Therefore it is necessary to build not 1 filter, but several, each of which is set up for its own model (flat, trend).

The simplest variant is 1 filter that works with the trend and another one with the flat. The Kalman filter algorithm has a procedure that allows you to determine which filter is better now.

I hope you will not argue that there are systems that work well in a flat trend but fail in a trend and vice versa, they work well in a trend but not in a flat. So Kalman filter can tell you when you need to switch.

P.S. I will still try to build Kalman filter, build different movement patterns and implement TS based on them. The Kalman filter has a unique feature that standard indicators do not, it is able to predict + let's say I build 3 filters flat, trend, gap. At some section none of these filters works, I analyze this section and build the 4th filter, etc. Not a dumb search from -100 to +100 (perceptor), but meaningful training of the system.
 

to Prival

I hope you are not setting up the Strategic Defence Initiative against quotes as an answer to the imperialist vultures? :о)))

 
Prival:
P.S. I will still try to build a Kalman filter, build different movement patterns and implement a TS based on them. For example, I will build 3 filters: Flat, Trend, Gap. At some section none of these filters works, I analyze this section and build the 4th filter, etc. Not a dumb search from -100 to +100 (perceptor), but a sensible learning of the system.


I recommend: don't try too hard, you may break your hand. At the beginning build only one filter, for flat or trend. While you are doing it, you will understand a lot about Forex, and it will be easier to move on. It is especially important to understand the following: no real trader aims to make a profit. of any situations. Everyone finds a very limited list for himself and works only on these particular patterns.

If you manage to create a filter that will statistically distinguish just one profitable situation that is not too rare, say, once a day, then it is a cornucopia that everyone here dreams of.

 

Prival, the topic is interesting, I got acquainted with your articles - sensible thoughts (though in my opinion it is better to model and filter trajectory in Cartesian coordinate system, but it is not relevant). However, some visual similarity of AFK Doppler trajectory of airborne target and price movement seems to me casual and not worth attention, because the nature of the processes is completely different. The dynamic properties of the aircraft and the quotes, alas, have little in common. As one thread here compared the price movement to the flight path of a fly, not of an aeroplane. But even a fly is not capable of geese, for example.

At the same time, if you define a linear regression of price as a signal, you can "catch" a few hundreds of pips over a year, or even zero - if the coefficient of the linear regression equals zero. The same can be said about stochastic resonance - we may detect a slow tendency at the expense of a high-frequency one with larger amplitude, but what can we earn from it? That said, sometimes price movements of several figures occur in just a couple or three days. And there may be dozens of such movements in a year for almost every pair. And in small timeframes, there may be hundreds of movements of half a figure or so in a year, and thousands of movements of 20 pips. So the question is: do we need a signal or noise?

From time to time I repeat on the forum my thesis that it is my belief that there is no point in dividing price into signal and noise. It is a time series describing a non-stationary process in a super-complex nonlinear dynamic system. It is a scalar that we observe as a generalised measure of millions of primary parameters. There is no need to divide it into components - there are none. Therefore application of traditional filtration methods (they seem already discussed on this forum at various times) is not adequate to the task.

Are there adequate methods? Yes, there are. On the eve of the championship on the suggestion of klot and other forum participants I got interested in artificial intelligence. The term, unfortunately, was flouted, and some publications here on the site even devalued it in my eyes, so for a long time I shrugged off the "new-fangled" theory. However - no, anti-advertising is also advertising, at some stage Perceptron's propaganda also played a positive role in attracting attention to the subject. But after reading the literature I saw how far the so-called "weak AI" has gone compared to, for example, thirty years ago, when I first heard about it. More specifically, how far behind it I am.

I've implemented a simple grid, but the examples of other Expert Advisors, and especially the current leader Better, show that it's a good direction to go in.

Prival, I don't want to confuse you with your plan, but I am convinced that you will find much useful in AI also for your professional activity and will build a profitable TS faster.

 

Gentlemen, let me give you some advice. You offer me other ways to build my TS (giving examples of AI, neural networks, but from my point of view you do not fully understand them). Or maybe you think that I have just started dealing with Forex (even though I remember the times when the minimum spread was 10-15 pips and it seemed to be a blessing), or I have no idea about the AI, neural networks and where the legs grow out of :-).

Now I will try to help you to understand what I want to do and what you do, if you create a "simple trained network".

For better understanding I will give you the link, read it again and compare it with what I write below (I can't find klot's statement, unfortunately, maybe it says there too).

How to find a profitable trading strategy ('How to find a profitable trading strategy')

Unusual Automated Trading ('Unusual Automated Trading')

Here's a quote "If we divide objects into two classes: longs and shorts, and take values of indicators or oscillators of technical analysis as features, then we only need to figure out the equation of the plane and try to use it for identification. The problem statement is clear.

This is the theory of recognition in its pure form, and the military uses it perfectly well and for a long time (examples of classes ("friendly - alien", (fighter - carrier, etc.). And here is a more complicated task than 4 classes (BMP - APC - BMP model - APC model), try to do with one straight line here :-), divide everything into 4 classes with it.

Neural networks, the notorious AI emerged just from this theory simply people are inherently lazy and want to create a system again quote "How realistic is it to trade successfully and comfortably using the MT4 trading platform, and not too burdening yourself, in doing so, with meticulous market analysis?". Training a computer to recognize a large number of classes and features is very difficult, that's why Neural Networks (NS) were created. NS have had some success but this is not a panacea, there is no INTELLECT there. The main thing in these tasks, and most importantly, is the partitioning into classes and the selection of features to recognise these classes.

After reading this material, I have an impression that either the authors don't understand what they are doing or they mislead us knowing it very well. Here is a sample fragment of the main code.

double perceptron() 
  {
   double w1 = x1 - 100.0;
   double w2 = x2 - 100.0;
   double w3 = x3 - 100.0;
   double w4 = x4 - 100.0;
   double a1 = iAC(Symbol(), 0, 0);
   double a2 = iAC(Symbol(), 0, 7);
   double a3 = iAC(Symbol(), 0, 14);
   double a4 = iAC(Symbol(), 0, 21);
   return (w1 * a1 + w2 * a2 + w3 * a3 + w4 * a4);
  }

If I created a combat algorithm that determines the "friend-or-foe" using as a sign (a technical indicator Acceleration/Deceleration (AC)), to the wall will be little (even if I take into account the acceleration of the object 7, 14 and 21 minutes ago). It does not change the essence. And note, I'm only talking about the task of splitting into classes and I'm not saying what to do with them after they're separated. The authors of the above material, mixing everything in a bunch (immediately into the fight to buy or sell).

Now I would like to go back to the material that I posted in this thread and show how the TC should look like (naturally from my point of view).

1. Here is a picture, think about it, maybe it will tell you how to improve the "meshes" you create. This is a Waldowsky detector in a picture (YES-NO logic, I talked about it a few pages ago).

2. Clearly define the signs of recognition (I want to use as them models of "behavior" which I lay down in Kalman filter.). What you put in I do not know, try to observe at least one condition features must not be correlated (strongly correlated).

Determine the class into which you are going to divide the input stream of quotes (say "flat trend"). And only after the recognition of the stream throw yourself into battle, ie choose a trading system that works well in the flat, the other one - in the trend.

P.S. I lost half a day again, saying the obvious to me things, but my goal has not been reached :-(. I've never found anyone who actually wants to help me in creating the TS. I'll probably delete the hell this thread.

 

I believe that constructive criticism is as valuable as a possible agreement between the parties at the design stage and discussion of a possible strategy. It can save time and effort. Therefore, I see no reason to be annoyed - we are in the same team and have a common goal.

You, Prival, have already been noticed several times that the aircraft has mass and, hence, it is always possible to choose a discreteness of division of a timeline such that the curve describing the movement of the aircraft in space, taking into account all possible somersaults, will be smooth. Overlaying an additive random interference on this curve does not change the picture - we can reconstruct the trajectory and are able to predict. I consider this as a fundamental point distinguishing the behavior of the LA from VR price type. There is no smooth curve in the price series laden with interference - there are evaluation criteria that say this with some certainty.

This is a criticism on my part, and you, so far, have not responded constructively to it.

1...789101112131415161718192021...85
New comment
Reason: