How to make money from unsteady markets? (Article)

 

As we know, markets are non-stationary. It is very easy to prove this: we take a market model for some financial instrument and fit it to a certain section of historical data, approximate it as closely as possible (approximate it). We obtain an extremum of a difference between the historical data and our model - the residue. Let's run the model obtained as described above on another history data section - outside the sample, of the same market instrument. We obtain much worse result as far as residuals are concerned.


Markets are constantly changing - they are non-stationary.


So anyone who tries to build a market model by the principle of maximum approximation of the model to a certain specific part of the market price data time series gets into trouble. Because all such approximation models are calculated on the fact that the markets will not change, their statistical and probabilistic characteristics will remain the same. But we know that this is not true. It follows that all attempts to build the most accurate models (formulas) of the market based on some historical data are doomed to failure, because any change of the market based on which the model was built would make it at least inaccurate.


What to do in this case? Maybe we should refuse any optimization - adjustment to historical data?


The answer is obvious: in order to build an optimal market model the optimization should be done in such a way that the market model is dynamic rather than static. I.e. to give the model some degrees of freedom.


It would seem that the last statement looks trivial. Yes. But look at what is done when building market models, and you'll notice that such a trivial truth is almost never used.


It's equally obvious that optimization should be carried out using algorithms of finding the optimal solution for the game of two persons (i.e. trading system of trader and market) with zero sum and mixed strategies (freedoms of strategy selection for both players) using payment matrices. For in this case we have:


1. A knowingly dynamic model of the second player, the market, assuming degrees of freedom for him to choose his strategy - a mixed strategy.
2. Already ready mathematical and algorithmic apparatus to proceed immediately. Mathematical apparatus for the game of two persons with mixed strategies and zero sum is complete, i.e. it does not contain any "white spots", and therefore we either obtain solution, or the notorious answer that no solution exists.

But, most importantly, the solution of the payment matrix for the zero-sum game of two persons gives us not one but two optimal solutions minimizing the costs of each player: one for the potential market strategy, the other for the trader's trading system.

And since the optimal strategy in solving the payment matrix is the minimization of costs, i.e. the worst option of mathematical expectation of profit that a player can get by strictly following his strategy, then due to non-stationarity of the market, the probability that the market will not follow its own optimal strategy and hence necessarily reduce the cost part of trader's trading system while increasing the profit part if this very trading system will strictly follow the optimal strategy with

I will not go further to describe some mathematical details of the two-person zero sum game for mixed strategies and its special features, as all information is open and available on the Internet, such as this link: Mixed strategies for two-person zero sum games

Payment Matrix

The payment matrix for a two-person zero-sum game is a two-dimensional number space. The point is that the expectation for a row or a column of this very matrix is calculated taking into account the optimal strategies of both players. The total (final) expectation of the whole game, i.e. for all rows or for all columns taking into account the optimal strategies of both players is called the game price.

Therefore, the payment matrix is most often filled with payout values. In essence, the payment matrix is then the rules of the game. If the number is positive, the first player pays the second player an amount equal to the value specified in the cell. If it is negative, then the second player pays the first player in the absolute amount of the specified value.

Thus, payment matrices allow modelling a set of games, the results of which depend on the players' choice of one or another outcome of the game, provided that none of them knows in advance the choice of the second player. With respect to trading we get an analogy, as the trader does not know in advance in which direction the price will go, and the market does not know in which direction a certain trader will open a position, unless the trader is an insider (holder of a significant part of assets in the market with which he or she can greatly influence quotes).

Practical example

Let us assume that we are trading stocks on the Chicago Stock Exchange. Our strategy is to buy and hold the security for a certain period of time. The problem statement is formulated as follows: At what time of the session and with what volume in lots is it most profitable for us to take a long position?

The trading session lasts for 7 hours and 30 minutes. Thus, it turns out that the entire session can be divided into 15 equal parts of 30 minutes each. Therefore, the analysis will be carried out on the timeframe M30.

We will take the quotes for the last 15 days - three complete trading weeks. Let's assume that the result of each period is the difference between the open price of a bar and the open price of the previous bar, i.e. for the bar number n in the history data it will be Open[n] - Open[n + 1]. The time is based on the bar number n + 1

Let's fill with these very differences in the opening prices an array of numbers where the horizontal bar is the session time - individual bars from the beginning of the session, and the vertical bar is the session itself, i.e. calendar dates. Thus, we have obtained a payment matrix. If we solve it for the player by columns, we will get a solution to our problem, i.e. at what time and with what volume we should enter into a long position. As mentioned above, this will be the worst-case estimate of the situation by mathematical expectation - the price of the game.

Since we know what the decision means for the trading system of the trader, then what exactly the decision will mean in relation to the market, i.e. the player also receives some values along the lines? These very values are the most profitable for the market and disadvantageous for the trader adhering to a bullish strategy, ranges for individual trading sessions. I.e., when choosing a payment matrix solution in relation to the market, the largest ranges were chosen for the days with a bearish trend, while for the bars with a bullish trend. As a result, there is a compromise for the market and the trader, according to which, if the trader sticks to the optimal strategy, no changes in the session range can worsen the expected payoff.

Note: The bottom line is that when choosing a strategy for the market, if a trader's strategy is bullish, then the ranges for bearish days will be increased, and for bullish days, or removed from consideration altogether (zero range). That is, eventually, if the historical data under study clearly shows a bullish price movement, then when recalculating with consideration of decreasing-increasing ranges of separate sessions for the market strategy, the forecast will be made for a bearish trend. That is, the trader's strategy, in this case, will be reduced to searching for the most frequently occurring bullish candles during the bearish sessions. But do not worry, if such candlesticks are sufficiently probable, we will definitely find a nice solution.

To be most confident we have to calculate the mathematical expectation, which is the price of the game taking into account both strategies. If it is positive, it is what you need, so the worst estimate is a guaranteed profit.

But what to do if the mathematical expectation taking into account the trader's and market strategies turns out to be negative? Some people having read bad books may argue that trader's strategy should remain the same in terms of volumes, but instead of buying and holding the security, they should sell - reversal of the strategy. But this should not be done. Why? Because after we calculate the optimal solution for the bull strategy and change it to a bearish strategy, the resulting minimum valuation, will be the maximum. It may be positive, but it will be the maximum, as if we obtained it in our terminal optimizer with the terminal fitting. And since the market is non-stationary, it is unlikely that such a reversal strategy will hold at the calculated maximum. Most likely, it will again turn negative. After all, according to the strategy before the flip, the largest buying volumes were on candles with a prevailing bullish trend. And now, if we start trading in bearish moods on bullish candlesticks, the increase in expenses will increase for the trader, and consequently, the probability of getting into negative mathematical expectation.

Yes, in case of negative expected payoff and considering the strategies obtained, we will indeed have to switch from bullish strategy to bearish one. But in doing so, we will have to recalculate the entire payment matrix, not for the player by columns, but for the player by rows. Or change the signs in all cells of the matrix itself and then you can recalculate the columns for the player, i.e. do not change the algorithm. Now the strategy will change and expectation will also change not only by sign but by value as well.

How to get solutions for already prepared payment matrix?


There are several ways, among which the most common are:


1. Through linear programming, namely the Simplex method. The method is not very good as it can jam (depending on the implementation) if the payment matrix has no solutions, and in some cases if there is a saddle point in the payment matrix or no solution at all (depending on the implementation).
2. iterative method. At each step of the iterative method, convergence to one of the potential solutions of the payment matrix is achieved. The point is that if the next step is ambiguous, the choice is made with the help of a random number generator. Therefore, the method may produce different solutions when recomputing the same payment matrix. If there are no solutions, or the price of the game is zero, then the convergence expectation will tend to zero.

I personally use iterative method, the algorithm is shown below. The listing is in Java, which makes it easy to convert to C using special utilities or recompile to machine codes using GCJ. Theoretically, everything could be written in MQL4 or MQL5 at once, but the low speed of MQL4 and bugs in MQL5 do not allow using this algorithm in the above programming languages.


A payment matrix in the form of an array is given as input to the function - matrix


The output is an array of numbers as the player's strategy by columns. If it becomes necessary to recalculate the strategy for a player by rows, the sign for all values in all cells must be changed in the payment matrix.


The function outputs to the console the current value of the algorithm passed (progress) as a percentage and the mathematical expectation, taking into account the optimal strategies for the player by rows and columns.

 

This is the source of a function to find the optimal mixed strategies for a two-person zero-sum game with cost minimization for the player on the rows (positive values in the payment matrix are wins for the player on the rows and losses for the player on the columns). The source code works. Checked, no mines.


  public double[] getData(double[][] a) {
    int m = a. length;
    int n = a[0]. length;
    double[] p = new double[ m];
    double[] q = new double[ n];
    double[] x = new double[ m];
    double[] y = new double[ n];
    int r = rand. nextInt( m);
    int c = 0;
    for (int t = 0; t < 100; t++) {
      System. out. print("Progress: " + t + "%  \r");
      for (int u = 0; u < 10000; u++) {
        for (int j = 0; j < n; j++) {
          y[ j] = y[ j] + a[ r][ j];
        }
        c = 0;
        for (int j = 1; j < n; j++) {
          if (( y[ j] == y[ c]) && rand. nextBoolean()) {
            c = j;
          }
          if ( y[ j] > y[ c]) {
            c = j;
          }
        }
        q[ c] = q[ c] + 1 d;
        for (int i = 0; i < m; i++) {
          x[ i] = x[ i] + a[ i][ c];
        }
        r = 0;
        for (int i = 1; i < m; i++) {
          if (( x[ i] == x[ r]) && rand. nextBoolean()) {
            r = i;
          }
          if ( x[ i] < x[ r]) {
            r = i;
          }
        }
        p[ r] = p[ r] + 1 d;
      }
    }
    System. out. println("Progress: 100%");
    for (int i = 0; i < n; i++) {
      q[ i] = q[ i] / 1000000 d;
    }

    double ep = 0 d;
    for (int i = 0; i < m; i++) {
      double result = 0;
      for (int j = 0; j < n; j++) {
        result = result + a[ i][ j] * q[ j] * p[ i] / 1000000 d;
      }
      ep = ep + result;
    }
    System. out. println("Expected Payoff = " + ep);

    return q;
  }
 
Reshetov >> :

Anyone who tries to build a market model by the principle of maximum approximation of the model to a certain part of the time series of the same market gets into trouble.

It is hard to disagree.

Because of non-stationarity of the market the probability that the market will not adhere to its own optimal strategy and therefore will decrease the expense part of the trader's trading system and increase the profit part, if this very trading system will strictly adhere to the optimal strategy.
I guess so. If one considers that the market has this optimal strategy. I personally feel closer to the analogy of playing with nature, where nature does not build meaningful strategies.
Let's take the quotes for the last 15 days - three complete trading weeks. Let's assume that the result of each period is the difference between the open price of a certain bar and the open price of the previous bar, i.e. for the bar with the number n in the history data it will be Open[n] - Open[n + 1]. The time is based on the bar number n + 1

Let's fill with these very differences in the opening prices an array of numbers where the horizontal bar is the session time - individual bars from the beginning of the session, and the vertical bar is the session itself, i.e. calendar dates. Thus, we have obtained a payment matrix. If we solve it for the player by columns, we will get a solution to our problem, i.e. at what time and with what volume we should enter into a long position. As mentioned above, this will be the worst-case estimate of the situation by mathematical expectation - the price of the game.

Great! Let's get into more calculations and tweaks next.

Only who says that the "15-day quotes" summarized in the matrix is the optimal market strategy, and not another, but perverted, section of the time series?

All the more so because nature, sorry market, does not have to adhere to this matrix...


Reminds me of the poem about the butt and his beloved dog...

The tsutzik wants meat badly. ;)


Bottom line - the gardener didn't understand the new "optimization on the fly" chip.

Explain the differences from conventional fitting methods, please.

 
Sorento >> :


Apparently. Assuming the market has this optimal strategy. I personally prefer the analogy of playing with nature, where nature does not build meaningful strategies.

Tried it. It does not fit. Since the "nature" is inevitably assumed a complete absence of any strategy. It only remains for this very nature to pick any of the many criteria, which may turn out to be not even close to the market.


So it is better to stick to the opinion that the market is not a dumb "nature" that does what it wants, but it is more efficient than you might think. So it's better to make a calculation considering that it will try to cheat the trader and it will do it the most effectively.


As one of my friends (also a trader) says: in our business, it is better to be safe than sorry.


Sorento >> :

But who says that the "15-day quotes" summarized in a matrix is an optimal market strategy, and not the next, but perverted, segment of the time series?

All the more, nature, I apologize, is not obliged to adhere to this matrix...

...

Explain the differences from conventional fitting methods, fire.

There are methods of additional fit testing for this. For example, on forward tests.


My task is to show only one of the methods of application, and how you use this material and model it is your personal problem. My job is to offer, your job is to refuse. So if you have a personal opinion on how to avoid fitting, no one forbids using it. And if not, there should be no judgment.


If you don't like the 15-day story, there's no reason why you can't take more or less.

 
Reshetov >> :

That's why it's better to stick to the opinion that the market is not dumb "nature" that acts as it thinks best, but it is more efficient than you might think. And therefore, it's better to make a calculation based on the fact that it will try to outsmart the trader and do it most effectively.


Then maybe we should solve the problem of identifying the supposedly optimal strategy of a set of traders from the current data. then find that proverbial antagonistic market strategy towards the traders.

And then our "optimal with respect to her" will come. ;)


As a gardener, I don't know much about games, especially if they are of the "I know that he knows that I know..." type.

 
Sorento >> :

Then maybe one should solve the problem of identifying the supposedly optimal strategy of a set of traders based on current data. Then find this notorious antagonistic market strategy in relation to traders.

and then our "optimal in relation to it" will come along. ;)


As a gardener, I don't know much about games, especially if they are of the "I know that he knows that I know..." type.

You have been told that you can use whatever models you want for your own purposes, i.e. all traders without exception + emergency intervention models + actions of central bank governors + actions of governments + .... + earthquakes + possible effects of alien invasion, etc. to the pulse of the world. I.e. what prevents you from modelling more than just all traders, if you really have unlimited computing resources to account for anything and everything?


But this thread is supposed to discuss the model proposed by the topicstater, and not different ideas from all sorts of nonsense generators.


Sorento >> :

As a gardener, don't know much about games, especially if they're like "I know he knows I know..."

You always have some good adviser, that knows everything about it like a pig in oranges.

 
Reshetov >> :

you can use whatever models you like for your own personal purposes, ... + possible effects of alien invasion etc. to the point of losing your pulse.


But this thread is supposed to discuss the model proposed by the topikstater, not various ideas from all sorts of ramblers.

Thank you for being constructive in the science and numbers game.

Questions removed.

I'm honoured.


Just a reminder:


Two players T-rader:) and B-time:) play a game based on flip of a coin. The players simultaneously and independently of each other choose crest (G - sell - price down) or tails (P - buy - price up) .

If the results of two coin flips are the same (i.e. GH or RR), player T gets one dollar from player B.

Otherwise player T pays one dollar to player B.

The following matrix of payments to player T shows the values of the minimum elements of the rows and the maximum elements of the columns corresponding to the strategies
of both players.


VG

BP


Line minima

TG1-1-1
TR-11-1
Column maxima11


The maximum and minimum values (prices) for this game are $1 and $1, respectively. Since these values are not equal to each other, the game

has no solution in pure strategies.

In particular, if player T uses strategy TG, player B will choose strategy BP to get one dollar from player T.

If this happens, player T can switch to strategy TP to change the outcome of the game and get one dollar from player B.

The constant temptation for each player to switch to a different strategy indicates that a pure strategy solution is not acceptable.

Instead, both players must use an appropriate random combination of their strategies.

Moulin E. Game theory with examples from mathematical economics. M.: World,


it should be noted that mixed-strategy decision presupposes that there are probabilities for the market to apply one or another strategy.

 
Sorento >> :

Thank you for being constructive in science.

Questions removed.

I'm honoured.

You're welcome. We're always happy to blow the whistle on some flounder.

 
Reshetov >> :

You're welcome. We're always happy to blow the whistle on some flounder.

Thanks for the scores.

Just as long as it's fun. :)

 
Reshetov писал(а) >>

There are several ways of doing this, with the most common being:


1. Through the linear programming, namely by the Simplex method. The method is not very good as it can jam (depending on the implementation) if the payment matrix has no solutions, and in some cases if there is a saddle point in the payment matrix or no solution at all (depending on the implementation).
2. iterative method. At each step of the iterative method, convergence to one of the potential solutions of the payment matrix is achieved. The point is that if the next step is ambiguous, the choice is made with the help of a random number generator. Therefore, the method may produce different solutions when recomputing the same payment matrix. If there are no solutions, or the price of the game is zero, then the convergence expectation will tend to zero.

Can't this be simplified by looking at only part of the trades that meet certain conditions, or by discarding part of the trades?

Suppose we have an intraday system with many trades. We leave the long deals that took place at a bearish daily candlestick and the short ones correspondingly. It may be not only bearish, but 50 most bearish deals of the trade history and so on. It is possible to think of a lot of filtering criteria. In fact, it is just a selection of the worst situations for a trader - a conditional asymmetric response (:)) of the market.

It can be the other way around: you can look for the best strategies not on the entire history, but only on the most unfavorable, by some criteria, parts of the history for longs and shorts.

 
Avals >> :

Can't this be simplified by considering only part of the trades that meet certain conditions, or by discarding part of the trades?

Suppose we have an intraday system with many deals. We leave the long deals that took place on a bearish daily candlestick and the short deals accordingly. It may be not only bearish, but 50 most bearish deals of the trade history and so on. It is possible to think of a lot of filtering criteria. In fact, it is just a selection of the worst situations for a trader - a conditional asymmetric response (:)) of the market.

You can do the opposite: look for the best strategies not on the entire history, but only on the most unfavorable, by certain criteria, periods of history for longs and shorts.

Essentially, the game solution is the most unfavorable, by given criteria, parts of history, since when selecting a strategy for the market, as I said before, the sessions where a trader could make a profit will be excluded or even eliminated from the choice.


As for combining strategies for shorts and longs, in order to search for the most appropriate times for long and short positions, it is feasible. For example, if we run a search for strategies for long positions only, then this very strategy would recommend a trader to open with zero volume at certain hours. That is, it is better not to buy at such times. It will be possible to change the sign for all numbers in the columns of the pay matrix corresponding to such hours, which would essentially mean considering short positions. If the solution turns out to be non-zero values, it will be possible to short. The most important thing, as I said, is to achieve a positive price play for the trader.


Thanks for the suggested idea of mixed strategies for shorts and longs!

Reason: