Market prediction based on macroeconomic indicators - page 5

[Mykola Demko]

gpwr:

It's hard to explain in a nutshell. Read here first

https://ru.wikipedia.org/wiki/%D0%92%D0%B7%D0%B0%D0%B8%D0%BC%D0%BD%D0%B0%D1%8F_%D0%B8%D0%BD%D1%84%D0%BE%D1%80%D0%BC%D0%B0%D1%86%D0%B8%D1%8F

and then here (the chapter on mutual information, where the formula is given):

http://www.jclinbioinformatics.com/content/2/1/16

How is mutual information and entropy calculated?

ЗЫ А сори нашёл https://ru.wikipedia.org/wiki/%D0%98%D0%BD%D1%84%D0%BE%D1%80%D0%BC%D0%B0%D1%86%D0%B8%D0%BE%D0%BD%D0%BD%D0%B0%D1%8F_%D1%8D%D0%BD%D1%82%D1%80%D0%BE%D0%BF%D0%B8%D1%8F

[forexman77]

Here are some examples. Dow Jones index with Non-farm payrolls data.

The arrows mark the pivot point, the data includes a 20+ year history.

The pattern happened once, and it is possible to find another one in the interval of 2000-2001. It is possible to program it, but two signals in twenty years is too little for statistics.

[transcendreamer]

gpwr:
You can say that about any model, not only regression, but also neural models, ARMA and others. If there is no relationship between inputs and outputs, any model will generate a prediction, only inaccurately.

I agree, neural networks are even better, I wasn't clear at the time

regression is useful to quickly test an idea of whether something can be done with the data

i.e. to quickly build-check an empirical model

but the relationships may be "nonexistent" or indirect unobservable

I once built such a model:

MICEX index + 5-year Treasuries + 3-month LIBOR + price of BRENT + EU ZVR + something else

it turns out this combination is a relatively good predictor of the canadian

why? what is the connection? no one knows...

[Vladimir]

My calculation of mutual information:

function I = KMI(x,y,h)
% Calculate Mutual Information between x and y based on Gaussian kernels
n=numel(y);
if nargin<3
    h=(4/3/n)^0.2;
end
q=1/(2*h^2);
I=0;
for i=1:n
    Mx=1;
    My=1;
    Jxy=1;
    for j=1:n
        if j>i
            Kx(i,j)=exp(-q*(x(i)-x(j))^2);
            Ky(i,j)=exp(-q*(y(i)-y(j))^2);
            Kx(j,i)=Kx(i,j);
            Ky(j,i)=Ky(i,j);
        end
        if i~=j
            Mx=Mx+Kx(i,j);
            My=My+Ky(i,j);
            Jxy=Jxy+Kx(i,j)*Ky(i,j);
        end
    end
    I=I+log(n*Jxy)-log(Mx)-log(My);
end
I=I/n;

[Vladimir]

forexman77:

Here are some examples. Dow Jones index with Non-farm payrolls data.

The arrows marked the pivot point, the data includes a 20+ year history.

The pattern happened once, and it is possible to find another one in the interval of 2000-2001. It is possible to program this, but two signals for twenty years is too little for statistics.

To use macroeconomic indicators for the high-frequency trading, we should trade by dates of their release. That is, having a model of these indicators, we predict their next value, compare it with the published value and open a position just before the news release. To be honest, though, I am not interested in such trading. I am more interested in predicting crashes. Everybody can make profit on a rising market but avoiding losses on crashes is an art that requires an ability to distinguish the beginning of a crash from a correction.

Here is a more interesting picture. Before crashes, the number of PERMIT1 homes allowed to be built was falling (the vertical gray lines represent historical recessions):

The only recession before which the number of houses allowed to be built did not fall sharply was the 2002-2003 recession. Some economists argue that technically that period of time was not a recession because there were not two consecutive quarters of negative GDP growth. But the market price still fell quite sharply (dot com bubble). My model is pretty bad at predicting 2002-2003. What is needed is an additional indicator that is able to predict this period.

Here is another interesting example: bond yield curve = GS5-GS3M, predicts recessions well.

[Дмитрий]

transcendreamer:

I once built a model like this:

MICEX index + 5-year Treasuries + 3-month LIBOR + price of Brent + EU ZVR + something else

it turns out this combination is a relatively good predictor of the canadian

why? what is the connection? no one knows...

))) And how did you "predict" the canadoyen with that?

Moreover, I even KNOW "what's the connection", but the question is HOW did you predict kanadoyen with it?

Let's say you have a deviation of the forecast from the real rate of the canadooena and this model does "work" - what next? How do you predict the exchange rate of the canadooen will go to the rate of this synthetic, or the rate of the synthetic will go to the rate of the canadooen or they will both go against each other? How?

[forexman77]

gpwr:

To use macroeconomic indicators for high-frequency trading, you need to trade by their release date. That is, having a model of these indicators, we predict their next value, compare it with the published estimate and open a position just before the news release. To be honest, though, I am not interested in such trading. I am more interested in predicting crashes. Everybody can make profit on a rising market but avoiding losses on crashes is an art that requires an ability to distinguish the beginning of a crash from a correction.

Here is a more interesting picture. Before crashes, the number of PERMIT1 homes allowed to be built was falling (the vertical gray lines represent historical recessions):

The only recession before which the number of houses allowed to be built did not fall sharply was the 2002-2003 recession. Some economists argue that technically that period of time was not a recession because there were not two consecutive quarters of negative GDP growth. But the market price still fell quite sharply (dot com bubble). My model is pretty bad at predicting 2002-2003. What is needed is an additional indicator that can predict this period.

Here is another interesting example: yield curve, predicts recessions well.

Regarding crashes. Below is a chart of the Dow Jones with New homesales and theADP-EMPL-SEC data.

The ADP predicted a good fall in 2007, or rather it fell in sync with the Dow Jones.

It is interesting to note that the new home sales broke the trend at the end of 2005, but the index still rose after that, but then there was already a signal that all was not well in the market.

Regarding strategies. One could try:

1. One based on expectations and actual data. (The problem is that so far I found only actual data and no historical data based on expectations, forecasts.)
2. Only actual intraday data.
3. Only factual data long term.

P.S. I have little programming experience. I read data from file in int init() build buffer once and then fill indicator buffer with matching dates. In my Expert Advisor I get indicator data once per day. With this design optimisation speed is not bad.

The problem is if we use intraday data to build candlesticks using this data, if available. In this case reading from the file will be very long.

What equations do you use in the function you have given?

[Vladimir]

For those who read this thread, check out my first post, I updated it a couple of days ago.

For anyone interested in trying to predict the market manually using economic indicators, here is a list of indicators: https://www.conference-board.org/data/bci/index.cfm?id=2160

It is as follows:

1.	BCI-01	Average weekly hours, manufacturing	0.2781
2.	BCI-05	Average weekly initial claims for unemployment insurance	0.0334
3.	BCI-08	Manufacturers' new orders, consumer goods and materials	0.0811
4.	BCI-130	ISM new order index	0.1651
5.	BCI-33	Manufacturers' new orders, non-defense capital goods excl. aircraft	0.0356
6.	BCI-29	Building permits, new private housing units	0.0272
7.	BCI-19	Stock prices, 500 common stocks	0.0381
8.	BCI-107	Leading Credit Index	0.0794
9.	BCI-129	Interest rate spread, 10-year Treasury bonds less federal funds	0.1069
10.	BCI-125	Avg. consumer exposures for business and economic conditions	0.1551

It is interesting that the fed resurv considers the S&P 500 to be the leading indicator as if it predicts the economy rather than the other way round. The only leading indicator on this list in my opinion is Building Permits, but it has the least weight among all other indicators. Apparently the Fed doesn't know what they are doing and therefore can't predict the coming recession and prevent it with their monetary policy.

[Mykola Demko]

gpwr:

My calculation of mutual information:

On mql it goes something like this.

struct SDoubleArrayKMI{ double m[]; };
//+------------------------------------------------------------------+
double KMI(const double &x[],const double &y[])
  {
   return(KMI(x,y,pow(4./3./ArraySize(y),0.2)));
  }
//+------------------------------------------------------------------+
double KMI(const double &x[],const double &y[],double h)
  {
// Calculate Mutual Information between x and y based on Gaussian kernels
   int n=ArraySize(y);   

   SDoubleArrayKMI Kx[],Ky[]; ArrayResize(Kx,n); ArrayResize(Ky,n);
   for(int i=0;i<n;i++) { ArrayResize(Kx[i].m,n); ArrayResize(Ky[i].m,n); }
//---

   double q=1/(2*h*h);
   double I=0;
   for(int i=0;i<n;i++)
     {
      double Mx=1;
      double My=1;
      double Jxy=1;
      for(int j=0;j<n;j++)
        {
         if(j>i)
           {
            Kx[i].m[j]=exp(-q*pow(x[i]-x[j],2));
            Ky[i].m[j]=exp(-q*pow(y[i]-y[j],2));
            Kx[j].m[i]=Kx[i].m[j];
            Ky[j].m[i]=Ky[i].m[j];
           }
         if(i!=j)
           {
            Mx+=Kx[i].m[j];
            My+=Ky[i].m[j];
            Jxy+=Kx[i].m[j]*Ky[i].m[j];
           }
        }
      I+=log(n*Jxy)-log(Mx)-log(My);
     }
   return(I/n);
  }

[Vasiliy Sokolov]

gpwr:

My calculation of mutual information:

Thank you for the implementation. I will study it.

Urain:

In mql it goes something like this.

Double thanks to you Nikolay! MQL rules!