Interview with Leonid Velichkovsky: "The Biggest Myth about Neural Networks is Super-Profitability" (ATC 2010)

Automated-Trading | 19 August, 2010

Interview on Automated Trading Championship 2010 from 19.08.2010.

- Leonid, you are a rare representative of the traders' community, who use neural networks for trading. These are quite complicated developments, but the army of their fans keeps growing. What attracts you in neural networks?

- Six years ago, at the very beginning, neural networks attracted me with their novelty, unusual mysterious character and seemingly high profitability. Over the years, many myths have gone, but neural networks still attract me with their ability to adapt to any curve and to find patterns where nothing and nobody else can find them.

- Could you tell more about the myths associated with neural networks? Have you met any disillusionment in this field?

- The biggest myth associated with neural networks is their super-profitability. But this applies not only to neural networks, but to Forex as a whole. At first it seems that it is easy to earn - buy and sell, there's nothing complicated in it. Later, however, some factors appear, of which you did not even know - only then you start to comprehend and understand them. In neural networks, disappointing is the very thing that attracts you - their ability to train and adapt to any market with any data available. Their big advantage is a significant disadvantage when applied to financial markets. This is an amazing metamorphosis!

- How did you come to understand that neural networks do not bring super-profits? Was there any personal experience?

- There is no super-profitability in Forex also, not only in neural networks. Strictly speaking, neural networks are the same trading systems (hereinafter - TS). They only use a neuronet instead of common indicators. And then the most important facet is money management, i.e. the trader's "greed".

When you start trading, you don't have the concept of money management as such. But then you come to realize the necessity of this tool. Work on Forex, and generally in financial markets, is always connected with risks. You must be aware that the risk of $100 and $100,000 are two different things. When I traded on an initial deposit of 100, 500 and even 1,000 dollars, there was certain risk, and the whole approach to trading was specific. And when I began to trade larger amounts, the attitude toward trading became something quite different - the level of risk increased, and I quickly understood that I could lose everything. A certain responsibility came along with this.

For example, when trading on the deposit of $100, the profit of 100% per year can hardly be satisfying, I think. But trading on the deposit of $100,000 a profit of 100% per year is not bad at all. Thus, there is a kind of psychological conflict - traders who trade on small deposits, seek to earn as quickly and much as possible. This pushes traders to go beyond all conceivable risks. The result is the natural loss of deposit! Therefore, I believe, trading on small deposits is doomed to failure because of the natural desire of a trader to earn as quickly and much as possible. And $100, for example, is not a large enough amount to keep you away from risks.

- For the past six years, you've been working with neural networks in trading. How do you create these mysterious neural networks? What do you use?

- I'm not a programmer, I'm a trader. Programming neural networks and using them in financial markets are completely different things. Programmers help me to develop Expert Advisors - Roman Kramar (bstone), Yuri Zaitsev (YuraZ), Victor Nikolaev (Vinin) and Dmitriy Fedoseev (Integer). All of them are professionals in their field, I don't need to explain much - they know everything perfectly well. And I am very grateful to all of them for their work and professionalism.

Also, I have cooperated and continue to work with Steve Ward (Ward Systems Group) and Sergei Dolenko (Neuroproject) who gave me invaluable information on the application of neural networks in financial markets. In addition, I worked closely with Dennis Meyers (Meyers Analytics), Philippe Lonjoux (Noxa Analytics, Inc.) and Mark Simpson (Bowfort Technologies Inc.), with whom I tested new systems and indicators.

I'd like to note that application of neural networks in financial markets has many features and innovative concepts and techniques, and differs much from their use in other areas.

I use MetaTrader 4, of course; now I try to make friends with MetaTrader 5. Another indispensable tool for the work is NeuroShell, without which I can't do. I use MTFeed as a bridge between MetaTrader 4 and NeuroShell.

- There are many methods of training neural networks. Leonid, how do you train them? And finally, the question that torments many beginners in neural network trading: How to avoid the so-called "over-training"?

- It's a complicated question, to which I (and not only me) have no answer and which is impossible to clearly systematize. Nevertheless, I will try to touch upon the core problems of training and ways to avoid over-training. Because of its strong nonlinearity and the ability to adapt to any data, a neural network is very well adjusted, trained and as a consequence - over-trained. A neural network with only a few neurons in its inner layer easily remembers the history of a few thousand bars. It should be noted that "over-training" is inherent in neural networks only when applied to financial markets. What does this mean? We all know that the market changes over time - what happened in the past will be gone in the future. Well, it will exist, but somewhat differently, there will be no 100 per cent matches. Patterns, laws, market areas - all this will be different on different parts of the market.

Accordingly, if a neural network learns the lessons (examples) of the past too well when being trained on history data, in the end it may simply fail to notice or to identify new patterns and market areas in the future. Because all of them have undergone some changes. That is, the neural network has too well adapted to the market conditions, which existed in the past, but it was unable to recognize the new patterns in the changed market conditions.

Are there any ways to avoid over-training? There are many ways, but the main are two of them: the early stop of training and increase of the training interval. However, both methods have their serious disadvantages. In the early stop, there is a difficult questions, to which there is no answer: At what point shall I stop training? What criteria should be used for that? There are many answers to this question - use errors, profit level, drawdown and other mathematical criteria. But they do not give one hundred per cent guarantee of timely stops. Therefore, this timely stop of training depends only on the trader's skills.

There is a misconception that the better it was in the past, the better it will be in the future. Or the smaller the error on the interval of training is, the better the network will operate in the future. However, that's not true - the market is changing, and being too well trained in historical data, a neural network can fail to see the future. I know from my own experience that the ratio of errors on the part of training and profits on OOS (Out Of Sample - outside the interval of optimization) or on a real account is the following - the error decreases gradually with the increasing training time, but the profit first increases and then falls, forming a maximum in a certain moment in time. This is the very maximum we need to catch.

Further, as the training time increases, the error will also gradually decrease, and profit on OOS can produce several more maximums, but they are usually less than the very first one. Although I encountered a situation where the second and even the third maximums were higher than the first one. But it is believed that the first maximum is better than the rest in terms of profitability and efficiency. In fact, our task is to catch this first maximum. And it depends on skills and experience of the trader - I don't know any other, more accurate criteria. Although, of course, we may and should be guided by the percentage of profitability, error, drawdown, Sharpe ratio, and many other parameters. But ultimately, it depends solely on the trader what criteria to use. And it depends on how he understands his TS and knows how it behaves.

While on the interval of training, completely different things happen. The error and profit behave exactly opposite - the error gradually decreases, and profit increases smoothly. If profit increases during optimization, this means the Expert Advisor is simply fitted to the market curve, transforming the price into a smooth curve. This curve should rise up and is called equity. In fact, this optimization is also to reduce the error. And we get the following thing: the larger profit on the section of training or optimization is, the more likely you will have over-training or over-optimization (fitting) and, as a consequence - losses in the future.

The second way to avoid over-training is to increase the interval of training, i.e. to increase the amount of data, on which the network is trained. But this method also has its pitfalls. Increasing the amount of data in financial markets leads to the fact that the network may simply fail to see or recognize those patterns and market areas, which exist at the given training section. The section is too large for that. This is because the market changes with time. And a specific pattern appears too "different" in this large interval, and the network can't define that this is the same pattern, which only has changed over time.

Then a natural question arises: What part of the market should be given to a network for training? Here is the answer: the part where the network successfully recognizes patterns and market areas necessary for a TS and the trader. This depends on a trader's skills - the way he sees the market and how well he can choose the right part for training. In my experience this is from 500 to 2000 bars depending on the timeframe and market state.

There are some more ways to avoid over-training, but they are not so significant.

You see, much depends on how skillful and experienced a trader is. So I think that this profession requires not only mathematical knowledge but also some creativity. It is also clear that all the features and nuances of the use of neural networks, as well as usual TSs, come from the fact that the market changes over time, and the past never repeats exactly in the future. This feature exists only in financial markets. There is a popular myth that you need to give a lot of data to a neural network and let it train - it will learn independently what it needs. For the normal use of neural networks this may be true, but financial markets have their own peculiarities, which I described above, so it's not so easy in this case.

In my opinion, these two ways to avoid over-training also apply to the optimization of common Expert Advisors, without neural networks. Over-optimization, or "fitting", is specific to financial markets only. And the ways to avoid it are the same. The essence of over-optimization also lies in the fact that the nature of financial markets changes in time. Strictly speaking, "the market is not stationary".

- What are common errors that a trader can face when starting to work with neural networks?

The popular delusion of traders who start using neural networks and use non-normalized data at the input, trying to get the price of the next bar, is "Today will be like yesterday, and tomorrow will be like today" (if we consider daily bars). This is a common over-training of a network. While data on Forex do not differ much from each other (100 points make only 0.7% of the price), then the training error will also be small, and the network will quickly find this local minimum of training.

- Some "neural" traders use preprocessing of input data. Do you use anything like this in your neural networks?

- Generally speaking, I never use pure time series for the inputs of neural networks. Time series are always transformed by some indicator, which normalizes the data into a certain row. For example, from -100 to 100 or from -1 to 1. Further normalization is not required, because if the indicator values are greater than 1, they can always be divided by an appropriate number, to achieve a value not exceeding 1. I try to do as little change of input data as possible, since any transformation brings additional nonlinear distortion in the input signal. This accordingly leads the wrong training of a neural network, since the distortion can be incorrectly interpreted by the network. Moreover, with strong transformations, and consequently large nonlinear distortions, the network can be trained not on a real input signal, but on nonlinear distortions, which may lead to wrong operation and loss of deposit.

Here are a few examples of nonlinear distortions that are visible to the naked eye. Take, for example, the usual stochastics. It seems that such a simple indicator would bring no distortions. But in some moments, it makes strong nonlinear distortions, which may mislead the neural network in the training process and in further work on a real account.

These areas are marked with a white oval on the chart. In the first case the price goes up, and the stochastic indicator virtually stands still in its maximum values. In the second case, the price is almost at one and the same level, and the stochastic indicator sharply goes down from its maximum to minimum values. In the first case the stochastic indicator will bring no information to the network, while in the latter case it will simply "confuse" it." In both cases, the behavior of the stochastic indicator will have a negative impact both on the training and on the work of the neural network on a real account. And that can lead to financial losses.

It should be noted that these two examples are quite noticeable distortions that you can easily see. And there are much more distortions that we can't see and analyze! Believe me. And all these distortions (both large and small) are combined together. Therefore, be extremely careful when doing with the pre-processing of input data.

Of course, there are indicators that make much stronger distortions. There are also those that make less strong ones. However, the fact remains - distortion are made by any indicators. Although, you can select specific parameters to any indicator (even stochastic), so that it will bring minimum distortions in the original signal with specific market conditions. Naturally, the nature of the market may change, and you will have to change the indicator parameters in order to reduce the distortion introduced. And in this situation, a proper selection of indicator parameters and their timely adjustment (both automatically and manually) is also totally dependent on a trader's skills and experience.

- How do you assess the results of a neural network after training or TS after optimization? What are the criteria to use them on a real account?

Currently, I almost never take into account the results of a TS, which have been obtained in the training (optimization) interval. I analyze results on OOS or real, because I believe that in the period of training (optimization), the results of a TS can tell nothing. This can be fitting or over-training; and it's almost impossible to define whether it's fitting or not. You can only define it by testing it on OOS or better on a real account. Sometimes, I simply compare the results on the real account (OOS) and that of training (optimization). Therefore, the figures show equity on a real account with the trading leverage of 1:1 (deposit use is 1% with the leverage of 1:100 given by the dealing center). If we increase the leverage, the range of equity will also increase.

Actually, I analyze the results of a TS only with the leverage of 1:1, i.e., with a "disabled" money management. Because money management can give the wrong idea about the actual drawdown of the TS and, accordingly, an "unexpected" margin call and other troubles. In the figures, you can see the equity with the leverage of 1:1.

By the way, it is the same trading system that participated in the ATC 2008, though with slightly modified parameters.

Recently, I noticed the following: if the profit factor is extremely large in the training (optimization) interval with the leverage of 1:1, we can say for sure that it is over-training (over-optimization). And in the future, at the unknown data, the trading system with such parameters will work poorly (i.e. will lose the deposit). One can note that in the figures, the equity rises smoothly rather than sharply. You could conclude that the profitability of such a trading system is not quite small. Although if you increase the trading leverage or use a more aggressive money management, the profits can increase manifold. Everything depends on the drawdown, which appears with the leverage of 1:1, and the drawdown allowed by a trader.

- Almost two years have passed since the ATC 2008. What lessons have you learned from the results of that Championship? Why couldn't your Expert Advisor win the contest?

The Championship is a competition. Nothing ventured, nothing gained. I dared; and I exceeded all possible risks because of my money management. I managed to earn 110,000 and then fell to 14,749 because of that too aggressive money management. For 3 months, the profit was almost 50%, which was pretty good. But the drawdown was 92%, which is unacceptable in real life. Then, having run my EA with reasonable money management over the same period, I got almost the same result of $14,000, but with a drawdown of about 25% - this is a good result for real life. The conclusion is that you shouldn't chase the excess profits, otherwise you can lose. But the Championship makes its own rules and, of course, you need to take risks to win.

- Has anything changed fundamentally in your developments over this period? Perhaps, have you found any know-how and applied it in practice?

No, in fact everything remains the same. Nothing new has happened. Moreover, the same Expert Advisor with the same parameters can still work, though I found other, more "profitable" parameters. The essence of the market doesn't change - only its character is changing, which an experienced trader must keep track of, timely adjusting his TS to the new, changed market conditions.

- The Expert Advisor of Alexander Topchylo, the winner of the ATC 2007, consisted of three independent subsystems. However the author was going to progress in this direction and create a committee of neural networks. Do you use such committees in your developments?

- No, I refused to use committees due to them being hard to implement and maintain. Over the years, I have come to use simple TSs, because a too complex TS, as well as that with committees, cannot guarantee a more stable and larger profit in comparison with a simple one.

- The author of the only one multicurrency Expert Advisor among the winners of the ATC, Nikolay Kositsin believes that rules of the upcoming Championship are favorable to multicurrency EAs and leave little chance to single currency robots. Do you use multicurrency in your Expert Advisors? What pairs do your EAs trade?

- Of course I use them. This allows hedging trades and getting a smoother equity. Besides, if you use multicurrency for analysis, this helps to create more stable and reliable trading systems. On the Championship, I intend to trade EUR/USD, USD/JPY and AUD/USD - depends on how the market situation changes closer to the Championship.

- Leonid, thank you for the interview. Good luck in the Championship!