Over-fitted strategy
If you are into algotrading, or trading with the usage of algorithms, you probably heard of strategy over-fitting or over-optimization.
Over-fitted strategy means, that strategy was created (or its parameters were fitted) to perform perfectly on specific historical data, but on real trading, it won't work the same. It is a common issue when we develop a new strategy (or adjusting existing one) with machine learning and genetic algorithms. We then get a really nice-looking strategy, but when we put it on the real market, the strategy performs poorly and we are suddenly at the loss.
Fortunately, there is the solution to this problem, and that is doing Robustness tests.
Robustness tests are series of simulations, which we are running on the strategy, to ensure that even with changed market conditions, missing trades, or higher volatility, the strategy will still be profitable (at least on some scale). We can compare several attributes, like Profit Factor, Ret/DD Ratio or Net profit. I use condition such as "Return/Drawdown ratio for the test simulation has to be at least 50% of the Return/Drawdown ratio of the original backtest". With this approach I instantly know, if strategy can have similar profit and drawdown as the original backtest, so I know what to expect from the strategy in the future.
The goal is to simulate real market conditions, where we can have a higher spread, higher slippage or volatility and the strategy shouldn't fail. Other tests are for finding, whether our strategy has a real Edge and we are not following any extremes (for example best possible setup of parameters, which if we change, the strategy fails).
I am developing new strategies with software Strategy Quant, where I have a lot of various robustness tests at my disposal. As I don't wish to have an over-fitted strategy, which could look perfect on historical data but struggle in real trading, I always do series of robustness tests:
1) Testing on OOS data: backtest the strategy on unknown data and see, if results are similar with backtesting during strategy development. This test is run automatically during strategy creation.
2) Higher backtest precision: tested on high precision data from Dukascopy
3) Test on additional markets: strategy should be profitable also on other markets or timeframes which proves that strategy has real edge
4) What if simulation: backtest with excluded few of the biggest trades, strategy should have reasonable profit also without these big trades
Monte Carlo Simulations:
5) Randomize trades order with method Exact/Resample: testing against market cycles, where order of the trades is changed
6) Randomize strategy parameters: backtests with changed strategy parameters tells us whether the strategy is overfitted or not
7) Randomize history data: a test of volatility, so the strategy performs the same even in a highly volatile market during various market cycles
8) Randomize slippage and spread: very important test, which simulate real market or broker condition
9) Walk-Forward Matrix: test for finding if the strategy is overfitted. Series of backtest when i.e. 2 years of data are used for finding parameter and next 0.5 year of testing this parameter. We can use these results for regularly optimization or simple to find out, if strategy has real edge or not. As if the strategy has real edge, it should perform well even with parameters in broad range, not only with single set of parameters.
Another interesting tests available in Strategy Quant:
Monte Carlo - Randomly skip trades: test for realistic behavior of the trading, when sometimes trades are not executed (caused by malfunction VPS for example)
Optimization Profile and System Parameter Permutation: optimization tests ensuring that strategy has a real edge and doesn't follow any extremes (for example over-optimized parameters tunned to specific backtest data). Strategies with slightly different parameters should behave similarly, which means we are following real Edge.
How to read Monte Carlo analysis results?
It's quite easy. Monte Carlo always simulates a specific scenario, for example, it randomly skips trades with some probability and it does that 200-times (200 backtest results). Then it shows results, which 95% of backtests achieved. When you are looking at the table, you can see highlighted row with the number 95% and then some results. That means 95% of done backtests had at least these results. The picture below shows, that the original strategy has a profit of 35432$ and 95% of done backtests with the skip trades achieved a profit of 28256$. That's really great, as it signals a robust strategy, which is still profitable even if we don't take 10% of all trades. Similar simulations can be done in various numbers, so now you can understand the results.
