Completely different results from same expert
I now have backtesting reports with the exact same expert, the exact same dates, the exact same settings file and utterly different results. I have done my best to ensure the data is the same (reconstructing it from downloaded history data yesterday, I still got very different results today to then).
Elroch:
An expert I developed achieved excellent results in backtests and across a wide range of variations in optimisation and walkforward optimisation runs.
Then suddenly, without me having consciously made big changes the results became mediocre/marginal.
Then today again the results suddenly became superb again.
The tester uses the CURRENT spread. You're EA is highly sensitive to the changes.
As I mentioned I spotted that but, since I got mediocre results on the run today with a spread of 1.2 pips, this is not the explanation.
If your entry's and exit's are really only based on indicator values, you are trading moderate lotsizes and earning a good average in pips/trade there are not many options left:
a) currupt history file
b) something like randomness included
c) dependend on some global variable (which got changed betwen the two tests)
d) some broker settings might have changed, (unlikely)
Why not running the test in every tick mode? This will show you the quality of the data-
Are the psudo ticks strategy tester creates created identically for each test run and do indicators used by the EA, create values for each tick of the latest bar as they would when live trading ?
Thanks for your interesting points, zzueg.
(a) it seems infeasible that a corrupt history file could explain the sequence of backtesting results with the same EA: i.e. mediocre results, superb results, mediocre results and similar superb results over a period since Friday last week, during which time I have done fresh installations of metatrader, reconstructed all data from downloaded supposedly high quality 1 minute data, etc.
(b) the problem with any explanation due to random effects (including bad data) is that it is very difficult to see why it should make the trades consistently enormously better.
(c) For a couple of days, I thought it might have been some accidental change in my code. But today when I got the unbelievably good results, I immediately saved a copy of the expert (source and ex4) and the settings file. But the exact same expert and settings gave mediocre results in a later run.
(d) yeah, not sure there is much from the broker that is relevant beyond the spread. These are all backtests rather than live trading, of course, and the data was mostly constructed from downloaded 10 years of 1 minute data.
(e) With the expert entering and exiting positions at the bar open and not using stops or profit targets at all (in the version under study anyhow), my understanding is that the only values that can influence the results are the open, high, low and close of bars from the 5 minute and higher charts (actually it happens that even the close is irrelevant: there are several variants of the same indicator being used, and these happen to use (H+L)/2 as the data point, while entry and exit occur at the open of a bar, so the open data and the spread matter as well. Whether a trade gets opened or closed depends on the values of the indicators on earlier bars, not the current bar.
...but for the sake of thoroughness, I did a run with "all ticks". The results were virtually identical to the previous one, with a tiny difference explained by the last trade being closed at the last 1 minute bar rather than the last 5 minute bar.
I did not mean that the results change in every tick mode, i am intersted in the amount of mismatched chart errors reported.
Gaps in history are very unlikely since they would not explain such a difference with a strategy based on indicators.
I had similar issues only with corrupt data, since my terminal is always offline i never experienced anything like that anymore.
With global variables i ment: do you have something like GlobalVariableGet() in your code, this variables are not limited to test environment.
At last just to have asked all: any erros in the log?
@SDC, yeah pseudo ticks are generated identically for each run. From my experience way of the 90% accuracy, but still always the same.
Thanks again for your observations. I am now in the interesting situation of having two installations of metatrader giving the two extremely different sets of results (even when I copy the EA and the settings straight from one to the other and run strategy tester on both. This would point to the data, but it is still very difficult to see how it could explain it due to the reasons discussed. I am wondering what is the best way to see what differences there might be.
I have just checked the logs on the two runs on the separate installations of metatrader and there are no errors in either of them.
There is nothing like GlobalVariableGet() in my code. The EA uses 5 copies of a single custom indicator on 4 different time frames with one having a different parameter to the others.
I have found what may be crucial evidence, but I don't really understand its significance.
On the installation that was giving mediocre results, I logged out of the connection to the real money broker a/c and logged in again to the practice a/c on the same broker. After doing this (and nothing else) strategy tester gave similar spectacular results to the other copy of metatrader (running on the same practice broker a/c). Incidentally the broker is Oanda.
I am not sure how the metatrader client handles history data when it switches between different a/c's but the fact that all old data comes from the same downloaded 1 minute data rather than from the broker makes it very difficult for me to make sense of what is happening here (the trades at the start of the year are much better in the backtest run with a connection to the practice a/c than when there is a connection to the real a/c).
- Free trading apps
- Over 8,000 signals for copying
- Economic news for exploring financial markets
You agree to website policy and terms of use
I am very, very puzzled with results I am getting from metatrader's backtester. I am hoping someone else has had similar experiences, and may have an idea of what might be causing mine.
An expert I developed achieved excellent results in backtests and across a wide range of variations in optimisation and walkforward optimisation runs. Then suddenly, without me having consciously made big changes the results became mediocre/marginal. Then today again the results suddenly became superb again.
I now have backtesting reports with the exact same expert, the exact same dates, the exact same settings file and utterly different results. I have done my best to ensure the data is the same (reconstructing it from downloaded history data yesterday, I still got very different results today to then).
The only thing I found that could be different is the spread. As far as I can make out, MT4 assumes the spread throughout an entire backtesting run will have been the same as the spread right now. A rather strange "feature" of the system, but not the explanation of my observations, as with tight spreads today, results were mediocre, while with slightly wider spreads yesterday, the results were incredibly good.
After these experiences, I am in a position where I am not sure I can rely on the information I get from the software, and in desperate need of an explanation that I can use to avoid being misled (one way or the other) in the future.
Here are graphs of the mediocre results and the unbelievably good results for backtests with the same expert, the same period (2011/1/1 to 2011/7/26) and the same settings. The trades are typically open for hours or longer and not using take profits or stop losses (all entries and exits are based on the states of indicators only). The number of trades in the two runs are 211 and 173. There are losses in the second graph, but the profit factor is very high.