Can Identical Settings Provide Different Backtesting Results?
and the part related to code is not necessarily a bug. can be how the code is supposed to work. (like those garbage EAs using random entry direction on martingale/grid and whatnot)
if all factors staying constant, the results should be exactly identical. but that's a hard factor assumption i guess. (all factors staying constant)
and the part related to code is not necessarily a bug. can be how the code is supposed to work. (like those garbage EAs using random entry direction on martingale/grid and whatnot)
The EA doesn't do anything randomly. I have run tests on the MACD/Moving Average sample EA's that MT provide, the results are similar to my experience.
Using identical settings with the internet disconnected:
MACD Sample EA showed a profit of 155.59 on EURUSD on the first test, and then a profit of 155.59 on the second test.
On EURGBP, the first test showed a profit of 32.95, and the second test a profit of 33.02.
Moving Average EA showed a profit of 1945.55 on EURUSD on the first test, and then a profit of 1945.55 on the second test.
On EURGBP, the first test showed a loss of 592.14, and then the second test a loss of 592.06.
These results seem to be consistent with my experience, but haven't had the chance to scrutinise the logic of the sample EA's to understand if there could be any other reason for the difference.
koranged:
then would it be correct to say that the only reason for differing results would be some kind of bug in the EA itself?
I believe so. The only time I saw inconsistencies with backtesting results was when I found a bug in the Tester, which has since been fixed.
The EA doesn't do anything randomly. I have run tests on the MACD/Moving Average sample EA's that MT provide, the results are similar to my experience.
Using identical settings with the internet disconnected:
MACD Sample EA showed a profit of 155.59 on EURUSD on the first test, and then a profit of 155.59 on the second test.
On EURGBP, the first test showed a profit of 32.95, and the second test a profit of 33.02.
Moving Average EA showed a profit of 1945.55 on EURUSD on the first test, and then a profit of 1945.55 on the second test.
On EURGBP, the first test showed a loss of 592.14, and then the second test a loss of 592.06.
These results seem to be consistent with my experience, but haven't had the chance to scrutinise the logic of the sample EA's to understand if there could be any other reason for the difference.
when results are not consistent(profit wise), is the total number of trades changing as well ?
when results are not consistent(profit wise), is the total number of trades changing as well ?
No, the number of trades are equal on each test.
No, the number of trades are equal on each test.
Are you using "fixed spread" or "current"?
Are you using "fixed spread" or "current"?
Fixed. All the settings are identical from test to test.
In this case it does not matter because ...
Forum on trading, automated trading systems and testing trading strategies
Can Identical Settings Provide Different Backtesting Results?
koranged, 2018.10.16 17:53
The EA doesn't do anything randomly. I have run tests on the MACD/Moving Average sample EA's that MT provide, the results are similar to my experience.
Using identical settings with the internet disconnected:
MACD Sample EA showed a profit of 155.59 on EURUSD on the first test, and then a profit of 155.59 on the second test.
On EURGBP, the first test showed a profit of 32.95, and the second test a profit of 33.02.Moving Average EA showed a profit of 1945.55 on EURUSD on the first test, and then a profit of 1945.55 on the second test.
On EURGBP, the first test showed a loss of 592.14, and then the second test a loss of 592.06.These results seem to be consistent with my experience, but haven't had the chance to scrutinise the logic of the sample EA's to understand if there could be any other reason for the difference.
Assuming that the backtesting is done at a fixed spread, from date X to date Y, with tick data, and that the platform is not connected to the internet from the start to the end of the tests (to avoid the quote currency exchange rate differential at different points of the test), is there any reason other than there being some kind of bug located in the EA, that backtesting should provide different results from one test to another during the exact same period with the exact same settings?
Whilst using the open prices only method, I have been experiencing some noticeable differences from one backtest to another in terms of the net profit on the crosses. When testing the majors, I get an identical result on Test A as on Test B every time, but on the crosses, net profit changes from one test to another, sometimes albeit only by a penny on two, but sometimes much more.
It is worth noting that I haven't downloaded historical data from the MT History Centre, I am currently in the process of downloading tick data. I am hoping that using this will give me more reliable results.
Numerous threads on here have discussed the factors that can influence backtesting results from one test to another, things such as the spread, dates, quote currency conversion are the usual culprets, but if all these factors can be eliminated, then would it be correct to say that the only reason for differing results would be some kind of bug in the EA itself?
Are these normal single back-tests or multiple Optimisation runs?
Please note that the Strategy Tester saves data (cache files) for Optimisation runs and then uses the data in future optimisations when aplicable. However these cache files (.\tester\cache\), readily get corrupted for reasons unknown.
So I recommend deleting them before starting a new optimisation run so that you get clean results. This is what I do because I often get these strange results too on MT4.
EDIT: What build of MetaTrader v4 are you using?
In this case it does not matter because ...
When I do backtests with no internet connection I get strange results. I always thought the reason is MT4 does not know some of the symbol properties like tick value, tick size, stop level
- Free trading apps
- Over 8,000 signals for copying
- Economic news for exploring financial markets
You agree to website policy and terms of use
Assuming that the backtesting is done at a fixed spread, from date X to date Y, with tick data, and that the platform is not connected to the internet from the start to the end of the tests (to avoid the quote currency exchange rate differential at different points of the test), is there any reason other than there being some kind of bug located in the EA, that backtesting should provide different results from one test to another during the exact same period with the exact same settings?
Whilst using the open prices only method, I have been experiencing some noticeable differences from one backtest to another in terms of the net profit on the crosses. When testing the majors, I get an identical result on Test A as on Test B every time, but on the crosses, net profit changes from one test to another, sometimes albeit only by a penny on two, but sometimes much more.
It is worth noting that I haven't downloaded historical data from the MT History Centre, I am currently in the process of downloading tick data. I am hoping that using this will give me more reliable results.
Numerous threads on here have discussed the factors that can influence backtesting results from one test to another, things such as the spread, dates, quote currency conversion are the usual culprets, but if all these factors can be eliminated, then would it be correct to say that the only reason for differing results would be some kind of bug in the EA itself?