Simulation Reliabiliy

[John]

I have been testing some EAs on the Strategy Tester. I have used data (1M) from Alpari, FXDD, and Dukaskopy. Max Bars in History and Max Bars in chart were set to max 9s and the system defaulted to its maximum value. No mismatched errors and Quality 90% in all of them. I was expecting differences but not to the amount that I saw. Eg Difference in expected Pay off values of 5 points which is huge. So these differences were probably due to the differences in the data sets. I think this is a safe assumption. However the results were almost all towards the same direction. ie gain or loss. No differences there.Good news here. Issue 1 Obviously the quality of data is important. My broker offers its historical data (10years- tick or 1M up to 1Month - Bid and Ask). This is not 3rd party data. It is from their servers. So I suppose it should be reliable and with no gaps. The downside is that it is expensive. It costs 500USD. My concern is: Is it worth it? Yes the simulation will be more accurate. But if the EA is truly based on algorithms that do not use any "curve fitting" Technics during simulation, and were not designed to fit the curves of historical data, why should I need the very accurate data? If the simulation is genuine you can do the following: You can test the EA on the suggested Time Frame for the Suggested pairs and then you start testing all other pairs (using the spread of the pairs that actually interest you, so that you are consistent at your simulation). If you do that for a few years back, you have simulated all sorts of conditions, since each pair behaves differently in many ways (eg volatility). Then you repeat the process for similar timeframes as the suggested one for the EA you are testing. For example if the suggested timeframe is 1H you also test 30MIN and 4H. In my opinion, if we talk for 1M data it seems that it is not worth buying any historical data set. For the tick data, it might be a whole different story. So, I would like your opinion on that, especially from experienced users on EA testing and ESPECIALLY programmers that sell their EAs. Issue 2 On my argument above I have made the assumption that the EA is not "fixed" to fit the data. I will be the devil's advocate here. The Historical data is known to an EA programmer as it is known to all of us. Moreover, it exists in the MT4 platform during the simulation. It is a known factor. Who can ensure us that the EA is not programmed in such way, so that during simulation, instead of taking the data time frame by time frame and execute orders accordingly, it does look at future data and executes orders so that the end result "looks good" (but has nothing to do with real simulation result)? Is the simulation foolproof or there are such holes in it that the programmer can take advantage of? Does MQL offer any assurance on that for the products that go on its market? In conclusion. using tick data, or 1M high quality data might help. Simulating conditions by testing many years on different pairs might also help. But if the simulation is rigged, all these are useless. If it is not rigged, then there is no need for cent accounts etc when using a new EA. You start conservatively and you move on. I would like to hear opinions on these issues. From any user that has the same concerns, from experienced users, and especially from EA sellers. After all the EAs cost money. For some buyers might be a lot and for some others peanuts. But this is not a point. The point is that if the simulation on the Strategy Tester can be rigged then we are talking about potential fraud by some EA sellers.

[Francis Dogbe]

Are you trying to write a novel about backtesting...? hahaha. lol. just kidding.

As a matter of fact the main idea of backtesting is for us to observe how the EA works. It's certianly a mistake to use backtest to predicat the profitablity of an EA in the future. I have seen an EA that scores good marks in 15years backtest but yet it failed miserably in just one month of forwardtest. The market is always changing

[Ex Ovo Omnia]

Francis Dogbe:

Are you trying to write a novel about backtesting...? hahaha. lol. just kidding.

As a matter of fact the main idea of backtesting is for us to observe how the EA works. It's certianly a mistake to use backtest to predicat the profitablity of an EA in the future. I have seen an EA that scores good marks in 15years backtest but yet it failed miserably in just one month of forwardtest. The market is always changing

Yours is the developer point of view. From a trader point of view the backtesting is only used for verification of the performance. With the proper knowledge the trader recognizes a robust solution from the curve-fitting one. And for the test of robustness it is always valuable to add genuine trading data, preferably as the last confirmation.

[John]

Francis Dogbe:

Are you trying to write a novel about backtesting...? hahaha. lol. just kidding.

As a matter of fact the main idea of backtesting is for us to observe how the EA works. It's certianly a mistake to use backtest to predicat the profitablity of an EA in the future. I have seen an EA that scores good marks in 15years backtest but yet it failed miserably in just one month of forwardtest. The market is always changing

Hahaa.. I m just trying to figure out how it goes. Trying to figure out if it points to the right direction (ie profit or loss) and if it can be rigged by the developer.

MQL5 reviews each EA before releasing it to the market. I wonder if they check things like that.

And 1 month was maybe to short time.  In the 15 years for sure there were bad months as well.  Maybe you should give it more time.