different backtesting results depending on dates tested - page 2

New comment
 
gordon:

With respects to everybody, I think u r overgeneralizing (IMHO).

That's what we're here for...


a. MT4 Tester has many technical limitations which makes the results differ greatly from what would have happened in a Live account

You'll get a different answer from everyone about this bit, but there are two things which particularly spring to mind. Firstly, the spread issue. One possible simple metric is that your s/l or t/p needs to be a decent multiple of the spread in order for MT4 backtesting to be reliable. Secondly, historic data tells you (or is supposed to tell you) the price at which someone got filled historically. That's not necessarily the price at which you would have got filled. For example, if your backtesting tells you that you would continually have got filled exactly on a day's low or high, then that's not, er, realistic.


b. Backtesting (whether in MT4 Tester or not) does not give any useful information about the performance of experts in the FUTURE.

I know precisely one serious player in the world of algorithmic trading, and his view would be the exact opposite of this: back-testing is everything, and forward-testing is meaningless - unless you do it for so long that it ceases to be "testing" in any meaningful sense. However, his views are pretty extreme. I've never seen a strategy discussed on this forum which he'd have any time for. He doesn't believe in identifying patterns in historic data - or, more accurately, he argues that modern computing power means that you will find patterns if you go looking for them, but the patterns will be chance.

 

jjc wrote >>

Firstly, the spread issue. One possible simple metric is that your s/l or t/p needs to be a decent multiple of the spread in order for MT4 backtesting to be reliable.

I usually require that they be a decent multiple of the average movement (high-low) in M1 bars... Same concept I guess.


Secondly, historic data tells you (or is supposed to tell you) the price at which someone got filled historically. That's not necessarily the price at which you would have got filled. For example, if your backtesting tells you that you would continually have got filled exactly on a day's low or high, then that's not, er, realistic.

I agree, but from my experience that usually averages out. You can test if it does - you need to run a series of identical tests that simulate cases where your t/p or s/l where slipped differently. This is a simplified example:

extern int seed_rand;      // we optimize on this variable, for example: start=1, step=1, stop=100
int max_slippage = 20;     // max expected slippage in points. setting to 20 is just an example, the value should be takes from actual Live account results

int init() 
   {
   MathSrand(TimeCurrent()*seed_rand);       // produce a different pseudo-random series for each test run
   }

int start()
   {
   // ...
   
   int simulated_slippage = MathRand()%(2*max_slippage+1)-max_slippage;   // random number from -max_slippage to +max_slippage
   tp = tp + simulated_slippage*Point;                                    // simulate slipped tp (this is a simplified example... ignores stoplevel limitation)
   
   // ...
   }

From my experience, with EA's that pass your first point (that have t/p or s/l that are a decent multiple of spread), running a test such as this will usually prove that unrealistic fills average out. You would usually find that the tests slightly differ in profit, but there won't be any extreme cases where a win becomes a loss.


I know precisely one serious player in the world of algorithmic trading, and his view would be the exact opposite of this: back-testing is everything, and forward-testing is meaningless - unless you do it for so long that it ceases to be "testing" in any meaningful sense. However, his views are pretty extreme. I've never seen a strategy discussed on this forum which he'd have any time for. He doesn't believe in identifying patterns in historic data - or, more accurately, he argues that modern computing power means that you will find patterns if you go looking for them, but the patterns will be chance.

I would agree to most of this. Proper forward testing is impractical (IMHO) and the time-saving advantages of backward testing outweigh the disadvantage of testing accuracy problems (a lot of which can be solved or measured to a certain degree). And yeah, the more patterns you search for the more u will find.

 
gordon wrote >>

With respects to everybody, I think u r overgeneralizing (IMHO).

I'll ask you all a simple question. Which one of the following is the reason u would not rely on the backtester (besides for debugging and functional testing):

a. MT4 Tester has many technical limitations which makes the results differ greatly from what would have happened in a Live account.

b. Backtesting (whether in MT4 Tester or not) does not give any useful information about the performance of experts in the FUTURE.

(if u have completely other reasons... then just add them).

This topic about BackTesting has been discussed left and right. There is no such thing as curvefitting. Understand?

If your system or your EA does "GOOD" in backtest but "NOT GOOD" in live or forward test, then it only means one thing.

It means that your strategy or your idea is no good. You need a mentor.

There is no such thing as curvefitting because who would be dumb enough to cherrypick and backtest on one currency alone.

It is simply impossible to curvefit because I assume that one should be smart enough to backtest on various different dates and various currency pairs.

And I assume one should be smart enough to know that backtesting is not about curvefitting for profit or for looking good.

Backtesting is about feasibility, stability. You know? How does the ea strategy stack up on various different dates and different currency pairs.

That kind of stuff.

 
gordon:

[...] I would agree to most of this. Proper forward testing is impractical (IMHO) and the time-saving advantages of backward testing outweigh the disadvantage of testing accuracy problems (a lot of which can be solved or measured to a certain degree). And yeah, the more patterns you search for the more u will find.

Another rule which this acquaintance of mine has is that he will run perhaps 25 optimisations, at most. And he will throw an entire strategy in the bin if the spread of results from the optimisation doesn't fit with a predetermined, expected pattern - i.e. an expected result from varying a parameter in a particular way. He's the only person I know who's anything like this rigorous but, then again, he's the only person I know with over USD 100,000,000 AUM. These things may be connected...

 
jjc:

Another rule which this acquaintance of mine has is that he will run perhaps 25 optimisations, at most. And he will throw an entire strategy in the bin if the spread of results from the optimisation doesn't fit with a predetermined, expected pattern - i.e. an expected result from varying a parameter in a particular way. He's the only person I know who's anything like this rigorous but, then again, he's the only person I know with over USD 100,000,000 AUM. These things may be connected...

Yeah, that would probably rule out 95% of the strategies discussed on this forum (i.e. TA based strategies).

12
New comment
Reason: