the annoying issue with the backtesting...

New comment
 

hi forum,

the last two weeks i made a life test with my EA wich in most of the times runs pretty good. the result was really bad, and i testet the same period with the same mashine in the strategy tester.

the results were completely different !

in the realtime result the final margin was 1600 ( begun with 10,000). in the backtest, the result magin was a bit more than in the beginning.

The more funny is, that I testet the same bot with a little different strategie on another mashine. the result was different, too, between backtest and realtime test.

the problem i have now: how can I optimize my bots if all the backtesting data doesnt have anythign to do with the realtime data?

I allready tested with importing data like described in the forum several times, but this doesnt change so much.

how do you guys test you bots for realtime ?

thanks for your hints..

filip

 

Hi,

I'm fairly new to this as well. In my limited experience along with looking at other's postings like yours, I have these things to offer:

1st: Many broker's 'stack the deck' and purposely alter the data so that your EA will appear to do better than it will in real live 'forward testing'

2: I and others have found back testing to be so inaccurate that it is completely meaningless! When you are checking the results are you looking at and taking into account what its accuracy rating is?

Along with this dilemma is the problem of 'over optimizing' on the same data set. In an effort to optimize an EA, in order to get results that are properly correlated to each other, many people always reuse the same data set for this understandable reason. But then it is optimized for that specific data set only, and not the reality of live trading where the data is always changing and unknown.

I think that the closest you can come is to ‘forward test’ on a demo account. Here again many have found that the dealers are biasing and skewing the results so that they appear better on a demo account than a live trading account )< 8(

I currently have a fairly decent commercial EA running on one currency pair on my live account, but also have the same EA with 5 variations running on a demo account with the same currency pair as I have it trading on with my Live account. While this may not give the same results that you may find on a Live trading account, you can find out what the settings bring the best results for you for that particular EA and currency pair this way.

From the info I initially found here I have a number of different instances of MT4 installed in order to do these things. But I have recently discovered that is unnecessary. You can open up multiple instances of the same MT4 and just log onto different accounts from only one installation of MT4. A word of caution from something I learned the hard way: I got caught with leaving what I thought was running some forward test on my Demo account when in actuality it was on my live account! When my MT4 opens up, it logs onto the last account that was open when the MT4 client is closed, which is an option that you can change in MT4. So I have gotten into the habit of always disabling the EAs with the button for this purpose before I close any MT4 clients down. Then I do not have to worry that it will start trading on the wrong account, on the wrong currency pairs, with the wrong EA etc. until I get it configured how I want it to be and only then do I reactivate the EAs to let them begin trading.

I hope this helps a bit. . . . . . . (8 >) . Prosperous Trading . (< 8)

 

On backtesting, I have found that it is useful only for debugging your EA, and then only if you have coded liberally with Print() statements to monitor various operational parameters, variables and sub-routine executions. I always include an extern bool Verbose input to toggle these debugging logs.


Backtesting does not prove much of anything else. Even if it did, the problem of over-fitting to the test data is not only present, the "optimize" feature of MT4 makes it almost a certainty! Think of something like optimizing the Stop Loss setting -- you are totally dependent on the volatility of the test data. An SL setting that maximizes profits in your test data may very well maximize losses in the real market, broker tampering notwithstanding.


I only trust forward testing, and that I only trust partly. I'm trading without a broker counter-party, so I don't have to worry about counter-party tricks, but I still have to worry about the Liquidity Provider banks and the market itself, which is plenty of worry indeed -- execution varies, not because of cheating, but because that's the nature of the Forex spot market.


Your most reliable testing is Real Money testing. If you can find a broker that lets you trade micro-lots at the same leverage as mini and standard lots, and has the micro lots on the main feed to the LP, then you can test your EAs for pennies on Real Money. This is the only way to know that the same EA on the same pair with the same settings will perform the same when you start throwing more significant money at it. Otherwise, it's a roll of the dice.


My 2 cents. YMMV.


RS

 
DougRH4x:

Hi,

I'm fairly new to this as well. In my limited experience along with looking at other's postings like yours, I have these things to offer:

1st: Many broker's 'stack the deck' and purposely alter the data so that your EA will appear to do better than it will in real live 'forward testing'

2: I and others have found back testing to be so inaccurate that it is completely meaningless! When you are checking the results are you looking at and taking into account what its accuracy rating is?

Along with this dilemma is the problem of 'over optimizing' on the same data set. In an effort to optimize an EA, in order to get results that are properly correlated to each other, many people always reuse the same data set for this understandable reason. But then it is optimized for that specific data set only, and not the reality of live trading where the data is always changing and unknown.

I think that the closest you can come is to ‘forward test’ on a demo account. Here again many have found that the dealers are biasing and skewing the results so that they appear better on a demo account than a live trading account )< 8(

I currently have a fairly decent commercial EA running on one currency pair on my live account, but also have the same EA with 5 variations running on a demo account with the same currency pair as I have it trading on with my Live account. While this may not give the same results that you may find on a Live trading account, you can find out what the settings bring the best results for you for that particular EA and currency pair this way.

From the info I initially found here I have a number of different instances of MT4 installed in order to do these things. But I have recently discovered that is unnecessary. You can open up multiple instances of the same MT4 and just log onto different accounts from only one installation of MT4. A word of caution from something I learned the hard way: I got caught with leaving what I thought was running some forward test on my Demo account when in actuality it was on my live account! When my MT4 opens up, it logs onto the last account that was open when the MT4 client is closed, which is an option that you can change in MT4. So I have gotten into the habit of always disabling the EAs with the button for this purpose before I close any MT4 clients down. Then I do not have to worry that it will start trading on the wrong account, on the wrong currency pairs, with the wrong EA etc. until I get it configured how I want it to be and only then do I reactivate the EAs to let them begin trading.

I hope this helps a bit. . . . . . . (8 >) . Prosperous Trading . (< 8)

 

-differences between tester and demo are depending strongly on your strategy. if you trade higher timeframes >=H1 the result will be quiet the same. (on the same period of course, past results are never indicative of future results)

here are some things, when implemented and trading higher timeframes the result will be quite the same.

1) enter/exit the market with limit/stop orders.

2) strategy should not depend on data from the current candle

3) never ever open more market order in the same tick. this is never possible on forward test!!!

4) stritly rigorous error handling!

5) test on real market conditions, let a spreadlogger run on your demoaccount and then set the spread of the test-machine to a higher level than the average spread is.

6) always test in offline mode

if that all i implemented and your not trading m1/m5 the results will not differ more then 2-5pips per trade on average. and that is mostly because of the spread.

-

regarding optimisation: simply don't use it. and if you want use it. optimise it on a short period and then test in on the whole dataset. if results are comparable, well then it's not over-optimized.

-

my 2 cents

New comment
Reason: