Backtesting/Optimization - page 7

New comment
 
lomme:
Thats right.

But for backtesting finest granularity is 1min as well.

I can imagine, that tick data would not change results for 1min backtesting.

Did anyone backtest this data yet? I also agree that 1-M data will not make a significant of a difference unless the expert advisor uses excessive scalping, then even seconds may count.

 

Back testing and optimization

The first item to note is that optimization should only be used to verify an existing idea based on a behavior or fact of the market, not discover a great trading idea. The trading idea must come first. The danger in optomizing for a market is overfitting the curve, what worked on the historical data may not work in the future, but you found a great combaniation for the past data. It can however be used across several markets to find a set of parameters that should not be used, rather the sensitivity of the trading system. Another use of simulation/backtesting is to understand the weaknesses and strengths of your system. When does it perform well, when does it lag, and when does it just flat out fail, so you can develop new ideas to solve those problems.

At that point it is important to understand that the data is the most important aspect of back testing. Since backtesting is really just a simulation on historical data rather than a mathmatical distribution, so you need good quality data over various pairs that represents a varied number of market conditions. Bull,Bear, and sideways with varying volatility. If the system can with stand various market conditions including shocks then it is sufficient for trading. Without representative data the usefulness of your results will have no bearing on the real market.

Then you have to decide how many test need to be performed in order to achieve a confidence level in your results. This means running the back test over 15-30 data sets that are sufficiently representative and developing a confidence interval. This would assume that the data is normally distributed which financial data is not, it is skewed to the left with a thick tail on the right side. Therefore one option is to use the central limit thereom to improve your experiment and make it more reliable.

Tradestation's backtester is probably more reliable than metatrader because it can make tick trades rather than waiting for the closing price but that goes same for the EA's in metatrader. I do not think that metatraders back tester is terrible though it simply has to be used correctly, and using it without an understanding of the simulation process will always lead to overfitted results and unrealistic expectations. This will result in a loss of capital. Really the point of all this expect giving some tips is to understand that backtesting, optimizing, simulation are tools used to improve a system and compare improvements between systems not to replace the analyst designing the system. My suggestion for most if your using backtesting understand simulation methedology to save yourself a lot of headache and capital loss.

If i were really going to perform backtesting I would, and am trying to program a backtesting system into simulation software such as Arena or Witness, but am interested in trade stations abilities.

 

Back testing tutorial and tips

I have seen a lot of talk about back testing, and there appears to be a great deal of confusion so I rather than post on every thread pertaining to it I decided to post information here that may get long but I hope will clarify what backtesting is and how to use it PROPERLY.

Back testing is a form of simulation that uses historical data to evaluate improvements to a trading system.

The first thing to understand is that simulation is a tool much like charting software is a tool. It does not by itself discover an idea or even improve it, the analyst must do this based on observation of fundamental characteristics in the market. Normally simulation attempts to take historical data to improve a system and to reduce variance in the output of the system, but financial simulation is a special breed. Instead we are trying to alter the system to perform better against the variance in the data in which we have no control over. Therefore we want a system that is robust whose performance is not very sensitive to changing markets.

There are several parts to a simulation. There is the methodology, the system and the data. I will discuss each of these in detail following this post. I hope you find this information useful.

 

Simulation Methodolgy

When deciding to improve your trading system it is important to be systematic in order to engineer a successful result. Fortuantly a proven method for simulation has been developed. The first 6 steps should take about 40% of your time as should the last 5 steps. The experimentation step should only take 20 % of your time. The steps are as follows.

1. Define the problem - This is the most important step of the simulation process. You must clearly state what it is you intend to achieve through the simulation(backtest). Better put, where is your system performing poorly now. Is it not entering quick enough, exiting quick enough, is it trading too often, not often enough? Do you wish to compare two systems and decide which one is more valuable and reliable. These are valid problems.

2. Plan the project - This step is meant to help you make sure you have the tools, and a roadmap to succesfully carry out the experiment. Most of us have what we need in metatrader or other software including excel, but perhaphs you do not know enough programming to do some it yourself so you may need to find help. A Timeline is helpful to determine if your are getting done quick enough.

3. Define the system - In a financial simulation this step would involve deciding what markets you intend to trade, and what tools you're system uses to do this. Unlike traditional simulation you want to include as much detail as possible.

4. Conceptualize the model - During this step you will outline how your system works, and develop a rule base that will be programmed into the trading system. You may consider volume, breadth, momentum, oscillators, cycles, seasons, behavior techniques, adaptive techniques, risk control, money management, distribution systems or whatever aspects you wish to cover.

5. Preliminary Experimental Design - Up to this step I hope that all of you can work these ideas out in your head preety quickly but it is still important to write them down so you can review and compare your results when your done. This step involves deciding how you are going to perform the experiment. I will make a seperate post detailing experimental design.

6. Input Data Preperation - Good data is vital to any simulation, particularly financial simulation. Poor data could inflate or deflate your results. Not enough data is not representative of the market. I will make a seperate post detaling market data. During this step you will gather and organize your data into a format useable by the software. I think metatrader prefers date, time, open, high, low, close, volume colums. Then you will split it into various data sets used in the experiment.

7. Model Translation - This step is where you actually program in your system, at this point you should have had a good idea of how it was organized and broken apart into various functions so that it is a smooth process. There are normally two types of systems; integrated and vertical. An integrated system works together in one specific way while a vertical system can have parts added and removed.

8. Verification and Validation - This is a very important step in which you verify that the model represents your trading system and that the data is representative and accurate to various market conditions. In the strategy tester opening the chart after running a sample run is a good way to visually inspect that the system works as you had anticipated.

9. Final Experimental Design - After validating the idea make any changes to the system that you have realized are necessary in the process of going through the methodolgy. This may mean you noticed a new idea while studying the data, or that some new reporting measure was needed to satisfy your results.

10. Experimentation - This step is simply where you run the iterations of the experiment and collect the output data for analysis. This will probably also include recording the data.

11. Analysis and Interpretation - After the experiment has been run and output data collected you should analyze the data and interpret it. This is where you will decided if improvements to the system provide a valid contribution or if not.

12. Implementation and Documentation - Finnaly implement the improvements and document your results to refer back to and improve the life of the simulation.

 

Experimental Design Expanded

Note that many of the methodolgy steps you may have to run through several times if your forgot something in the process of performing the simulation. But one very important aspect is the experimental design.

There are two types of systems in the world. Stochastic(Variable Results) and Deterministic(Single Result). Financial Markets are the ultimate stochastic system next to the weather. It is full of variance and the results can very greatly from day to day and market to market. But what we want is a system that reacts to this highly variable data in a low varying way thus increasing our performance while decreasing our risk. Variance is the rule in nature and there are two types. Random variance and non random variance. Random variance is inherent in the system and cannot every be totally eliminated, this means that our system will always perofrm differently from month to month. Non random variance should be controlled. This includes performance outside the range of acceptability, or the systems reaction to large spikes or holes in data, power outages, loss of connection, poor programming. A stable system is one that has managed all of these through some measure.

The point of our experiment is to determine where our system is performing outside of stated requirements(unacceptable variance) and to attempt to improve it. So the first step is to define our metrics ( measured values). Meta trader does many of the important ones for us, drawdown, profit factor, percentage good trades. We may be concerned with functions of the system other than these such as the difference between entry signals and where it should have entered. This more complex metrics may have to be manually programmed.

After defining the metrics that are important to gather from your simulation/backtest you need to decide how you will carry out the experiment. Since market data is stochastic one run( a single test of the system on a single set of market data) is not enough to give us any confidence in the performance of the system. Statistics would encourage that we run several thousand runs but unfortunatly we dont have that much data or time. The number of runs depends on the variance found in the data. The does not mean the variance in the market data but rather the variance in the performance of your trading system. Everything about trading is understanding variance and controlling it. In this case its best to run independent runs (differnet data sets/markets), probably 15-30 runs, over sufficiently long periods of time to represent all kinds of markets, namely bull, bear, and nontrending. Non trending may be the most important for forex to represent to see how a trending trading system will deal with the range trading market, but it depends on your system.

From this information you can develop a confidence interval around your results. A confidence interval is a performance range you have a level of confidence for your metric to fall within. So based on your testing of 30 data sets you could say that you are 90% confident that your will make between 100,000 and 200,000 dollars. This is dependent on how representative the market data you used int eh simulation was of varying market conditions, preferabbly that you see them represented as many times as possible. If a new one arises never seen there is no reason your results will be similar to your confidence interval. Calculating a confidince interval is simple use x(mean)+-(student t value for alpha/2)*(s/sqrt(n)). Where the student t value can be found in a table and n is the sampel size. s is the standard deviation and alpha is your confidence level - probably .9 for 90%. The mean is the mean of your results for the metric in question probably profit.

IF your comparing two different systems read up on hypothesis testing, probably using an ANOVA test.

So what we learned is how many runs to make, how long they need to be, what they should represnt, and how to evaluate it to determine its signifigance. This may be a confusing section for many, feel free to ask questions.

 

Backtesting Data Expanded

I talked about the methodolgy and experimental deisng as well as purpose of backtesting. Now I want to talk about the data you use.

I'm getting tired so if something doesn't make sense I'm sorry HA.

Data is very important and must have severl characteristics to be evaluated appropriately. The data should be sufficiantly long, represent the conditions you will be performing in, accurate, and valid.

Expanded this means that the data should represent various market conditions several times if possible. Bulls, Bears, High volatility, Low volatility, severl price shocks, range markets, non trending markets, anything else you can think of because you will most likely run into these at some point and you want your system to be abel to handle them. One mistake many people make backtesting is that they do one run, run the optimizer and think they found the holy grail of trading systems. ALWAYS REMBEMBER if your have results that are really really good or really really bad as questions, somehting is probably wrong with your data, programming, or your overfitted to the curve. Overfitting is the process of optimizing your system to work on previous data reallly well. Unfortuantly you may never see market conditions like that again and your system is so optomized it is very sensitive to new market conditions and will crash quickly. In order to be representative it should last several years, if available several decades, and should not be included if a major change in the market has occured such as markets combining. I disagree with some and feel that market data is market data if its good data for back testing, as long as it represent some condition, so use your own judgement.

Gathering market data for testing can be a long trial, because its never in the format you want and always takes three times longer to procure than you hope.

Finnally the data needs to be independent this can be a problem in forex markets where all the dollar pairs are highly correlated, meaning they move in similar fashions. This is often broken even within one market with autocorrelation, meaning that the next peice of data is correlated or dependent on the last. Second observations should be identically distributed throughout the duration of the process. Again financial markets distributions change all the time but the basic shape is probably close enough to work. Finnaly they should be normally distributed. This is ALWAYS broken as prices tend to stay at lower leves, rather price distributions and price movement distributions are skewed to the left and have a thicker tail on the right that expected. Although this is what allows us to make money it violates a basic assumption of simulation. There are methods for correcting for this in analysis such as log transformations but the central limit theorem is usually easier to experiment with. This means that if you run 15 runs where each run value is the mean of 15 other runs this will tend to be a more normally distributed and be a more realistic confidence interval. If that didnt make sense just ignore it but remember that there is NO correlation between the results of backtesting and the physical system you trade with, its just a tool and a good one but it has to be understood and used properly.

Many other topics should be considered and may be later such as,

backtesting procedure

optimizing

statistics

probability

transformation

practical considerations

distributions and understanding them

random streams and syntehtic data

hypothesis testing

monte carlo optimization/faster than metatraders

but until then I hope this was somewhat helpful and you can perform better backtesting and achieve better results when you implement your trading system. Feel free to ask questions I know it was a lot.

 

Great explanation.

Thank You Cro2314

 

Short question about diffrent backstest system...

Is Tradestations software better for backtesting then Metatrader software?

Does anyone have experiance with both of them?

My conclusion so far is that tradestations backtests seems more advanced then metatrader, but on the other side metatrader is free and tradestation cost $$$...

Any feedback about the diffrents between them would be nice.

 

I only know a few

edit***May 27. Its important to note that backtesting is a simulation practice and not a simulator. One simulator that appears to work well is listed on this site as Forex tester. Can be used as a backtester but not with EA's yet. Simulations/backtesting is actually meant to provide statistics on the performance of your trading system so you can analyze the weaknesses. Ultimatelly attempting to reduce the variance in the slope of the curve of your equity over time. I wonder if that is why people do not like metatraders backtester. Maybe there are looking for a simulator. But if its a visual performance you are looking for simply open the chart after you have run the strategy tester and you can see exactly how the EA is trading. Very helpful for validation of the system.***

I actually have little experience with tradestations backtester. I do recognize a couple of advantages though. Tradestation can make intraday (tick) purchases rather than using closing price in the strategy tester, (just happened in newest version) while metatrader uses closing price. Depending on your strategy this can have significant consequences in your system if you trade very very quickly and try to catch the two or three bar moves that pop up once and a while. But goes against many fundamental strategies as well of waiting for closing price. The other difference is of course the language you can use to program with varies between them, and some common task such as suming variables appears to be easier on tradestation. Finnaly I have heard that there are some memory limitations within metatrader that are not limited in tradestation, but I haven't run into this. I don't know what limits it has on system variables such as those that keep an EA from loading its indicators onto the screen for you as in metatrader. It also appears the prewritten statistics that tradestation gives you are more robust on the backtest results. TradeStation def. has some advantages, but remember the back tester should react the same as the software will if you simply run a fully automated system. If thats the case I suggest backtesting on the same software as you use to automatically trade or else you may get surprised.

Over all I imagine it is programmed better, but unlike many people I beleive metatraders backtester works precisely as it is supposed to if used correctly and you have an understanding of how it executes your code. That is always the case with simulation software though so I don't see it as a drawback. What software you use is usually not as important as how you interpret the data it gives you. Now I'm just ranting so I'll stop here.

By the way thank you Felix for your compliment, it was greatly appreciated.

 

My backtester goes very slow .. help

Sometimes it's fast and sometimes it goes way too slow. I don't know why. I found a 1.5 GB file in the logs and I deleted it but it still is slow. Is their a better way to backtest programs? I'm using Metatrader and I often only have 20 % modeling quality.

1234567891011121314...95
New comment
Reason: