Need Help On Optimization Result - page 3
You are missing trading opportunities:
- Free trading apps
- Over 8,000 signals for copying
- Economic news for exploring financial markets
Registration
Log in
You agree to website policy and terms of use
If you do not have an account, please register
I find it hard to decide an optimal setting from the result of an optimization.
Could anyone please tell me which is the best to choose from an optimization result, the lowest drawdown, the profit factor or any other thing that I should consider?
Thanks!
Just to share my experiences with this, I made a discriminant analysis on these 6 columns to see if I could guess which settings would perform the best forward comparing to a random pick and my conclusion is that it's not usable: It's choosing the best ones when the agent is adapted to the market and chosing the worse ones when it's not, without beeing able to know if it's adapted at the present moment.
With the agent I used, the most dicriminant column was the number of trades, and the best choice was a sort of subtle balance with number of trades, profit and drawdown, but I couldn't express a clear rule.
Now I'm trying what zzuegg is suggesting: I take every profitable setting from optimisation and test it in the past in order to keep the best ones. The idea is to have settings that are at the same time adapted to short and long term market. As you can imagine, the goal is to make automatic optimisation.
Just to share my experiences with this, I made a discriminant analysis on these 6 columns to see if I could guess which settings would perform the best forward comparing to a random pick and my conclusion is that it's not usable: It's choosing the best ones when the agent is adapted to the market and chosing the worse ones when it's not, without beeing able to know if it's adapted at the present moment.
With the agent I used, the most dicriminant column was the number of trades, and the best choice was a sort of subtle balance with number of trades, profit and drawdown, but I couldn't express a clear rule.
Now I'm trying what zzuegg is suggesting: I take every profitable setting from optimisation and test it in the past in order to keep the best ones. The idea is to have settings that are at the same time adapted to short and long term market. As you can imagine, the goal is to make automatic optimisation.
IMO, Using the optimizer as a hope generator can be tricky business. As Phillips always says "it's like expecting the market to behave exactly like it did during the optimization periods". The optimizer is a # cruncher which tailor fit results based (usually on) sl_tp or period_time. Having gone through it myself, I suspect that allot of promising parameters performs poorly outside the optimized periods or in forward testing.
IMO, One should create strategies independent of the Optimizer, test it on 2010 data and if it shows promise, move to 2009,2008,2007...etc. Even with the best of results for the last 10 years, you still have to keep your cool and not think you have a Holy Grail because none exist. IMO, Any strategy one employs on the market carries Risk, like un-controllable risks for example.
IMO, the best usage for tools like the ones provided by Phillip, Gordon, WHRoeder, BB and all the other gifted analysts/programmers on this site, is to help compare if System-A <is better than> System-B. So, have I created any system that pass 10 year back-test on first run? ... heck no. But lately, I've been programming more systems which does well in 2010->2008 and breaks somewhere along the way. This is progress compared to newbie textbook programs which looked like if RSI>80 && MA-1 crosses MA-2 then Sell. Shooting a system like that through the back-tester would 99% of the time Fail. Then most fall for the trap of putting it through the Optimizer.
With my newly formed systems, I'm still tempted to just look at the short periods where it didn't perform well and find ways to exclude that from the system. But my fear is thats just another form of Curve-fitting. I'm not just talking about hard coding (Monday 1-5-2003=don't trade) that would just be plain wrong lol. I mean using any type of Indicator or Oscillator. When a system fail like that the best approach for me is to log what works to memory and archive the EA, then start programming from scratch.
Anyways, nice thread guy. I'm really enjoying following this one.
With my newly formed systems, I'm still tempted to just look at the short periods where it didn't perform well and find ways to exclude that from the system. But my fear is thats just another form of Curve-fitting. I'm not just talking about hard coding (Monday 1-5-2003=don't trade) that would just be plain wrong lol. I mean using any type of Indicator or Oscillator. When a system fail like that the best approach for me is to log what works to memory and archive the EA, then start programming from scratch.
Yes, and i tried as well to cut the history in several distinct periods according to different levels of slope or ATR, but optimizing on discontinued periods is more or less like optimizing on the last three months, the curve-fitting risk is still there like at the first day.
The hardest thing to understand is that even when you arrive to an agent that is profitable on the last 10 years when optimized on the last months with large range values, you are still not sure of how much you are over-optimizing on historical data, because to get there you were throwing away hundreds of nice ideas, which is exactly what the genetic algo of the optimizer does.
No, really, we have a tough job.
The way you use risk of ruin calcs is you define a time period (monthly, weekly, yearly, etc), or an increment of some kind (per trade, per 10 trades, etc), and then you assemble the statistics for those periods. It matters in terms of what the resulting output of the risk of ruin calc actually means (its units).
Sometimes it helps to first focus on making sure you are asking the right question (in other words make sure the answer you are trying to compute is actually going to answer the question you need/want answered).
For example, my client's operate on a monthly timeline. They don't care about the daily or weekly stuff, for them its all about the monthly results. So in my case it does no one any good for me to compute risk of ruin for weekly results or on a per-trade basis.
As such, what I must do in the backtest is capture the average rate of return (profits, losses, ROR, etc) on a monthly basis. Something like the following:
To compute risk of ruin metrics you must first assemble the data for your time frame. In my case in this example I needed to assemble the monthly rate of return results, compute the average monthly ROR and then compute the standard deviation of those monthly rates of returns.
In this example the average monthyl ROR was 12% with a standard deviation of 8%...from the backtesting results this should have been golden going forward, only it wasn't (as you can see). Note I refer to risk of ruin as risk of loss (ROL) as I'm not interested in knowing when my account will be ruined (that is the trivial question) but rather I am more interested in knowing how frequently I ought to expect the account to be underwater by some fixed amount, say 20%.
So a 12% monthyl ROR with an 8% standard deviation means I can expect my account to experience a 7.4% loss in account value around 6.5% of the time (6.5% of the end of month events, or about once every 15 months).
So while I was initially caught off-guard by the seemingly inexplicable forward test result, it was perfectly reasonable to expect such a monthly result based on the backtesting results, once very 15 months I can expect such a negative drawdown on my account's highwater mark.
So you see, in order for you to compute a meaningful risk of loss assessment you must first decide what time period is pertinent to you and then you must assemble the results in a form that are amenable to computing the average and standard deviation of the RoR in that time period.
Once you have that data from the backtest you can then perform a myriad of statistical-based analyses using risk of ruin (or risk of loss as I like to refer to it) calcs.
these are my results for 30 groups of 50 trades ( from 18/03/2001 - 21/12/2010 )
with initial deposit of $700
for the fllowing I have computed the:
mean ROR of => 2.9%
stdev of => 6.5%
(which is quite bad.... right ? )
now for each group of 50 trades.. the maximum % of loss I can expect is: $0.1 X 50 X (20pips+3pip spread ) = $115
115/700 = 16.5%
I can't lose more than that per 50 trade session
Is there a way I can take that into consideration when computing the risk of loss ?
for the fllowing I have computed the:
mean ROR of => 2.9%
stdev of => 6.5%
(which is quite bad.... right ? )
Yeah you want your μ/σ ratio to be >1, preferably 2 at a minimum.
And Sharpe is not enough, the σ value itself matters in determining ROL at any given Sharpe value (this is a not so well known fact, but it is the fundamental reason that leverage is problematic and is why the CFTC is reducing max leverage in the US).
This is why the mantra exists that "Beta kills". The best μ in the world isn't going to save your account from eventual risk of ruin of the σ is large. Leverage makes σ large, excessive position sizing makes σ while people are seeking large μ.
But if the only way you can achieve large μ is by making σ large then you are predestined for ruin. A mathematical certainty that many are not prepared to comprehend or investigate, so the CFTC is going to do it for them by forcing a cap on just how large σ can become by taking away the novice's ability to dial up large position sizes.
There is a lot of outrage directed at the CFTC over their leverage controls in forex but the CFTC is really trying to save people from their own worst enemies.
now for each group of 50 trades.. the maximum % of loss I can expect is: $0.1 X 50 X (20pips+3pip spread ) = $115
115/700 = 16.5%
I can't lose more than that per 50 trade session
Is there a way I can take that into consideration when computing the risk of loss ?
Risk of loss itself is a probability distribution, the maximum value that you stand to lose ($115) is just one point on that distribution. What you are probably interested in is knowing how frequently such a scenario is likely to occur. Will you experience a $115 loss once every 5 yrs or once every 5 months?
Those are the questions you use RoL to answer. "How frequently am I to expect to lose XYZ?" You are asking how likely are you to lose $115, the max allowed RoL based on your steps, and the answer is 9%.
Oh ...
There is one thing I still don't understand.... aren't there any other factors influencing this calculation ??
Lets take your ROR results computed on a monthly basis
Lets assume (for the sake of argument) that the more trades there are in a month the higher the ROR you get
for example : for 0 trades you earn 0%
for 10 trades you make 5% ( on average)
for 20 trades you make 10% (on average)
and so on...
lets assume in for that period of 30 months there was a high spread of values in the ROR distribution ( for each month there was a completely different number of trades made )
so that the mean return is 10% and the standard deviation is 20% and there were no negative ROR ( you can't have a negative number of trades... )
now the computation of the risk of loss of one standard deviation which is 20% will give us approximetly 37% ... which doesn't really make any sense..
Do you see where im going here ?
I'm really not trying to be daft, but I'm not following. I don't comprehend the argument you are positing here. Flip a coin 10 times a month, or 20 times a month, the probability of getting heads is always 50/50 on every try regardless the history of coin tosses to date. That the reality of coin tosses differs from the statistics at times does not make the statistics invalid, just makes them of limited utility.
The purpose of computing RoL is that it is something that is mathematically valid, but it is not guaranteed. It is merely a guide, a way of assessing results, that is better at capturing and presenting the indicative (systematic) risk in your trade strategy. But chance is just that, chance. There is a chance you'll never experience a loss, and there is a chance that you will experience total loss.
RoL helps you compute this chance in a way that lends itself to rank-sorting and downselecting trade strategies and optimized parameters. It is no holy grail for it cannot predict the future, but if the future is statistically comparable to the past then you have legitimate claim to attempt to map the statistics of the past to that of the future (this is what weather modelers do in preparing their forecasts with ensembles of models) in an effort to not predict the future market but to merely predict the statistical character of the future market.
Its about the best you are entitled to hope for, really.
Thanks alot for your time !! I really appreciate this
so by the following test results I've just been extremely lucky ??
the maximum amount of consecutive losing trades for this whole period (of approximately 23 groups of 50 trades - couldn't squeeze the whole thing into one test ) was 41 (which may split amont 2 groups of 50 trades )
I should've seen that kind of drawdown more frequently ?
eventually If I continue "playing" this game I will get 9% of the time that kind of lose strike ?
Now one more thing
according to this report
the average chance for me to have a profitable trade is 8.85% and a losing trade is 91.15%
so according to this the chance of having 50 consecutive losses is : 0.9115^50 = 0.97% ...
which is quite far away from 9% ... how can this be ??
If I were to lose by 95.3% of the time it would be correct, and with such percentage my expectancy would be negative O_O
Ex= 0.953*(-23)+0.047*(247) = -10.31 pips profit per trade
One thing that I see here is that you are ascribing the distributions the functional form of a gaussian distribution whereas my understanding is that the risk of ruin calculations were derived based on poisson/stochastic process expectations.
If you want to understand the derivation of the equations I recommend you pursue reading the following book: The Theory of Stochastic Processes by Cox and Miller
They are the cited authors of the risk of ruin equations.
In the course of my own work to implement risk of loss assessment into my backtest/forward test characterizations I have taken the equations from Cox and Miller as well as Chamness, done some simple substitution of variables and (IMO) a helpful integration step and the resultant equation helps me rank-sort trade strategy parameters on the basis of a risk of loss on a normalized ROR basis. (similar to 1 over the RAROC)
One thing that I see here is that you are ascribing the distributions the functional form of a gaussian distribution whereas my understanding is that the risk of ruin calculations were derived based on poisson/stochastic process expectations.
If you want to understand the derivation of the equations I recommend you pursue reading the following book: The Theory of Stochastic Processes by Cox and Miller
They are the cited authors of the risk of ruin equations.
In the course of my own work to implement risk of loss assessment into my backtest/forward test characterizations I have taken the equations from Cox and Miller as well as Chamness, done some simple substitution of variables and (IMO) a helpful integration step and the resultant equation helps me rank-sort trade strategy parameters on the basis of a risk of loss on a normalized ROR basis. (similar to 1 over the RAROC)
its wrong to treat it the way I do (as a guassian distribution) ?
seems like a heavy book
but bought it anyway... will try to dig in and understand
thanks !
It's not wrong, its just not going to give you the results you think you are computing.
For example it is not wrong to say "1+1 = 2"...as the math is obviously correct.
But you would be wrong to say "1+1=2, this means I can expect to double my account in 6 months because the number from my math is a 2".
At any rate you'll want to direct your attention towards the derivations that lead up to equation 91 on page 61.