What's your opinion on optimization? I need some advice.
From what you described, yes, I would probably say that.
No, I would not agree to that.
The whole idea of curve fitting is that you have tuned your strategy to fit the data used. You aren't considering the robustness of the strategy itself.
To decide the robustness of the strategy, you might want to take a look at:
- out-of-sample testing
- walk-forward analysis
- monte carlo analysis
Hi,
I've been working on a EA for some months and got some very interesting results without optimization. Below is a graph using fixed 0.01 lots from 2012 to 2017 on EURUSD:
The problem is that in 2012 there was a situation where my account could have easily reached a margin call (I mean, it reached 43% drawdown, but I don't like anything above 20%).
As you can imagine, this type of graph behavior (with DD spikes) means that my EA uses several strategies that a lot of people don't like (hedge, averaging, martingale). But I think that if well made, these are powerfull tools to make money - as you can see from 2013 to 2017 there were almost no DD.
My question is: I was thinking of running an optimization for 15 years of data, for example 2003-2018.
You will propably say that I will only be curve fitting my EA and you may be right. But on the other hand, the EA will be optimized for how this particular forex pair behaves, right?
How long it stays trending/ranging, the average size of trends, ranging periods and sizes, etc. And because I use renko charts there are less possible mutations on market behavior (there are no candle patterns, no spikes, etc, only waves up and down, all with the same candle sizes).
So my thinking is: With enough information, my EA will know what are the best parameters to use based on how this forex pair has performed over the past 15 years. After all, how much and how fast does the market behavior change?...
I don't know if I am being too opttimistic, that's why I am asking your opinion.
Thanks!
PS - If you want to discuss further about the EA itself (it's strategy), let me know.
Hi Rafael !
By optimizing, you logically try to fit the best to the sample used. It's always a good idea to have a graph with loss deals as you know you won't never ever avoid it : we already talked of together, you could never know when these huge drawdown will happen in the future, what matter the most is how these situations are handled/recovered by the EA.
Could you show us the same ea with the same setting on GBPUSD & USDJPY on the same period ? With these 3 majors it's possible to make yourself a solid opinion on how the ea performs itself.
...
To decide the robustness of the strategy, you might want to take a look at:
- out-of-sample testing
- walk-forward analysis
- monte carlo analysis
Anthony, thank you for your answer.
Out-of-sample testing:
So optimizing for a period, for example 1 year, then running the EA with the optimization on all other 14 years would be a better idea? That would "prove" the robustness of the EA.
I think that is the definition of out-of-sample testing, right?
But I don't see how that would make the EA more robust. Wouldn't the EA be more "robust" if instead of optimizing it for 1 year, optimize it for all 15 years? More data analyzed = more optimized = better chance of being successfull in the future, because it has "seen" and learned more.
Maybe my way of thinking is wrong, but I am just trying to understand what is the better approach (if there is one...).
Walk-forward anaysis:
This means putting the EA to run live, either on demo or real account, correct? Again, I don't see the point of it. If it runs as predicted during the walk-forward test, it doesnt mean it will continue that way afterwards. Isn't walk-forward test the same as backtest, but slower? You are only seeing what already happened.
Monte Carlo analysis:
While I have heard about it, I never used it. I don't think there is an automated way to test it, there is?
Hi Rafael !
By optimizing, you logically try to fit the best to the sample used. It's always a good idea to have a graph with loss deals as you know you won't never ever avoid it : we already talked of together, you could never know when these huge drawdown will happen in the future, what matter the most is how these situations are handled/recovered by the EA.
Could you show us the same ea with the same setting on GBPUSD & USDJPY on the same period ? With these 3 majors it's possible to make yourself a solid opinion on how the ea performs itself.
Icham, hello again :) I am still working on that EA.
You said "you could never know when these huge drawdown will happen in the future". That's exaclty my point here. I'll try to explain:
Market behavior tend to repeat itself, so If I optmize for 15 years of data, my EA will be ready if market keeps doing it's "cycle". If market changes unexpectedly, I could have a trigger to close all trades at for example 20% drawdown.
The point is, the most amount of data used on the optimization the better, right? Because it will be prepared for more diverse situations.
I could even run optimization on several pairs and find a set of parameters that could be used on all of them. For example, find the parameters that fit both EURUSD, GBPUSD and USDJPY. Actaully, my idea is to run this EA on 8-10 pairs.
Again, I may be wrong about what I am saying, but to me it seems to make sense.
Rafael Grecco:
So optimizing for a period, for example 1 year, then running the EA with the optimization on all other 14 years would be a better idea? That would "prove" the robustness of the EA.
Nothing "proves" that an EA is robust. You are shooting for statistical probabilities.
Out-of-sample testing is where you have a training set and a test set. Some people use 60%/40%. Some people use 50%/50%. Look at the "Forward" option on the MT5 tester. The name "Forward" is confusing because it's not really the same as the forward-testing I will describe below.
Rafael Grecco:
Wouldn't the EA be more "robust" if. . .
What makes an EA more robust is if the optimized parameters work on a training set -AND- on an out-of-sample test set. If it doesn't work on an out-of-sample test set, there is no confidence it would work on future market data.
This means putting the EA to run live, either on demo or real account, correct?
What you describe is often referred to as "paper trading." That's also a good idea, but that's not walk-forward analysis.
Walk-forward uses sliding time windows.
More information: https://en.wikipedia.org/wiki/Walk_forward_optimization
Anthony, thank you for your answer.
Out-of-sample testing:
So optimizing for a period, for example 1 year, then running the EA with the optimization on all other 14 years would be a better idea? That would "prove" the robustness of the EA.
I think that is the definition of out-of-sample testing, right?
But I don't see how that would make the EA more robust. Wouldn't the EA be more "robust" if instead of optimizing it for 1 year, optimize it for all 15 years? More data analyzed = more optimized = better chance of being successfull in the future, because it has "seen" and learned more.
Maybe my way of thinking is wrong, but I am just trying to understand what is the better approach (if there is one...).
Walk-forward anaysis:
This means putting the EA to run live, either on demo or real account, correct? Again, I don't see the point of it. If it runs as predicted during the walk-forward test, it doesnt mean it will continue that way afterwards. Isn't walk-forward test the same as backtest, but slower? You are only seeing what already happened.
Monte Carlo analysis:
While I have heard about it, I never used it. I don't think there is an automated way to test it, there is?
Icham, hello again :) I am still working on that EA.
You said "you could never know when these huge drawdown will happen in the future". That's exaclty my point here. I'll try to explain:
Market behavior tend to repeat itself, so If I optmize for 15 years of data, my EA will be ready if market keeps doing it's "cycle". If market changes unexpectedly, I could have a trigger to close all trades at for example 20% drawdown.
The point is, the most amount of data used on the optimization the better, right? Because it will be prepared for more diverse situations.
I could even run optimization on several pairs and find a set of parameters that could be used on all of them. For example, find the parameters that fit both EURUSD, GBPUSD and USDJPY. Actaully, my idea is to run this EA on 8-10 pairs.
Again, I may be wrong about what I am saying, but to me it seems to make sense.
Sure it makes sense : an optimization on 15 years is better than a monthly one. Or like Anthony said it, forward testing 1 years + 2 years out of sample is great too.
Live testing is good but not with this long term strategy, after one month you can conclude it's alright while there weren't these dangerous cases making that strategy so hard to manage, the bad surprise can happen after a whole year waiting to withdraw, and you loses all the gain suddenly.
I truly recommend what I'm doing with my hedging's one, a deep correction of this intrinsic lack. it's easy to say that you'll add unlimited hedging orders, it's easy to say that you'll eventually increase your deposit, it's better to correct it. i wouldn't run myself on live without a reliable solution, but that's only my humble opinion.
Here is my thought on EA optimization in general.
Well, if you have too many parameters in your EA, you can always come up with phenomenal backtesting result by optimizing these parameters but it will be like a mirage because EA will fail terribly on live markets.
There should be some degree of freedom available in a trading system so that it can adapt to the changing market conditions.
"Loose pants fit all but there is a problem, they look awful"
- Perry Kaufman
As you said your EA is using a strategy like martingale, averaging etc. So, it will be even more difficult(but not impossible) to optimize it for real trading because the margin-call event that will occur in the future would be totally different from the margin call event that was occurred in the backtest.
"If you can't take a small loss, sooner or later you will take the mother of all losses"
- The new market wizards.
So in your situation, my suggestion would be that you should optimize it for only 2-3 parameters at max by keeping all the other parameters constant.
Do a rollover walk-forward optimization(instead of static walk-forward) and perform a Monte-Carlo simulation on the out-sample result. If your EA passes both of them, You can go live with some confidence.
My opinion:
Backtesting as well as optimization is a complete waste of time.
It does not generate any profits.For this strategy particularly, it absolutely has to be corrected from the inside.
You optimize for 15 years, you suppose markets are cyclics but they aren't really, at the 16th year aka the next big change on market (brexit) it's the one which will make your ea drawdown 46%. What is 46% ? Half the balance.
You have more chance of survive it with a MACDSample optimized on 3 years than with this one.
This strategy is super frustrating, I am the first to say it : have a break.
Hi,
I've been working on a EA for some months and got some very interesting results without optimization. Below is a graph using fixed 0.01 lots from 2012 to 2017 on EURUSD:
The problem is that in 2012 there was a situation where my account could have easily reached a margin call (I mean, it reached 43% drawdown, but I don't like anything above 20%).
As you can imagine, this type of graph behavior (with DD spikes) means that my EA uses several strategies that a lot of people don't like (hedge, averaging, martingale). But I think that if well made, these are powerfull tools to make money - as you can see from 2013 to 2017 there were almost no DD.
My question is: I was thinking of running an optimization for 15 years of data, for example 2003-2018.
You will propably say that I will only be curve fitting my EA and you may be right. But on the other hand, the EA will be optimized for how this particular forex pair behaves, right?
How long it stays trending/ranging, the average size of trends, ranging periods and sizes, etc. And because I use renko charts there are less possible mutations on market behavior (there are no candle patterns, no spikes, etc, only waves up and down, all with the same candle sizes).
So my thinking is: With enough information, my EA will know what are the best parameters to use based on how this forex pair has performed over the past 15 years. After all, how much and how fast does the market behavior change?...
I don't know if I am being too opttimistic, that's why I am asking your opinion.
Thanks!
PS - If you want to discuss further about the EA itself (it's strategy), let me know.
Hi! All martingales fail (sadly), that is a mathematical fact, there is NO way to escape it really, and what is worst... If you have an edge you are wasting it completely on this as you cannot compound your gains without skyrocketing your risk even further. Hedging on the other hand may cause a very painful margin call sooner or later, although it can help you control "when" you take your losses (some manual traders prefer this than the traditional SL).
About optimization, it is unavoidable really. People often fool themselves when they think they are not "optimizing" their strategies but still cherry pick, ignore or play darts with the parameters, so you are much better knowing what is good unless you are using a "parameterless" strategy (ie: fractals, price action, etc..). To avoid overfitting:
- Use a long period for backtesting (10 years is my personal favorite), but don't try to go back a LOT in time for the sake of it, just try to catch every kind of market for a particular symbol.
- The more trades the better for statistical significance, but also consider the average winner size as these strategies (low winner/loser size) are affected a lot by commisions and slippage.
- Use "simple" strategies, the less parameters the better.
- Keep in mind that hard stops and "trailing" add more layers of complexity, if you don't need them don't use them.
- Do not optimize optimized results.
Nothing "proves" that an EA is robust. You are shooting for statistical probabilities.
Out-of-sample testing is where you have a training set and a test set. Some people use 60%/40%. Some people use 50%/50%. Look at the "Forward" option on the MT5 tester. The name "Forward" is confusing because it's not really the same as the forward-testing I will describe below.
What makes an EA more robust is if the optimized parameters work on a training set -AND- on an out-of-sample test set. If it doesn't work on an out-of-sample test set, there is no confidence it would work on future market data.
What you describe is often referred to as "paper trading." That's also a good idea, but that's not walk-forward analysis.
Walk-forward uses sliding time windows.
More information: https://en.wikipedia.org/wiki/Walk_forward_optimization
Of course! I see it now, it doesn't matter the period I use to optimize if I don't have out-of-sample data to test the optimization. It makes perfect sense. I'll read about walk-forward and use it to test me strategy. Thank you!
...
As you said your EA is using a strategy like martingale, averaging etc. So, it will be even more difficult(but not impossible) to optimize it for real trading because the margin-call event that will occur in the future would be totally different from the margin call event that was occurred in the backtest.
...
So in your situation, my suggestion would be that you should optimize it for only 2-3 parameters at max by keeping all the other parameters constant.
Do a rollover walk-forward optimization(instead of static walk-forward) and perform a Monte-Carlo simulation on the out-sample result. If your EA passes both of them, You can go live with some confidence.
As you said, if I were using a "traditional" strategy with fixed TP and SL, it would a lot easier to predict future behavior. I actually have a traditional EA running on live ccount, it is a "trend tracker". The problem is that while it is profitable on the long run, it can hit a sequence of losing trades for weeks. So I am trying a new, different approach.
I'll read about rollover walk-forward and Monter-Carlo simulation and try to put my strategy to test. Thanks!
My opinion:
Backtesting as well as optimization is a complete waste of time.
It does not generate any profits.But how would you know if a strategy is profitable? Try it on live without any test?
For this strategy particularly, it absolutely has to be corrected from the inside.
You optimize for 15 years, you suppose markets are cyclics but they aren't really, at the 16th year aka the next big change on market (brexit) it's the one which will make your ea drawdown 46%. What is 46% ? Half the balance.
You have more chance of survive it with a MACDSample optimized on 3 years than with this one.
This strategy is super frustrating, I am the first to say it : have a break.
When you say "This strategy" you are talking about simple hedge/sure fire strategies, right? If you remember our conversation from the other topic, my strategy is something different... not simple as adding lots blindly on a fixed rate.
Hi! All martingales fail (sadly), that is a mathematical fact, there is NO way to escape it really, and what is worst... If you have an edge you are wasting it completely on this as you cannot compound your gains without skyrocketing your risk even further. Hedging on the other hand may cause a very painful margin call sooner or later, although it can help you control "when" you take your losses (some manual traders prefer this than the traditional SL).
About optimization, it is unavoidable really. People often fool themselves when they think they are not "optimizing" their strategies but still cherry pick, ignore or play darts with the parameters, so you are much better knowing what is good unless you are using a "parameterless" strategy (ie: fractals, price action, etc..). To avoid overfitting:
- Use a long period for backtesting (10 years is my personal favorite), but don't try to go back a LOT in time for the sake of it, just try to catch every kind of market for a particular symbol.
- The more trades the better for statistical significance, but also consider the average winner size as these strategies (low winner/loser size) are affected a lot by commisions and slippage.
- Use "simple" strategies, the less parameters the better.
- Keep in mind that hard stops and "trailing" add more layers of complexity, if you don't need them don't use them.
- Do not optimize optimized results.
Hi Arturo, I agree that all martingales do fail!
I say I use martingale because I add lots on losing trades, but it is not as simple as that. If my first trade fails (which is not random, as martingale would suggest), my EA analyses the market, try to understand if it is trending or ranging and based on this information, will open a new trade (based on finobacci sequence) either to try hedge or average with the first trade.
If the second trade fails, it will try again (following fibonacci sequence to add lots). Also, there is a feature that each trade is independent of each other, I mean, each single trade (or group of trades) can close itself in combination with any other trade (or group of trades) to keep the number of opened trades as low as possible. This would unbalance a simple hedging system, but because how my system works, it doesn't matter how many buys/sells are open.
So while I agree that a simple martingale or hedge or averaging are all prone to fail at some point, when I take a piece of each strategy, modify it and make it work together with custom rules, the results can be very different.
About optimization:
Thank you very much for all advice! You are the second person to tell me to lower the number of parameters to optimize. I'll keep that in mid!
- Free trading apps
- Over 8,000 signals for copying
- Economic news for exploring financial markets
You agree to website policy and terms of use
Hi,
I've been working on a EA for some months and got some very interesting results without optimization. Below is a graph using fixed 0.01 lots from 2012 to 2017 on EURUSD:
The problem is that in 2012 there was a situation where my account could have easily reached a margin call (I mean, it reached 43% drawdown, but I don't like anything above 20%).
As you can imagine, this type of graph behavior (with DD spikes) means that my EA uses several strategies that a lot of people don't like (hedge, averaging, martingale). But I think that if well made, these are powerfull tools to make money - as you can see from 2013 to 2017 there were almost no DD.
My question is: I was thinking of running an optimization for 15 years of data, for example 2003-2018.
You will propably say that I will only be curve fitting my EA and you may be right. But on the other hand, the EA will be optimized for how this particular forex pair behaves, right?
How long it stays trending/ranging, the average size of trends, ranging periods and sizes, etc. And because I use renko charts there are less possible mutations on market behavior (there are no candle patterns, no spikes, etc, only waves up and down, all with the same candle sizes).
So my thinking is: With enough information, my EA will know what are the best parameters to use based on how this forex pair has performed over the past 15 years. After all, how much and how fast does the market behavior change?...
I don't know if I am being too opttimistic, that's why I am asking your opinion.
Thanks!
PS - If you want to discuss further about the EA itself (it's strategy), let me know.