Optimisation and Out-of-Sample Testing. - page 3
You are missing trading opportunities:
- Free trading apps
- Over 8,000 signals for copying
- Economic news for exploring financial markets
Registration
Log in
You agree to website policy and terms of use
If you do not have an account, please register
I think this would be a useful addition to the mt4 tester. It is possible, and likely to be implemented somewhere by someone else.
You are right - it has been implemented. Look for example here (MT4NetTester project). Just don't ask to give it, it's bespoke.
Guys, I've had it all working for a while now.
True under TradeStation, and not for free ... :))
I don't see the point of doing it under MT, we're not used to paying for work.
I have almost finished too))) And there is no need to embed anything in the Expert Advisor - the program is downloaded a file with a set of parameters
Guys! And who prevents you from writing an article in this thread, post its development with a description and get a fee?
I'm not looking for a freebie. I have enough diligence. Not enough special knowledge. What difference does it make - who will pay you?
Good afternoon everyone.
After optimizing an EA, we often have to nerd out-of-sample more than a dozen sets of parameters suggested by the optimizer.
I have an idea of optimizing Expert Advisors outside the sample. Suppose we "charged" the Expert Advisor with optimization by a number of parameters. For example, from 1 Jan. 2006 till January 1, 2007.
We have received several thousand Expert Advisors. After that, we save the page with the OPTIMIZATION RESULTS as a separate file. Next, we set the following history period for optimization, i.e. we add a month or two, or as many as we need.
In our case, we set for example from 1 Jan. 2007 to June 1, 2007. And again we enable optimization. The optimizer should not take parameters in EXPERT'S PROPERTIES, but re-select them one by one from the file we have saved after the first optimization. After this second optimization, we are left with only those vAriens that have yielded profits outside the sample!
The result, ideally, is that we get the "ideal parameters" to work with and test online later on!
I think this will be a useful addition to the mt4 tester. Probably, and most likely, it is already implemented by someone somewhere. If anyone knows, please share the link!
I, due to my modest knowledge, cannot figure out how to go about implementing the idea in practice.
It is an old and popular topic in the west.
OK, the general idea is clear. Well, then the last question to all those who have implemented this project (i.e. Belford, Mak, Integer): is it worth it? Of course, it is nice to have an "optimizer" which does not only curve fit (like metaquote) but also tries to test strategy on out-of-sample data, but does it really deserve a higher score than MQ optimizer (which is also good, but only as a curve fitter)?
We select the best ones in the first series from among the best ones in the second one and so on. - this is also optimization with its own fitting.
My optimizer is just a fairy tale :)))
First, it is genetic and very efficient - it searches very quickly.
Secondly it allows you to use any custom optimization criteria (this is a great power).
Thirdly, the new version implements optimization by multiple criteria at once.
I posted an example of implementation of my optimization criterion in the topic'init() during optimization'. And I'm trying to write an article about it. The method used, I think, allows to do a lot of things, from WFO to own genetic algorithm.
leonid553, you are going in the right direction, as the existing "optimization" without testing on out-of-sample data is pure curve fit. But the basic algorithm should be more complex, like in neural network programs. All "optimization" should go simultaneously on all datasets at once (this is the wish of developers). You can, of course, work with only two datasets, but it is better to provide for three - training (A), validating (B) and testing (C). Well, under existing conditions, you will have to work more or less as you suggest.
It seems to me that testing (training, validating, running, etc.) outside the sample reduces the set A="Parameter sets profitable in the sample" to the set B="Parameter sets profitable in the sample, as well as outside the sample". Optimization over sampling + out-of-sample gives set C="Parameter sets profitable in sampling + out-of-sample".
Is set B contained in set C? In my opinion, obviously yes, because set B contains "Parameter sets profitable both in-sample and out-of-sample", and hence they are "Parameter sets profitable in-sample + out-of-sample".
So what is redundant in C compared to B?
Obviously, among C set there will be parameter sets that are unprofitable in the sample (only in the sample), but that gain revenge outside the sample. That is, we lose money in the sample, but more than pay off outside the sample.
Also, among the set C there will be sets of parameters profitable in the sample (only in the sample), but unprofitable out of the sample, i.e. we make money in the sample, but lose money out of the sample.
So a pass-through gives us set C, which collects subset D="Parameter sets unprofitable either in sample or out of sample, but generally profitable in sample + out of sample", and which is not in set B. If this is a problem for us, it is much easier to solve than out-of-sample testing.
But do we avoid curve fitting by out-of-sample testing? No. Moreover, it can be argued that the most severe (profitability must be respected in individual plots) fitting remains, and only subset D (maybe the child sought?) is thrown out, which can hardly be called a fitting at all. All we avoid by dividing the testing period into segments is the exclusion of parameters giving unprofitable results in one of the segments. All we reach is profitability in every segment. The logical conclusion of this approach is profitability of each deal separately. Who only will guarantee that such marvellous parameters are not presented to us by the devil of absolute curve fitting?
An EA is not the same as an EA! There are EAs for which the testing I've described is the most optimistic.
What, for example, do we have to lose here? Nothing!
I would formulate it this way: we mark some vicinities of certain extrema by the first optimization and check their stability by the second one (out-of-sample). No one will of course give any guarantee that they will remain stable in the future. But lack of performance guarantees is a characteristic feature of all forecasts.
Of course we can get the same information having processed results of optimization for the whole sample. It seems to me, however, it will be much more time-consuming process, both in terms of software implementation and in terms of counting time.
I would formulate it this way: we mark some vicinities of certain extrema by the first optimization and check their stability by the second one (out-of-sample). No one, of course, will give any guarantee that they will remain stable in the future. But lack of performance guarantees is a characteristic feature of all forecasts.
--> What does splitting adds to it, compared to the end-to-end run?
Of course we can get the same information by processing the optimization results on the whole sample. It seems to me, however, that this would be a much more time-consuming process, both in terms of software implementation and counting time.
--> If by "by processing optimization results on the whole sample" you mean discarding set D, then I disagree - discarding results that give intermediate losses (in sample or out of sample) is an elementary task, solved during optimization itself on the whole sample, i.e. no processing after optimization is needed. The result is set B immediately. And no time consuming additional checks outside sample.