75,000 options - 4GB RAM and 4GB disk cache not enough??? - page 7

New comment
 
I ran tests on an IBM from early 1970 to early 2000, the history is correct and normalised to take into account the multiple splits. The zipped symbol history file in HST format is attached, you can import it into the terminal via History Center (F2) - > Import.

Here are the initial results:
  • I took the best results of 401 and ran the test in MT4 with the same parameters





    What I got (full reports in the attached StrategyTester_Mak. zip):



    There are serious differences in trades - this is a separate topic. But net profit $1664 in MT4 is more or less similar to net profit $1884 in Omega.

  • I then launched the enumeration of parameters with genetics in MT4 and obtained the following results:



    Here is the list of results (archived report in OptimizationReport.zip):



    MT4 re-calculated in 3 min 34 sec, retrieving 15,616 variants out of 35 billion possible:



    I should clarify that out of 15616 variants there were 9291 repeating (by crossing you can get repeats) sequences and of the remaining 6325, 1027 passes were discarded as useless. Repetitive sequences do not take up resources and do not require recalculations because they are taken from the temporary cache. Final log entries:

    2006.10.16 23:40:12 There were 6325 passes done during optimization, 1027 results have been discarded as insignificant
    2006.10.16 23:40:12 MACD Sample: optimization stopped, 9291 cache records were used, 9291 cache records rejected


  • why mt4 pulled a profit of 3600 and TSGO is half of 1800 ?

    I think it's a direct consequence of the super economical approach to optimization in TSGO where the emphasis is on reducing passes. It's very simple - TSGO made 1000 passes, while MT4 made a net 6000 (9000 out of 15000 were repetitive and were missed). Besides, I'm very much interested in TSGO's report - optimized MATrendPeriod parameter, where this parameter is fixed at 100 (a few times it slips to 97, but it's nothing serious). We can see how the genetic optimizer has hit a local extremum and trivial failure to climb out of it. This is a usual situation in the genetic enumeration and a direct consequence of the search area overloading.



    Has MT4 been in those regions where such values reside? Yes, I have been many times and have not found anything special there:



    Apart from mathematical results and reports, you always want to see with your own eyes the picture of the distribution of the best results. MetaTrader shows it easily in the "Two-dimensional surface" visualization mode where you can select any parameters to be optimized along the axes. For example, we can see here that MATrendPeriod value in the area of 100 is clearly worse than values of 12-14.



Conclusions:
  • It is necessary to think about local extrema and not to make too rough searches. Especially when you provide the tool for inexperienced users who cannot understand the processes in details and may take the results of the genetic tester at face value.

  • In the background of shown breakdown into local extremum at 1000 passes, the idea of 100-200 evaluation runs can in no way be taken seriously.

  • When we were choosing a sufficient overshooting mechanism for our genetics, we did some research and settled on this simple formula: always do 30 complete population runs, and then for the next 10 complete population runs we wait until the increment of the target function is guaranteed to decay. That is, if the initial population is 256, then the first stage is 256 * 30 = 7680 runs, then at least another 256 * 10 = 2560 refinements and then additionally before fading. This redundant mechanism allows us to get out of local extrema in most cases.

Perhaps we are overkill a bit more than necessary, but for regular users this is more or less a guaranteed way to work.

Thanks to Yury (Mak) for bringing up the subject - it's very interesting to discuss it + we fixed our errors.
Please comment on it.
 
Renat, I haven't found the parameters of your best option.
Bring them up and I'll check in Omega what results you get there.
I suspect that there is a big difference in what is obtained when testing in the two platforms.
In the meantime I'll run a test in Omega for a lot more than 1000 runs.
 

Sorry, I didn't specify the parameters. Here they are:

TakeProfit=819; Lots=0.6; TrailingStop=248; MACDOpenLevel=8; MACDCloseLevel=8; MATrendPeriod=12;

Parameters can be seen in the tooltip next to the pass number in the tester report:

 
Renat, you're jumping to conclusions again.

Did 7000 runs in TSGO.
The picture below.

Note the numbers of the runs that are at the top of the population.
The best for 7000 runs was run number 618 (i.e. after 700 runs there has been no improvement)




I ran your best found parameters in Omega.
It turned out like this :((



Apparently there is a significant difference in the operation of the systems or testers or in the data.
Maybe I didn't transfer the system to Omega correctly ...
You can't transfer one to one here, Omega works differently.

But in any case, your conclusions are that
that TSGO didn't find a twice better solution and got stuck in the local high is wrong.
TSGO and could not find such a system because it is unprofitable in my test with Omega.
 
Yes, I have noticed that you have positions opening where they shouldn't. Check manually the first five trades - you'll notice right away.
I got IBM from yahoo, I had to split the stock 4 or 5 times myself to get the normalized data. It matched your story completely.

Apparently you have to start again.
 
Renat:
Yes, I have noticed that you have positions opening where they shouldn't. Check manually the first five trades - you'll notice right away.
I got IBM from yahoo, I had to split the stock 4 or 5 times myself to get the normalized data. It matched your story completely.

Apparently you have to start again.
It seems to me that the mismatch of results is not fundamental.
Different testers and generally different systems ...

Any genetic algorithm is a random search.
Genetics differs from just random search in that
that the parameter distribution function is uniform in a simple random search
in genetics is non-uniform and varies as you search.

This greatly increases the search speed in genetics compared to a simple random search.
In this sense all genetic optimisers are the same.
 
Hello, dear friends!

I apologise for the off-topic, but the incomprehensible terms give me a severe yawn and a headache. Could you, Renat, explain in a nutshell what a horse in a spherical vacuum is and what it is put there for?

The subject is very interesting for me, because I myself have recently started to use GO to optimize my own Expert Advisor and I want to understand the limitations of GO application. Only I don't have exactly MACD, but two muwings based on Zero Lag MA and with two hysteresis ("backlashes" according to the article "Genetic Algorithms in MetaTrader 4. Comparison with the optimizer's direct overshoot') on entering and exiting a position. I am not very satisfied with the results so far, but I have some hope. I think I will soon add new filters to my Expert Advisor. Well, I like the optimizer so far.
 
Mathemat:

... what is a horse in a spherical vacuum and why was he put there?

Well, actually, Renat was a bit exaggerated here. Of course, a horse or a horse is not so basic. As for the spherical vacuum, everything is simple - a sphere of sufficiently large diameter (and sufficient strength), a horse is placed there and the air is pumped out. The vacuum inside the sphere takes a spherical shape - hence the term spherical vacuum. Of course, it is necessary to take into account the measurement error, because the vacuum does not really turn out to be absolute and the sphere is not perfectly spherical-smooth, but in principle, the error does not greatly affect the measurement results.
Yes, we should not forget that the horse/horse naturally dies as a result of the experiment. Alas nothing is perfect in this world. :)

P.S. Well, the horse is naturally planted for the sake of the experiment. The purpose of the experiment is not disclosed, so to speak, know-how. :)
 
Strange, I thought of it as a spherical horse in a vacuum. I mean, it's round as it is, but in a vacuum it's not realistic to see where it is at all :)
 

"A spherical horse in a vacuum" is an idiom that originated on the FIDO network in the 1990s. It denotes the elusive ideal conditions of an experiment or operation.

There was an old anecdote about a physicist being commissioned to develop a model of horse racing so that he could predict the winning of this or that horse.

Did you know that...

  • A spherical horse has an absolutely black body.
  • A spherical horse breathes a perfect gas.
  • The rumbling of a spherical horse is a single harmonic and propagates without dissipation.
  • A spherical horse grazes on homogeneous fields.
  • When a spherical horse gallops, its trajectory is described by a cycloid.
  • The hooves of a spherical horse collide with a flat horizontal surface in an absolutely elastic way.
  • The spherical horse with probability one will escape from the potential pit in a finite time.
  • To get on a spherical horse on horseback, one has to find the saddle point. The resistance of the horse in this case is negligible.
  • When viewed from an infinitely distant podium, the spherical horse appears as a material point.
  • The problem of combing the hair of a spherical horse is unsolvable.
  • A spherical horse moves in a vacuum uniformly and rectilinearly.
  • The spherical horse does not need a spherical horse because it exists in speculative space indefinitely.
  • The projection of a unit spherical horse onto n-dimensional space is described by the equation r2 = 1.
  • Horsepower - The force exerted by a spherical horse with a diameter of 1 metre and a mass of 1 kg.
    • 12345678
    New comment