Optimisation results differ from single tests on them
Here's another example. I have just started reviewing overnight optimization for UsdJpy. Right after the optimization without changing quotes and without touching them I have double-clicked selected parameter values from result 2631 (profit 117.07 and drawdown 34.86) into tester and have run a single test using these parameters. I got a profit of -3925 and a drawdown of -4049.04. Noticeable difference...
The report says about 51 chart mismatches. There are 51 entries in the log about it. But all entries refer only to volumes, and I don't use them anywhere, not even in indicators (I don't use indicators).
The optimization and testing period is the same as in my first post. Sequence of actions - the same, but with the addition of the control tester run after p.3 to check the absence of errors mismatch. All other conditions are the same. The optimization was performed at night from Saturday to Sunday - the market, and therefore the spread should be frozen.
I was surprised again. I opened the "Quote Archive", pressed Upload on this pair and allowed recalculating all timeframes.
I again double click to insert parameters into the tester from the optimization results and start testing. The results are now the same as in the optimization. There are no errors of mismatch in the report and log.
I am adding questions (I keep numbering them):
4. Where did the discrepancies appear after the optimization (and maybe during it) if before the optimization I purposely did one test run and made sure that after loading the pair in "Quotes Archive" (between point 3 and point 4 in my previous message) there were no discrepancies?
5. How can I make sure that all optimization results are obtained on correct quotes and that there was no glitch in the optimization process?
6. How do volume mismatches have such a critical impact on the results of tests that don't use volumes?
Z.I. I have antivirus installed and working. But both the Metatrader executable and its whole folder with all subfolders and files are in exceptions. My other computer has no antivirus at all, but such glitches are present. It means we exclude the antivirus. Same glitch on three different computers. We exclude disk and hardware problems and possible influence of surrounding programs. Something is wrong in Metatrader, it does something wrong with quotes.
I have seen the same glitches with optimization in other, fundamentally different strategies. So, in essence, it doesn't really depend on my Expert Advisor. It's just that it wasn't that important to me before and I wasn't testing it enough. Now I'm doing dozens of optimizations a week, and I'll do even more. And we are already talking about real money. Please advise me on the correct actions or amend the tester and the optimizer. Thank you!
The archive contains the following files:
UsdJpy_1_optim.txt - optimization results
UsdJpy_1_Orders.txt - list of orders (results) of a single run coinciding with optimization
UsdJpy_1.log - log of this test
UsdJpy_1.htm - report on it
UsdJpy_2_Orders.txt - list of orders (results) of a single run with a huge loss (with mismatches)
UsdJpy_2.log - log of this test
UsdJpy_2.htm - report on it
Now it's even more interesting. This is on the third computer and on a different pair. I do everything according to the scheme from my first post. After several hours of optimization, I check the results. I paste the selected set of parameters by double clicking (without touching or changing anything else) and get absolutely different results than in the optimization string with the same parameters. I check mismatch of quotes. Although I have downloaded and recalculated quotes in "Archive" before the optimization. Ok, I think, maybe the quote recalculation will help. I launch "Recalculate all timeframes" and test it again. The discrepancies have disappeared. But I have now calculated a third variant of the results that does not coincide in any way with the first two. *smiley_with_great_eyes*
A few more attempts at recalculating the quotes in the Archive and testing. All the results now match the last, third option. Hooray, we've found the right variant! But half a day of work on the optimization was wasted and the optimization results are obviously incorrect. That is, I have to re-install the optimization at night and have time to get up early in the morning to choose suitable parameters for the next week. Besides, you cannot be sure that they are correct.
I've noticed some more details. First, the number of "Bars in History" differs in tester reports for different variants. Second, even with the same amount of bars in the history, the results may be different. Third, my Expert Advisor is working with hourly candlesticks and opens the first orders in the series strictly at a certain hour and 00 minutes, i.e. at an Open candlestick. But these orders open at different prices in different results. This can be seen in the reports I attached in my first posts so I am not writing any more logs for the time being. If you need anything else, I will post it.
Am I the only one who has this problem with the tester?
And more questions on the "Quote Archive", since the topic of their conversion has been touched upon. Can you tell me where to read it in detail or answer my questions here.
I do all actions on Saturday-Sunday, i.e. the quotes archive should not be changed.
I open a pair with a non-opened chart window during the last week, i.e. I understand that the quotes for this week were not downloaded into the local archive. I press Download. The traffic monitor installed on this computer shows MT requests to the server and data blocks returned from the server. Everything is correct. After downloading, I click Download again. But here I have two options: either I keep counting all timeframes (which is logical when downloading is completed completely) or I download something again (and traffic monitor confirms it) - it's not clear why it didn't download at first. And if you press it a few times, the download from the server may be after recalculation, which is not logical and understandable at all. If already downloaded, recalculated, what else is being downloaded? I do not close the archive between downloads, do not change the pair.
Sometimes, it happens, that after opening the archive pressing the Download button a few times in a row (5 or more), it still downloads something from the server and does not suggest recalculating all the timeframes, no matter if the archive with the pair was downloaded before (at the previous archive opening) and if it was recalculated then. It's as if not everything was loaded the first time, and now every time you click, the missing stuff is pulled up bit by bit, but not all at once. You can see in the monitor that it's not just requests to the server, but also blocks of data returned by it. Why and what didn't load the first time? How many times should we really press the Upload button to be sure that all the quotes for a pair are downloaded?
If I close the Archive, open it again and then click Reload, it will not load the quotes the first or second time, but will offer recalculation. The logical work sequence is restored. And why is it happening? How do you understand it? Maybe after each download I should close the archive, open it anew and then do the recalculation? Please, explain.
Can you make it so that when you click to load, the local archives are checked and all the necessary files are loaded in one go, and then all the timeframes are automatically recalculated? And can you build the "Validate" button into the archive, which will show all the gaps in the history, inconsistencies and the like? And then, on the user's confirmation, everything would be automatically pulled from the server and repaired. I would like to use the MT4 version since the migration to the 5 version will go on for a long time, and the 4 version is very popular and in demand. And the money we need to make now, on the 4.
Once again, MT is connected to Alpari demo servers since installation and they have never changed. There are only two folders in the program history folder: Alpari-Demo and downloads. The settings of the number of bars in the history and in the chart window have not been changed and remain default also since the installation of MT on the computer. But the results of pressing the Download button are unexpectedly different. What actually happens in the Archive and why is it so unpredictable?
There is a parallel discussion on this topic on the Alpari forum http://forum.alpari.ru/thread58122.html
Doesn't anyone here have any answers or even questions? And the developers too?
1. You need to remove the chart misalignment errors. Delete all the history of quotes, upload a new one.
2. If the broker has a floating spread, then it is necessary to optimize and test on one spread. Preferably on a typical one.
3. Start overnight or overnight on a demo account of the Expert Advisor and compare the results of the demo and the tester. If the orders approximately coincide, and the number of orders is not significantly different, then the tester can be trusted. If the trade does not correspond to the tester, you should not optimize in the tester.
It would be nice to see at least a tester's report.
For dimeon:
1. Tested on three different computers, three different instances of MT4. The symptoms are the same. After loading quotes in the "Archive" I check it by running the tester - there are no discrepancies for the required period. Then I launch optimization by the same period. And then I run optimization tests. Here is where the discrepancies might occur. That is, they appear after the entire history has been downloaded and recalculated by the Archive, after (or during) optimization. Deleting the history is unlikely to do anything here.
2. Alpari has a floating spread. I agree with you, you need to test it on equal market conditions. But how to test it on a selected spread, where to specify it? I have not found it in MT4.
This is a good idea. I will probably try it. Only I will have to take at least a few days. But it still does not solve the main question: why these glitches with quotes?
The tester reports are in the archives in my posts above. Even for two different pairs on different days. In the same posts there is a description of the files from the archives. Have a look.
Download the history, recalculate the timeframes. Disconnect from the internet and test.
This kind of crap usually indicates that the EA is extremely sharpened. Ticks are not downloaded, they are emulated. And if you open on the market, the slightest change of history leads to this nonsense.
A similar result can be obtained this way:
The computer has been connected to the Internet for a long time and the chart is open on H1.
We switch off the Internet and start testing. The Expert Advisor writes to us that it downloads the history of M5, M1 and this is when the Internet is disconnected. So, there are a lot of pitfalls. We also know what kind of history it downloads when the Internet is connected.
Thanks Mislaid. That is exactly what I will try to do with the next optimizations. I was also told about such an algorithm on Alpari forum today. I will try it... Although, frankly speaking, either I do not understand the program logic or the developers have some strange attitude to the most important part of the optimizer - quotes archive. So I still want to hear the developers clarification of this topic and answers to questions. And what kind of history does MT download from the Internet? I do not know anything about it. Explain, please, or give me a link where to read.
I don't know everything, conclusions from personal experience.
I see that you have already set up enough experiments to be able to formulate hypotheses.
You have uploaded an archive of quotes on minutes. It is downloaded from the MQ server. If you have noticed, it is missing the last few days, or even weeks. Hypothesis: the archive of last quotes on the MQ server may not be formed yet. You download the rest of the history from the server through "Update".
Another experiment. You kill the history. You try to download the minutes (five minutes) through "Update". A small part of history will be downloaded from the server. Hypothesis: only a limited part of the latest history is stored on the DC server.
Hypothesis: the tester downloads М5, М1 from the archive on the computer and completes it from the MQ server. If the archive on the server has not yet been generated, this data is emulated. The same if there is no internet connection. The emulated data will not be saved?
Once a week I download the history for 27 currency pairs using the "Update" function. On some pairs more than 10000 bars are downloaded, although new bars for the week are about 7000. Hypothesis: the history has changed.
The test results may not coincide with the latest data for these reasons.
Testing results and demo (micro, real) trade results may not coincide due to some other reasons.
- Free trading apps
- Over 8,000 signals for copying
- Economic news for exploring financial markets
You agree to website policy and terms of use
I optimise the EA. Then I run a single test based on the results. The results of optimization and single tests do not always coincide.
Sequence of actions:
1. I select a pair (e.g. AudUsd) in the optimizer and set the period 2010.08.23 - 2010.10.30. Expert Advisor in the list and H1 timeframe are already selected - I test only one Expert Advisor now.
2. I enter optimization ranges of parameters.
3. I open the minute history for AudUsd by double-clicking on the Quote Archive. Click Download. After downloading, I open the clock and click Download again. It says, "Will it recalculate all timeframes? I answer yes.
4. I optimize 4 or 5 parameters and wait a few hours. I would like to see the optimization results in "AudUsd_1_optim.txt" attached to the archive.
5. I selected the line suitable for me in "Optimization results" (in my example it is number 2183 in the "AudUsd_1_optim.txt" file) and double-clicked its parameters in the tester. I do not change and do not touch anything else.
6. I start the single test. I look at the tab Report and see a completely different profit and drawdown (201.78/-116.28), not matching the optimization results (162.36/-34.25) for this combination of parameters. There is one error of volumes mismatch in logs, but my strategy does not use volumes in any way, so I think this error is not critical and cannot affect results that much. The number of trades and everything else is also different. The list of orders from this testing is available in the file "AudUsd_2_Orders.txt" and the log can be found in "AudUsd_2.log".
7. I am very surprised. Though I am slowly getting used to it.
8. I repeat step 3.
9. I start single testing again using the same parameters. Now I obtain profit and drawdown that coincide with optimization. The list of orders from this testing is contained in "AudUsd_1_Orders.txt" and the log is in "AudUsd_1.log".
It's been like this for more than two months now (I optimize every week). This glitch is periodic and detected no clear pattern. It doesn't depend on start time of optimization and a couple. Sometimes optimizing and next separate tests' results coincide but sometimes they don't.
Frankly speaking, I'm already tired of it. All the night it is counting and optimizing something. And then I have to reoptimize it again because I cannot understand what results I should trust. Besides, I am not sure that even a coincidence has happened to be a correct and not a glitchy variant of the tester's work.
I want to tell you right away that I have searched and read similar topics on forums. I have searched and read them both on Alpari forum and here. And in Yandex too. I have been trying to find information for a few weeks now. But all I have found is the developers words about "floating spread" upon which the testing results depend. And advices to test strategies on weekends (that's exactly what I do - it doesn't work). Although, to my mind, it looks like nothing more than an excuse for not wanting to somehow solve this issue. After all, in order to close the topic of "floating spread" it is enough to add the "Spread" field into optimizer's parameters filled by users at their discretion, and one can already carry out an adequate optimization and comparison of strategies by parameters. I do not see any sense in memorizing the last market spread for optimization purposes.
I have the latest version of Metatrader downloaded from the Alpari website - 4.00.226. Exactly the same glitch is observed on two other computers, with the same version, but installed at different times a few months apart. The settings and score are the same there as on the first MT.
I only do testing and optimization on one demo account at Alpari. I have not connected any other accounts. The Alpari-Demo server is in the settings - Alpari NZ Limited. I have not touched the server either. I have not changed the quotes manually, I have not added my own. Metatrader is always online (I always have Internet on my computer). I perform optimisation on Saturday and Sunday. Spread, in theory, on those days should not change and the glitch with the "last spread" probably has nothing to do with it. The attached logs are from Saturday, October 30.
The testing period is always stated explicitly.
The questions are:
1. How may it happen? What is the reason?
2. If this is a tester glitch, when will you fix it? And what should I do to make everything match now, in this version?
3. If it's my fault, what am I doing wrong and how should I do it?
I am ready to provide the complete real code of the Expert Advisor (anyway it is only a highly truncated version for optimization) and a set of testing parameters to the developer of Metatrader in his personal message.