Optimization results are not matching with results from a single test run
...
Has experienced the same discrepancies in the trading results and found out a reason for this?
If you need further details for an educated guess, let me know.
Without the code to reproduce it, it's just a (pseudo) random guess.
- Is it also happening if you choose USD as account currency in the tester ?
- And if you don't use a template ?
- What if you delete the cache and run the single backtest again ?
- Are you using function like GetTickCount() or GetMicrosecondCount() ?
Hi Alain,
thanks for looking into this! Interestingly, rerunning those 3456 simulations as often as I want, only the highest profit remains the same (51706.31), while others randomly differ. In a second run there were no more losses shown in the list, on a 3rd rerun they re-occur. So, yes, I agree, mate. There must be some randomness in the whole process.
The trading system I am honing now for several months, was working reliably, up to the point where I must have broken something by re-factoring or restructuring the architecture since the overall performance was not satisfying. However, I did not always "double-click" the optimization results. So at some point it must have been added some random aspect in my code which I am not aware of. Darn, I'd really love to have regression tests to Strategy Testing.
The latest optimization results (with the same 3456 parameter combinations) versus the 2 previous ones:
- I cleared the cache before double clicking a single result, and their profits and # of total trades still differ.
- Removing/Disabling the template (MarginCallCalculator.tpl) did not fix the behaviour.
- Rerunning with USD (instead EUR) results in a similar erroneous behaviour:
The high profits of over 50k are not generated, the best profit results are somewhat around 3.6k and are correct on a double click.
The lossy results are incorrect: double-clicking them results in all being profitable runs.
Heck, what is that??
I thought double-clicking a single result (from above screenshot with the selected line) updates the corresponding parameters in the 'Inputs' tab,
but this is obviously not happening:
Regarding your last question if I am using GetTickCount or GetMicrosecondCount():
Yes, I use GetMicrosecondCount() a couple of times, but they are always guarded by an input parameter which is set to false during testing:
ulong timeOfStart = 0; if (LogPerformance) timeOfStart = GetMicrosecondCount(); // ... if (LogPerformance && !m_StratTesterRunning) { comment = StringFormat("Finished computing buffers. Calculation time: %s ms", DoubleToString((GetMicrosecondCount() - timeOfStart) / 1000.0, 2)); Print(comment); }
---
Maybe you or someone else has an idea what may cause that input parameters are not taken over properly.
Thinking back of what I recently did: My last big overhaul was reducing a lot of input parameters, and since I am working with a partner who was confused by a lot of parameters that I intented to keep as "general trading settings for kind-of any situation* - I decided to make a trade-off: No more showing them as input parameters, but keeping them globally with defaults settings - and when I need them for another EA I would not have lost them ;).
Here's my input parameter list of my TradeManager.mqh which is dealing with the overall trading aspects:
sinput group "TradeManager - Volume:" // ############################################################################################# input double LotSize = 1; // Fix lot size when opening an initial position. input bool ActivateRiskToBalanceRatioForVol = false; // Activate 'risk to equity' ratio to adjust volume dynamically? input double RiskToBalanceRatioForVol = 0.005; // (If above true) use this risked/equity ratio to pick volume dynamically input bool ActivateTPIfRewardReachedMultipleRiskDistance = false; // Activate Take Profit (uses multiple of risk distance)? input double MultipleOfRiskDistanceForTP = 1.0; // Multiple of risk dist. for TP sinput group "TradeManager - Leverages:" // ############################################################################################# sinput double MaxAllowedLeverageByESMA = 30.0; // MaxAllowedLeverageByESMA: Fixed value of 30 for private investors. sinput double LeverageOfThisSymbol = 20.0; // LeverageOfThisSymbol: For S&P500 it's 1/20, seek broker's info sinput double MarginCallOutLevel = 0.5; // Margin Call Level by your broker (e.g. 50%) sinput string DefaultSymbolForCurrencyConversion = "EURUSD"; // Default symbol to convert symbol currency to account currency input group "TradeManager - Stop Loss Settings:" // ############################################################################################# input PlacementModeOfSLOnNewPos SLPlacementOnNewPosition = NO_SL; // Placement of initial SL after opened position input TrailingModeOnUpdateSL SLTrailingMode = MANUAL; // Trailing mode of SL input TriggerDistanceToUpdateSL InpTriggerDistanceToUpdateSL = DISTANCE_2_X_FIX; // Min. Distance to trigger SL update input SLUpdateMode InpSLUpdateMode = UPDATE_ON_NEW_BAR; // Timing for updating an SL input bool AutoStopLossOnAutoRepurchase = true; // After AUTO repurchase in profit: Auto Place SL betw. CP & OP? input bool KeepInitialSLInLossUntilNewUpdateIsInProfit = true; // Keep initial SL in loss until SL update in profit? input bool NeverLowerSLInProfits = true; // If in profits: Never lower SL? input bool NeverLowerSLInLosses = true; // If in losses: Never lower SL? input double FixSLDistanceInPercentOfPrice = 0.005; // Fix SL distance (in fraction of price!) input double RelSLDistanceBetweenCurrentAndOpenPrice = 0.35; // Rel. SL distance betw. curr. & open price (0.0 - 1.0) input group "TradeManager - Auto Re/Purchase Mechanism:" // ############################################################################ input bool AutoRepurchaseInProfit = false; // Auto repurchase in profit position? input bool AutoRepurchaseInLosses = false; // Auto repurchase in loss position? //input double MinimumProfitBalanceRatioForRepurchase = 0.0005; // Minimum (profit/balance) ratio for auto repurchase (0.0 - 1.0) input double CPToOPDistanceForRepurchaseInPercentOfPrice = 0.03; // Auto Repurchase if |Current Price - Open Price| > x% of price input bool WhenSLInLossZoneAfterRepurchaseMoveSLAboveFlat = true; // On repurchase & SL in loss zone: Move above flat? input RepurchasingMode RepurchaseMode = CONSTANT_VOLUME; // Repurchase mode //input group "TradeManager - Auto-Repurchase after Stop Loss:" // ############################################################################################# /*input*/ bool AutoReOpenAfterStopLoss = false; // After stop loss: Auto reopen? /*input*/ ENUM_ORDER_TYPE OrderTypeOnReOpenAfterStopLoss = ORDER_TYPE_BUY; // After stop loss: Order type for auto reopen //input group "TradeManager - Auto Purchase if Price enters Price Range:" // ############################################################################ /*input*/ bool IfPriceInPriceRangeAutoInitialPurchase = false; // If price in price range: Auto purchase? /*input*/ ENUM_ORDER_TYPE IfPriceInPriceRangeOrderTypeOnInitialTrade = ORDER_TYPE_BUY; // If price in price range: Order type /*input*/ double InitialPurchaseIfPriceIsAboveOrEqual = DBL_MAX; // If price in price range: Initial order if price >= x /*input*/ double InitialPurchaseIfPriceIsBelowOrEqual = DBL_MIN; // If price in price range: Initial order if price <= x input group "TradeManager - Prevent Loss of Profit:" // ############################################################################ input int MaxNumberOfBarsBeforeAutoStopLoss = 20; // # of Bars before auto SL (uses rel. SL distance), -1 for off! input double ProfitBalanceRatioForProfitLossPrevention = 0.05; // profit/balance ratio for profit loss prevention (0.0 - 1.0) input double ProfitLossToTriggerStopLoss = 0.7; // Amount of lost profit (in %) so that an SL is placed (0.0 - 1.0) input double InpSLLevelForProfitLossPrevention = 0.7; // SL level to prevent big profit loss (0.0 - 1.0) input group "TradeManager - Max Trades per day:" // ############################################################################################# input bool ActivateMaxTradesPerDay = true; // Activate max trades per day restriction? input int MaxTradesInLossPerDay = 1; // Max Trades in loss per day - Recommended: 1 (-1 for endless) input int MaxTradesInProfitPerDay = -1; // Max Trades in profit per day (-1 for endless) sinput group "TradeManager - Sounds:" // ############################################################################ sinput bool PlaySoundWhenPositionBecomesProfitable = true; // Play Sound when position becomes profitable? sinput double PlaySoundOnProfitGainingXPercentOfBalance = 0.5; // Play sound when profit gains x% of balance
I would assume these global variables (with /*input*/ ) pose no problem to MetaTrader.
You need to determine if the problem comes from your code or from MT5. I would suggest you to check with a "standard" EA like Moving Average or MACD provided with MT5.
Instead of double-clicking if you take the corresponding settings and set them manually, the backtest match the optimization result ?
I have seen a lot of topic like yours, most of the time it was a problem with the code, not MT5. But the latter happened too.
A standard EA like the MACD proved to work properly, i.e. single backtests always matched what was shown in the table of optimization results.
Taking input parameters from my EA manually over from the optimization table to the inputs tab, still does not lead to the displayed result from the optimizations.
First I thought the error "MQL5 start time changed to provide data" might be the cause for the mismatches. However, during tests last night I saw that they also occur when I let the optimizations start at that date where MT5 would change it to.
So, yes, the error must probably be in the code and not done by MT5. Gotta think about how to disable parts of my trading system in order to see when it happens..
Could this confusion come from the fact that my EA uses a trendline indicator that computes every tick and running the full optimization cannot be run with real ticks?
The journal lists an error (or warning): "2020.06.19 12:37:34.267 Tester real ticks optimization not allowed in Cloud Network"
Could this confusion come from the fact that my EA uses a trendline indicator that computes every tick and running the full optimization cannot be run with real ticks?
The journal lists an error (or warning): "2020.06.19 12:37:34.267 Tester real ticks optimization not allowed in Cloud Network"
Sorry, missed checking back here. Well, I use my LAN with up to 3 computers to do the optimization.
However, the errors occur both when optimizing on a single machine and in the LAN cloud.
Spending tons of time on this, is incredibly frustrating.
I can't rely on the results for 100% (often the results are way off) and codewise I haven't done anything spectacular or uncommon. Simple buy and sell positions in an EA that is getting all signals from an indicator that works fine. No erroneous behaviour during a debugging session - all data is coming through perfectly and on the point.
Maybe I have coded something I'm not aware of that has such an effect on optimization runs, but on the other hand I'm extremely thorough in my coding (extreme clean code, modular approach whenever possible, no external libraries used, stress tests on each utility class, indicator, connector and the EA itself).
Looks like I need to hire someone to show my complete code for an analysis - which to be honest, reveals my whole trading process that I have been working on with a partner for years... Feel like I'm kinda trapped here and close to regret I chose MetaQuotes for a huge trading project.. *sigh
Just found something odd:
if I rerun a backtest with the exact same parameters over and over again (here with visualization on),
there's a good chance (I'd say 10%-20%) that something goes really wrong and then instead of being the nice good trade bot...
... it suddenly becomes the weird tradebot, that has super tight SL and TPs:
Interestingly enough, after I have recompiled the EA and the required Indicator, it seems to have stopped.
Even after restarting a backtest for like 30 times in a row, the weird behavior has stopped.
However, I made a copy of the produced Journal logs, so in case you are interested to see the differences:
So here's the begin of the logs of both runs:
A) Erroneous run:
login (build 2485) account info found with currency EUR template file MarginCallCalculator.tpl added. 7286 bytes loaded expert file added: Experts\Advisors\MarginCallCalculator\MarginCallCalculator.ex5. 1814890 bytes loaded program file added: Indicators\Trendline\TrendlineIndicatorPlus.ex5. 361755 bytes loaded program file added: Indicators\Examples\RSI.ex5. 21825 bytes loaded program file added: Indicators\Examples\ATR.ex5. 20063 bytes loaded initial deposit 10000.00 EUR, leverage 1:30 successfully initialized 2155 Kb of total initialization data received AMD Ryzen 9 3900X 12-Core Processor, 32696 MB Usa500: symbol to be synchronized Usa500: symbol synchronized already, 18 bytes received Usa500: history synchronization started Usa500: load 27 bytes of history data to synchronize in 0:00:00.000 Usa500: history synchronized from 2017.04.03 to 2020.07.02 Usa500,Daily: history cache allocated for 848 bars and contains 101 bars from 2017.04.03 00:00 to 2017.08.21 00:00 Usa500,Daily: history begins from 2017.04.03 00:00 Usa500,Daily (ActivTrades-Server): 1 minutes OHLC ticks generating Usa500,Daily: testing of Experts\Advisors\MarginCallCalculator\MarginCallCalculator.ex5 from 2017.08.22 00:00 to 2020.07.03 00:00 started with inputs: ...
B) Working run:
login (build 2485) template file MarginCallCalculator.tpl added. 7286 bytes loaded 1482 bytes of tester parameters loaded 85692 bytes of input parameters loaded 5440 bytes of symbols list loaded (853 symbols) expert file added: Experts\Advisors\MarginCallCalculator\MarginCallCalculator.ex5. 2259800 bytes loaded program file added: Indicators\Trendline\TrendlineIndicatorPlus.ex5. 362195 bytes loaded program file added: Indicators\Examples\RSI.ex5. 21825 bytes loaded program file added: Indicators\Examples\ATR.ex5. 20063 bytes loaded 17309 Mb available, 216 blocks set for ticks generating initial deposit 10000.00 EUR, leverage 1:30 successfully initialized 2607 Kb of total initialization data received AMD Ryzen 9 3900X 12-Core Processor, 32696 MB Usa500: symbol to be synchronized Usa500: symbol synchronized, 4040 bytes of symbol info received Usa500: history synchronization started Usa500: load 27 bytes of history data to synchronize in 0:00:00.000 Usa500: history synchronized from 2017.04.03 to 2020.07.02 Usa500: start time changed to 2017.08.22 00:00 to provide data at beginning Usa500,Daily: history cache allocated for 848 bars and contains 101 bars from 2017.04.03 00:00 to 2017.08.21 00:00 Usa500,Daily: history begins from 2017.04.03 00:00 Usa500,Daily (ActivTrades-Server): 1 minutes OHLC ticks generating Usa500,Daily: testing of Experts\Advisors\MarginCallCalculator\MarginCallCalculator.ex5 from 2017.04.06 00:00 to 2020.07.03 00:00 started with inputs: ...
The input parameters are the exact same, as you can find it in the attached log files.
Another interesting difference, that is logged later on, is the following:
A) Erroneous run logs 'Execution mode: ENUM_SYMBOL_TRADE_EXECUTION::14538240'
B) In the working run it logs: Execution mode: SYMBOL_TRADE_EXECUTION_REQUEST
So, I'm really curious why there are such differences with the account info and the execution mode.
Any ideas are again very welcome. And also, thank you for your time and help. Really, it'd be much appreciated!
Update: I noticed a symbol (EURUSD) that my EA uses for currency conversion is not synchronizing in time when the EA is already past its initialization routine.
I added some proper history loading calls and additional checks found here: https://www.mql5.com/en/forum/265712
"Closing" this thread as all results are now correct.
- 2018.07.17
- www.mql5.com
You are using the Cloud ? You didn't say that previously, you talked about your computer so I thought it was local optimization.
Hi Alain,
I had a similar issue, but realized that clearing cache finally caused the problem to go away.
Specifically, 2 areas of cache had to be cleared.
1. MetaQuotes\Terminal\<Terminal_ID>\Tester\cache
2. MetaQuotes\Tester\ <Terminal_ID>
Would you be able to point me to an article on how to ensure that the cache do not corrupt? Or articles related to pre-checks that i need to do before an optimization run.
What worries me is that the user would not know when the cache is corrupt. And we might continue on our optimization doing the wrong thing.
I would like to go about doing optimization without worrying if its corrupt.
If the procedure is to always delete cache before doing a run, i suppose i have to accept it, but it kind of defeats the purpose for it to be called "cache".
With local at least we can control.
If we are using cloud, how would we know the results are accurate because we will not know if the cache used may be corrupt as well. Not very good for paying customers rite?
Regards
- Free trading apps
- Over 8,000 signals for copying
- Economic news for exploring financial markets
You agree to website policy and terms of use
Hey guys,
My recent optimization results are not what they seem to be. When double-clicking an entry in that table, both backtest and a visualized strategy run yield a totally different result (results of backtest and single visualization test match).
Not only the profit is mismatching, but also the total trades differ (as you see below). So, something is really off here and I don't know why.
My attempts to fix this error so far:
- Thought my hardware with an AMD Ryzen 9300X with its 12x2 hyper-threading cores might be the reason, but I can reproduce this error on a machine with a non-HT CPU (like Intel i7).
- Static variables in my code that could potentially be read by other threads? (Can this actually happen?) -> Anyway, I replaced them all with member variables -> still didn't fix it.
- I'm making use of OnTester events to capture data frames so after all strategies have run the data frames are read and their data placed into a database. The erroneous results are actually the same in the database, so I conclude the number in the 'optimization result' table are not just random numbers, but actual results from properly done simulations, but something must have happened there vastly different.
Optimization Results:
Result from a visualization run:
Has experienced the same discrepancies in the trading results and found out a reason for this?
If you need further details for an educated guess, let me know.