This post is a part of documentation of WalkForwardOptimizer library for MetaTrader4/5 - here is the table of contents.
Frequently asked questions
Q. Why should I use WFO?
A. WFO is an automated implementation of the process of EA testing and monitoring, which a trader does normally perform by hands. In addition, WFO provides a justified answer for the questions, which period of EA optimization to choose, and how long the optimized EA performance will last in average, so that one could know at which pace to perform re-optimization.
Q. Is WFO library available for MetaTrader 5?
A. Yes, of course. The library for MetaTrader 5 affords some advantages: manual operations (such global variables removal or file copying) are eliminated, work parameters of EA are automatically captured from the tester with their names and values (in MetaTrader 4 you need to pass this information explicitly in your code), optimization on mutiple cores, including local network and MQL5 cloud is supported. This makes this version more convenient and efficient. It's still unknown if MetaTrader 5 will support walk-forward optimization natively as a new mode of the built-in tester.
Q. How to install the library and start using it?
A. You may find detailed instructions on How to Purchase a MQL-Application in MetaTrader 5 online documentation or MetaTrader 4 online documentation. Rental of MQL-programs is very similar.
After the purchase you'll normally see the Install button in the Market window of your active MetaTrader instance. Clicking it should start and finish download of the library in a blink of the eye. If some errors happen you'll see them in the logs.
All MQL-programs are downloaded to MQL5 folder's subfolder which corresponds to the program type, and its nested subfolder Market. For example, indicators are placed in MQL5/Indicators/Market/, expert advisers - in MQL5/Experts/Market/ etc. Unfortunately, for libraries this may or may not work.
In some cases MQL Market can treat libraries as scripts (this is how the platform works, we can do nothing), which means that they are landed into MQL5/Scripts/Market/ folder. You should manually copy the file to MQL5/Libraries/ folder. By the way, you can copy the library to all instances of MT5 on the same computer freely (virtual PC are counted as separate computers).
The library may have a slightly altered filename than standard "WalkForwardOptimizer.ex5", for example, "WalkForwardOptimizer MT5.ex5". In this case, please, rename it to the standard "WalkForwardOptimizer.ex5". This name is referenced in the header file WalkForwardOptimizer.mqh, in the #import directive.
Optionally, you can have different versions of the library with different filenames (for example, for backup/test/rollback purposes) and use them in your source codes provided that the #import directive is edited accordingly.
Please, make sure you're using the latest header file WalkForwardOptimizer.mqh - it's published in the Comments section of the product page.
Q. Why WF optimization and testing periods are performed in a single tester pass?
A. A single tester pass, divided into optimization window and forward test, is used because it makes it simple to associate corresponding optimization and testing periods. In fact, MQL does not provide any means for linking 2 independent tester passes with each other. Moreover, even if it was possible, testing forward steps would be treated as optimization runs, and thus introduce a bias on optimal parameters search. And if we would mark forward tests with zero performance indicator to eliminate the bias, the tester would skip most of forward tests while using genetic algorithm. And refusal to use genetics seems impractical.
Q. Why standard optimization restrictions must be disabled in the tester while using WFO?
A. Indeed, all flags on the Optimization tab of the Expert Properties dialog should be cleared off before using WFO library. This is so because the restrictions are applied on corresponding values calculated for entire period, but the period is internally divided by WFO on optimization and forward testing parts, where only optimization values should matter. WFO library calculates all important parameters separately for both parts, and provides you an access to optimization indicators via special variables, that can be used in custom formulae. If you want to use a restriction, you may specify an expression for performance estimator where the specific parameter is checked against a threshold, and if condition is not met, estimator value set to 0. For example, if you want to drop all passes with relative drawdown larger than 50%, you can multiply your main performance estimator to the following expression "min(max(50 - DDREL, 0), 1)".
Q. Some values of profit factor in the tables seem strange. Why?
A. Because profit factor is calculated as a ratio of profits and losses, there can be cases where it's actually undefined - this happens if there are no losses at all. By default, MetaTrader 4 assigns DBL_MAX value for profit factor in such cases, but this is not a good idea, because a test with 1 profitable trade and another test with 10 profitable trades get the same profit factor. Moreover, if profit factor is used in a compound performance estimator, DBL_MAX will most likely introduce irregularity. This is why, the library provides 2 different ways of handling "overflown" profit factors. When the estimator is wfo_built_in_loose or wfo_built_in_strict, profit factors larger than 10 are adjusted as 10 multiplied on square root of number of trades. When any other estimator is selected, you may use wfo_setPFmax function to specify a relatively big number as profit factor upper limit, but significantly lower than DBL_MAX, for example, 100. Finally, if you choose wfo_expression, you can perform special cases handling by calculating profit factor surrogate from gross profit (GP variable) and gross loss (GL variable) with arbitrary specific formula.
Q. How do I know that a walk-forward test is passed or failed?
A. The WFO library does not explicitly mark walk-forward results as passed or failed, because user's preferred criteria may differ. As rule of thumb, a test is considered passed if efficiency is over 50% and consistency is over 50%, as well as drawdown does not exceed 30% (you may also check estimation of distribution of drawdowns built by Monte Carlo method, see screenshot below). One may also take care that distribution of profits on forward tests does not contain distinct outliers (when largest profit per pass produces more than 50% of total profit). All this factors can be checked visually in the tables of specific report.
Q. Why Walk-Forward Drawdown is marked as "rough estimate, normalized"?
A. This drawdown is calculated on forward balance curve using solely profits or losses on every step (as a whole), and does not take underlying trades into account. This makes the drawdown "rough estimate". In order to calculate an exact drawdown one should have a complete list of trades, but WFO library does not save them for the sake of efficiency. Indeed, if it would do this, every single pass would generate a separate file, and the number of passes can be hundreds of thousands.
The drawdown is called normalized, because it's calculated starting from annualized profit on in-sample data (optimization period) instead of initial deposit. Conventional drawdown may vary dramatically depending from an arbitrary deposit amount chosen by trader voluntarily. This makes it hard to understand the risks. For example, with initial deposit equal to 10000 a drawdown on 1000 is only 10%, but for 2000 it's 50%. When annualized profit is used instead of deposit, drawdown expresses the risks in universal measure scaled accordingly to the trading system's potential efficiency (average profitablility).
Q. What's the meaning of Deviation in the reports, how to interpret it?
A. The table row with Deviation shows standard deviation (σ) of corresponding value as a measure of its dispersion, that is how far it fluctuates up or down across its Average. If deviation is equal or greater than average (or mean), then corresponding parameter seems unreliable, because its observed value will most likely change sign easily regardless to expected one, and this is very important for example for profits. For example, if average profit is 100 and deviation is 100, then probability of a loss is approximately 15.8%, if deviation is 50 - probability of a loss is ~2.2%, and if deviation is 33, then probability of a loss is only ~0.1%. On the contrary, if deviation is 200, then probability of a loss increases up to ~31%. This is illustrated by the graph below.
We use Normal distribution here, that can be debatable, but it's well studied and applicable (under certain assumptions) for most of long running series of experiments (let us assume we have a sufficient number of walk-forward passes).
Q. My optimization has stopped with the error "Tester memory handler: tester stopped because not enough memory". What can I do?
A. You're trying to perform too massive optimization which consumed all resources available for the MetaTrader 4 tester. This is not an issue of the library. This is a limitation of the tester and operating system. You need somehow simplify your optimization settings or add more resources to your PC. Here is some methods:
- shorten the range of dates of testing (for example take 2-3 years instead of 5-6),
- decrease number of parameters enabled for optimization,
- reduce number of iterations for parameters being optimized (for example, you could decrease number of forward steps in the expense of increasing forward step size in the same number of times),
- try to free up MT4 memory by unloading everything except the EA being optimized (note, that MT4 should be restarted after unloading because it caches many things internaly and they left in memory),
- enable genetic algorithm in the tester,
- add phisical memory to you PC .
Q. Why do I get "incorrect input parameters" error?
There can be many reasons for this. You can find out what is the problem in your case by looking into the tester log during single test execution. The library prints diagnostic messages in the log. Please note that the log is only populated during single test pass, not during optimization. If optimization is enabled all output from EA is dumped by the tester, so you'll only see the error.
- Custom step size is not specified -- wfo_stepSize is custom and wfo_customStepSize is 0
- Custom window size is not specified -- wfo_windowSize is custom and wfo_customWindowSizeDays is 0
- Window size is less or equal to step -- wfo_windowSize is > 0 and wfo_windowSize <= wfo_stepSize
- Window size is less or equal to custom step -- wfo_windowSize is > 0 and wfo_stepSize is custom and wfo_windowSize <= wfo_customStepSize
- Custom window size is less or equal to step -- wfo_stepSize is > 0 and wfo_windowSize is custom and wfo_stepSize >= wfo_customWindowSizeDays
- Custom window size is less or equal to custom step -- wfo_stepSize is custom and wfo_windowSize == custom and wfo_customStepSize >= wfo_customWindowSizeDays
- Datetime overflow: [tester time] [real time] -- TimeCurrent() plus wfo_stepOffset multiplied by (wfo_stepSize or wfo_customStepSize) >= [real time]
All errors except for the last one are critical and require you to adjust input parameters appropriately.
The last case is not actually an error but a feature (a special intended behavior). It was introduced in WFO 1.3 for MetaTrader 4 and inherited in all versions of WFO for MetaTrader 5. Its purpose is to improve performance by means of early drop off of those optimization passes, for which in-sample data overlaps with ending date of the tester. Before this feature, all such passes were just a waste of time, because they produce void results.
This error may become critical if you use genetic optimization. The problem originates from the feature of MetaTrader, which drops further genetic generations if too few meaningful results obtained during current generation. Every pass with "incorrect input parameters" error is considered void, but the tester does not try another permutations (random parameter set) to get a valid result instead. Instead of this the tester just gives up, and the number of valid passes decreases. Unfortunately, if the number of productive passes is less than a half of predefined size of population (for example, for the 1-st generation the tester seeks to create 512 records), then optimization process is stopped with no result at all. I consider this a design bug in MetaTrader, but we can do nothing with this. As a workaround you can enlarge the overall period, or make number of forward interations smaller, or make window/step size smaller. All of these measures will minimize number of passes with overflown ending dates. Hence more valid results will be generated and the tester will not abort optimization. Other approach is to use full (slow) optimization when possible.
Q. What are the minimal custom window and custom step sizes?
A. It's recommended to set the custom window size starting from 1 month and up, ultimately - two weeks. The minimal custom step, as well as its increment (both are assigned in percents in this mode) should be selected so, that after their translation into the time scale (calculated as a fraction of window size) you'll get at least 1 full day. If the minimal step size is 10%, it makes no sense to use window size less than 10 days. This is also true for the their increments. For example, one may choose 10 days as the minimal window and its increment. Then you should use 10% as the minimal step and its increment to avoid fractions of days. If you need the step 5% and 5% increment, then the minimal window size and its increment should be 20 days. Since single day range or even two consecutive days range may map to a weekend, such range will be empty (no bars at all). This is why it's recommended to use minimal ranges at least several days.
Q. Why do I see discrepancies in some rows of walkforward tables?
A. Indeed, in some cases net Profit(1/2) can differ from combination of gross profit and gross loss, that is P(1/2)-L(1/2). There can be many reasons for this.
This can happen, for example, if you use wfo_built_in_strict custom estimator. Its purpose is to eliminate outlying trades from total income. If a sporadic enormous trade occured in specific pass, it's excluded from net profit, but remains in the gross parts (either profit or loss). Starting from version 1.9 of the library, such passes are marked by ª sign. You can run the single pass to view details of the trades in the tester log.
Another reason why the net and gross values may differ is the principle of how in-sample (IS) optimization window and forward test (out-of-sample, OOS) are organized. Since they are executed in the single pass, some trades may start before the split point between the window and forward step, and be closed after that point. The library provides an option to close all trades automatically on the split point by calling wfo_setCloseTradesOnSeparationLine(true) in OnInit. Then the situation is eliminated. But if you do not enable the option, the trades may "penetrate" through the separation line, and as a result IS and OOS statistics (including number of trades) will be biased. Again, the bias can be caused by different factors, such as the following ones.
Normally, you call wfo_OnTick before your trade code (you need to do so, because the event handler instructs your code whether it is allowed to trade or not). At this moment the library monitors current stats (running total and number of completed trades). After this you may want to close an open position (if any) on the same tick, which will change the stats, and the updated "status quo" will be monitored by the library on the next tick only. But what if the tester is running in open prices mode? Then the next tick will occur on the next bar only. And if the current bar and the next bar are on separation line, the library will count the changes for forward step instead of optimization window.
Another possible cause of mismatch is commission. The library counts trades by out deals, when a profit/loss alters balance. But commission (if any) can be charged at entry deal as well (apart from exit deal). Hence the monitoring process may "see" entry commission in the balance before the split date, whereas the trade outcome and exit commission are trapped after it. Other fees (swaps, etc) may also interfere in similar manner.
Usually, the potential number of such boundary trades is very few (if not 0), and their effect on statistics can be neglected, if the pass contains sufficient number of trades. At least several hundred of trades is recommended for proper (meaningfull) WF analysis.
Starting from version 1.9 of the library, all mismatches caused by abovementioned reasons are automatically fixed in the report (if the boundary trades are not closed; if they are closed - there are no mismatches of this kind). If you need a special handling of positions closure, you may close positions from your code instead of enabling wfo_setCloseTradesOnSeparationLine(true).
You can disable the automatic fix by special setup (wfo_setAdvancedOptions(WFO_FLAG_MISMATCH_FIX_OFF)), then all affected passes will be marked by ™ sign and can be investigated by running single tests.
Also current implementation of the library does not support windows larger than 994 days. If you try to exceed this value, you'll see inconsistencies in resulting reports as well.
Q. I think I found a bug. What should I do?
A. Show me your settings for the tester and for EA. Describe what you want to achieve and what you get. Send me generated files if any. If it's a library issue, send me the tester logs with errors. If it's a reporter issue, send me logs from the Experts tab.
