Walk-forward optimization library for MetaTrader 4: FAQ

15 August 2016, 19:43
Stanislav Korotky

This post is a part of documentation of WalkForwardOptimizer library for MetaTrader4 - here is the table of contents.

Frequently asked questions

Q. Why should I use WFO?

A. WFO is an automated implementation of the process of EA testing and monitoring, which a trader does normally perform by hands. In addition, WFO provides a justified answer for the questions, which period of EA optimization to choose, and how long the optimized EA performance will last in average, so that one could know at which pace to perform re-optimization.

Q. Why WFO library is not available for MetaTrader 5?

A. MetaTrader 5 will most likely support walk-forward optimization natively as a new mode of the built-in tester in future. If you desperately want this library to be ported to MetaTrader 5, please, send your request to the author - on high demand it can be re-worked and provided freely to all requesters who owns original version for MetaTrader 4.

Q. Why WF optimization and testing periods are performed in a single tester pass?

A. A single tester pass, divided into optimization window and forward test, is used because it makes it simple to associate corresponding optimization and testing periods. In fact, MQL4 does not provide any means for linking 2 independent tester passes with each other. Moreover, even if it was possible, testing forward steps would be treated as optimization runs, and thus introduce a bias on optimal parameters search. And if we would mark forward tests with zero performance indicator to eliminate the bias, the tester would skip most of forward tests while using genetic algorithm. And refusal to use genetics seems impractical.

Q. Why standard optimization restrictions must be disabled in the tester while using WFO?

A. Indeed, all flags on the Optimization tab of the Expert Properties dialog should be cleared off before using WFO library. This is so because the restrictions are applied on corresponding values calculated for entire period, but the period is internally divided by WFO on optimization and forward testing parts, where only optimization values should matter. WFO library calculates all important parameters separately for both parts, and provides you an access to optimization indicators via special variables, that can be used in custom formulae. If you want to use a restriction, you may specify an expression for performance estimator where the specific parameter is checked against a threshold, and if condition is not met, estimator value set to 0. For example, if you want to drop all passes with relative drawdown larger than 50%, you can multiply your main performance estimator to the following expression "min(max(50 - DDREL, 0), 1)".


Q. Some values of profit factor in the tables seem strange. Why?

A. Because profit factor is calculated as a ratio of profits and losses, there can be cases where it's actually undefined - this happens if there are no losses at all. By default, MetaTrader 4 assigns DBL_MAX value for profit factor in such cases, but this is not a good idea, because a test with 1 profitable trade and another test with 10 profitable trades get the same profit factor. Moreover, if profit factor is used in a compound performance estimator, DBL_MAX will most likely introduce irregularity. This is why, the library provides 2 different ways of handling "overflown" profit factors. When the estimator is wfo_built_in_loose or wfo_built_in_strict, profit factors larger than 10 are adjusted as 10 multiplied on square root of number of trades. When any other estimator is selected, you may use wfo_setPFmax function to specify a relatively big number as profit factor upper limit, but significantly lower than DBL_MAX, for example, 100. Finally, if you choose wfo_expression, you can perform special cases handling by calculating profit factor surrogate from gross profit (GP variable) and gross loss (GL variable) with arbitrary specific formula.


Q. How do I know that a walk-forward test is passed or failed?

A. The WFO library does not explicitly mark walk-forward results as passed or failed, because user's preferred criteria may differ. As rule of thumb, a test is considered passed if efficiency is over 50% and consistency is over 50%, as well as drawdown does not exceed 30%. One may also take care that distribution of profits on forward tests does not contain distinct outliers (when largest profit per pass produces more than 50% of total profit). All this factors can be checked visually in the tables of specific report.


Q. Why Walk-Forward Drawdown is marked as "rough estimate, normalized"?

A. This drawdown is calculated on forward balance curve using solely profits or losses on every step (as a whole), and does not take underlying trades into account. This makes the drawdown "rough estimate". In order to calculate an exact drawdown one should have a complete list of trades, but WFO library does not save them for the sake of efficiency. Indeed, if it would do this, every single pass would generate a separate file, and the number of passes can be hundreds of thousands.

The drawdown is called normalized, because it's calculated starting from annualized profit on in-sample data (optimization period) instead of initial deposit. Conventional drawdown may vary dramatically depending from an arbitrary deposit amount chosen by trader voluntarily. This makes it hard to understand the risks. For example, with initial deposit equal to 10000 a drawdown on 1000 is only 10%, but for 2000 it's 50%. When annualized profit is used instead of deposit, drawdown expresses the risks in universal measure scaled accordingly to the trading system's potential efficiency (average profitablility).


Q. What's the meaning of Deviation in the reports, how to interpret it?

A. The table row with Deviation shows standard deviation (σ) of corresponding value as a measure of its dispersion, that is how far it fluctuates up or down across its Average. If deviation is equal or greater than average (or mean), then corresponding parameter seems unreliable, because its observed value will most likely change sign easily regardless to expected one, and this is very important for example for profits. For example, if average profit is 100 and deviation is 100, then probability of a loss is approximately 15.8%, if deviation is 50 - probability of a loss is ~2.2%, and if deviation is 33, then probability of a loss is only ~0.1%. On the contrary, if deviation is 200, then probability of a loss increases up to ~31%. This is illustrated by the graph below.

Gauss distribution for expected payoff and its dispersion (standard deviation)

We use Normal distribution here, that can be debatable, but it's well studied and applicable (under certain assumptions) for most of long running series of experiments (let us assume we have a sufficient number of walk-forward passes).


Q. My optimization has stopped with the error "Tester memory handler: tester stopped because not enough memory". What can I do?

A. You're trying to perform too massive optimization which consumed all resources available for the MetaTrader 4 tester. This is not an issue of the library. This is a limitation of the tester and operating system. You need somehow simplify your optimization settings or add more resources to your PC. Here is some methods:

  • shorten the range of dates of testing (for example take 2-3 years instead of 5-6),
  • decrease number of parameters enabled for optimization,
  • reduce number of iterations for parameters being optimized (for example, you could decrease number of forward steps in the expense of increasing forward step size in the same number of times),
  • try to free up MT4 memory by unloading everything except the EA being optimized (note, that MT4 should be restarted after unloading because it caches many things internaly and they left in memory),
  • enable genetic algorithm in the tester,
  • add phisical memory to you PC .


Q. What are the minimal custom window and custom step sizes?

A. It's recommended to set the custom window size starting from 1 month and up, ultimately - two weeks. The minimal custom step, as well as its increment (both are assigned in percents in this mode) should be selected so, that after their translation into the time scale (calculated as a fraction of window size) you'll get at least 1 full day. If the minimal step size is 10%, it makes no sense to use window size less than 10 days. This is also true for the their increments. For example, one may choose 10 days as the minimal window and its increment. Then you should use 10% as the minimal step and its increment to avoid fractions of days. If you need the step 5% and 5% increment, then the minimal window size and its increment should be 20 days.


Q. I think I found a bug. What should I do?

A. Show me your settings for the tester and for EA. Describe what you want to achieve and what you get. Send me generated files if any. If it's a library issue, send me the tester logs with errors. If it's a reporter issue, send me logs from the Experts tab.

Share it with friends: