Trader's self-deception: distrust of forwards. - page 2

New comment
 
Youri Tarshecki:

Anyone can do the simplest experience - get trading results on a demo account and run the same system with the same settings in their tester for the same period. I.e. get a real and a test virtual forward and compare it.

This is not a valid comparison as the strategy tester uses only 4 quotes for the bar open, close, max, and min. As a rule, Expert Advisors work with all intra-bar quotes on demo or real accounts.

Youri Tarshecki:

In fact, the simulation of system behaviour by running it on a non-optimised period of history is the most efficient way of analysis for a trader. The reality check is of course the most reliable way, but unfortunately one has to live forever to go through all variants in real accounts. I.e. modelling like forward to the past is the most effective way of research. But then why are forwards completely absent from the discussions? Maybe it's because there is no handy software for processing and analyzing results of multiple tests, both back and forwards?

I agree with you. You seem to do a lot of that. Please tell me which financial tool you prefer for such analysis. I have an EA that is triggered only on bar opening and I want to test it with similar modelling.

 

 Скажите пожалуйста какой финансовый инструмент Вы предпочитаете для подобного анализа. У меня есть советник который срабатывает только на открытие бара. Хочу проверить его подобным моделированием.

What I mean is that regardless of the tool, the testing methodology should include forwards. But people ignore them en masse. Those who do use them, as a rule, estimate them by eye. In other words, there are no programmes that would introduce objective standards into the evaluation of forwards.
 
Youri Tarshecki:
What I mean is that regardless of the tool, the testing methodology should include forwards. But people ignore them en masse. Those who do use them, as a rule, estimate them by eye. That is, there are no programs that would introduce objective standards into the evaluation of forwards.
Different instruments have different properties of market behaviour. Including the "inertia of profitability" in the forward test. That is why I wanted to ask: have you found an instrument (or a kind of instrument) showing the best result in your experiments, all other things being equal?
 
Youri Tarshecki:
What I mean is that regardless of the instrument, the testing methodology should include forwards. But people ignore them en masse. Those who use them, as a rule, estimate by eye. That is, there are no programs that would introduce objective standards into forward pricing.

Massively ignoring forward tests. The reason is simple - successfully passed forward only indicates that the market on the forward section is similar to the optimization section. Strategy Tester perfectly adjusts to the current market dynamics. As a result, after dividing the optimization segment into 12 forward parts we will get 12 unrelated sets of parameters each of which will shed 11 of 12 time sections. Instead, it is better to find one, albeit averaged, set of parameters for a long history, than rush from one set of parameters to another.

The market doesn't change, it's just that each trading system captures only one particular state of the market, and the market has many of these states.

 
It is better not to trust the forwards than to trust them. And if you do, it's only if you use the WFA
 
Vasiliy Sokolov:

Massively ignoring forward tests. The reason is simple - successfully passed forward only indicates that the market on the forward section is similar to the optimization section. Strategy Tester perfectly adjusts to the current market dynamics. As a result, after dividing the optimization segment into 12 forward parts we will get 12 unrelated sets of parameters each of which will shed 11 of 12 time sections. Instead, it is better to find one, albeit averaged, set of parameters for a long history, than rush from one set of parameters to another.

The market doesn't change, it's just that each trading system captures only one particular state of the market, and there are many of these states in the market.

You cannot divide the "optimised section into 12 forward parts" because a forward is by definition an unoptimised section. That is, you are contrasting the method of forward checking itself and simply the notion of averaging.

The purpose of a forward is to find out how the adviser behaves in a situation of a non-optimised future. What you are talking about has more to do with finding the best piece of history for optimization. I.e. if you believe that the long history works better, no problem, let's load the tester with 2 years Back - 1 year Forward. And then 2month Back -1month Forward 12 times. If the first approach forward performance is better than the sum of the second approach forward performance, then the first approach is better, and vice versa. The very fact that you do optimize the Expert Advisor suggests that you believe in the inertia of the market, but simply believe that it is manifested in a significantly larger period of time. But it's not like you're optimising over ALL of the available history of 30 years, so you're also taking into account the volatility of the market. It's an eternal contradiction of constancy and variability, but by comparing forwards of different test periods you can more accurately understand how this contradiction is resolved in reality. Until you have checked - is your statement purely intuitive or just habitual.

 
Youri Tarshecki:
Suppose that I use OnTesterPass in the code of an EA that I am testing. How will it know that this run is a forward looking one and not an ordinary variable optimization run? And even if he manages to do it, it will be different files, and we need one table for many forwards of one EA . And then another table, and another one for another one, and all of it in one file.

As far as I remember, this is what I did - I saved the time of the first trade into the frame - this is the "balance". Accordingly, when recording parameters and indicators of all the runs in one file, it is easy to distinguish the backtest from the forward by two distinctive dates. I have saved all runs in exactly one file (opened in OnTesterInit and closed in OnTesterDeinit).

The details were here:

Forum on trading, automated trading systems and testing trading strategies

Working with Frames

Stanislav Korotky, 2014.11.07 15:23

In the OnTester handler you can add your own frames with data using FrameAdd - the important thing is that this is all done in the testing agent, which can be remote (e.g. in the cloud).

OnTesterInit, OnTesterDeinit, OnTesterPass are called on the terminal where the testing/optimization process is started and from where tasks are distributed to agents. Inside OnTesterPass you can access all those frames that have been added to the test agents using FrameAdd, roughly like this (bitten out of the old EA, may not compile):

void OnTesterPass()
{
  string  name;
  ulong   pass;
  long    id;
  double  value, data[];
  string  params[];
  uint    par_count;
  string  output = "";
  ushort eq = StringGetCharacter("=", 0);

  while(FrameNext(pass, name, id, value, data)) // data - массив положенный во фрейм с помощью FrameAdd
  {
    output = pass + ";" + (data[0]);
    
    for(int i = 1; i < 10; i++) output += ";" + data[i];
    
    if(FrameInputs(pass, params, par_count))
    {
      for(uint i = 0; i < par_count; i++)
      {
        string pair[];
        int n = StringSplit(params[i], eq, pair);
        if(n == 2)
        {
          long pvalue, pstart, pstep, pstop;
          bool enabled = false;
          if(ParameterGetRange(pair[0], enabled, pvalue, pstart, pstep, pstop))
          {
            if(enabled)
            {
              output += ";" + pair[1];
            }
          }
        }
      }
    }
    
    output += "\n";
    // TODO: FileWriteString(fhandle, output);
  }
}

and here:

Forum on trading, automated trading systems and testing trading strategies

Working with Frames

Stanislav Korotky, 2014.11.07 20:15

You should add frames, as I wrote above, during the run. By idea, if you have parameters in the EA, you should get them inside the frame along with your data, i.e. params should not be empty. Here is some more code from the same Expert Advisor.

int fhandle = -1;

void OnTesterInit()
{
  // готовим каким-то образом файл, как нам нужно
  fhandle = FileOpen("tester-" + _Symbol + GetTickCount() + ".csv", FILE_WRITE|FILE_CSV|FILE_ANSI);
  FileWriteString(fhandle, "Pass;First;LR;Profit;Expected Payoff;Profit Factor;Recovery Factor;Sharpe Ratio;Custom;Equity DD %;Trades\n");
}

void OnTesterDeinit()
{
  FileClose(fhandle);
}

double OnTester()
{
  // собираем данные по эквити
  double LR = CountChannels(Equity);
  
  // конструируем собственный критерий оптимизации
  double result = MathAbs(LR) * TesterStatistics(STAT_PROFIT) * TesterStatistics(STAT_PROFIT_FACTOR) * TesterStatistics(STAT_TRADES) * (100 - TesterStatistics(STAT_EQUITY_DDREL_PERCENT));

  // рассчитываем еще какие-то данные
  //...

  // складываем все в массив
  double data[10];
  // data[0] = first;
  // data[1] = LR;
  // data[2] = TesterStatistics(STAT_PROFIT);
  // ...
  // data[7] = result;
  // ...
  // data[9] = ...

  // отправляем данные в терминал
  if(!FrameAdd(MQL5InfoString(MQL5_PROGRAM_NAME), 1, LR, data))
    Print("Frame add error: ", GetLastError());
  else
    Print("Frame added, Ok");

  return(result);
}

 

Personally, I think it is more logical to check the results in the past rather than in the future, because the most recent market changes are the most significant.

For 15 minutes I optimised, sifted out unnecessary results, and saved the remaining 5-10 to the set and ran them on the history until the moment of optimisation - and chose the best ones by the chart.

I was also thinking about the process of automating these actions...

A simple solution seems to be simply putting restrictions on trading in the time range which can be switched in the EA itself, as a result we will get two virtual runs on the history for one real run, which can be easily compared in excel.

 
-Aleks-:

Personally, I think it is more logical to check the results in the past rather than in the future, because the most recent market changes are the most significant.

For 15 minutes I optimised, sifted out unnecessary results, and saved the remaining 5-10 to the set and ran them on the history until the moment of optimisation - and chose the best ones by the chart.

I was also thinking about the process of automating these actions...

A simple solution seems to be simply putting restrictions on trading in the time range which can be switched in the EA itself, as a result we will get two virtual runs on the history for one real run, which can be easily compared in excel.

Focusing on the LAST change is another kind of self-deception. Naturally, you need to use fresh settings for trading. But limiting yourself to optimizing for the most recent period, you will never be able to understand whether your Expert Advisor is resistant to future changes. Look at the second picture in my post - you can clearly see it. Forward is a comparison of the past with the past. Your method has nothing to do with forward analysis. Virtually any Expert Advisor can be optimized to a nice backtest. The pattern here is as follows - the more settings are optimized, the more beautiful the backtest is. And the uglier is the forward. Forwards help to find exactly the logic that uses more long-term patterns, and therefore is more promising.
 
Forward is a useful check, but not a panacea. It is possible to select a variant of parameters in the optimisation results that provide a good forward, but this does not guarantee good results on the real.
123456789...17
New comment
Reason: