Getting pass number while EA is going through optimization in strategytester? - page 2

[Dave Bieleveld]

Dave Bieleveld #:

This is about the same reason I need it. I already parse the xml and html reports. However, now I need to store the results in a sqllite database, which content is read by an external program.

So, the only way I can think of to solve this is to construct a custom identifying unique value yourself. For example a string with your parameter names and values, or a hash. When you store it in json format, you can interpret the identifier in external programs.

And remember that you keep it's signature consistent and normalize the parameter values, their order and number of parameters. Because "{inp_myParamx = 0.0, inp_myParamy = "value"}" is not the same as "{inp_myParamx = 0.000, inp_myParamy = "value"}" or  "{inp_myParamy = "value", inp_myParamx = 0.000, inp_myParamz = -1}" .

I stand corrected. As it turns out, in the OnTester() you a remote agent can not write to db or file. That makes sense, because where would it write to for a remote agent and how to retrieve that data then?

So you have to use frames, which include parameters and the pass for identifying an optimization pass.

The concept is actually not so difficult. With frameadd, you can "send" a value or an array of values to the main, local terminal. Send it from OnTester(). 

In the OnTesterPass() you can catch the sent frame with FrameNext, which contains that pass# and your array. With FrameInput you get all parameters and values of that pass.

That's it.

double OnTester() // On_end_of_test of the (remote) agent that processed a optimization pass.
  {
//--- Create a frame
   double testerStatistics[TESTER_STATISTICS_COUNT];

   GetTesterStatistics(testerStatistics);

   FrameAdd("TESTER_STATISTICS", 1, 0, testerStatistics);   // Send to the local terminal

   return(0.0);
  }

void OnTesterPass()  // On_frame_received on the local terminal that started the optimization test
  {
//--- Receive frames
   double testerStatistics[1];
   ulong  pass    = 0;           // The number of a pass in the optimization, during which the frame has been added
   string name    = "";          // Public name/frame label
   long   id      = 0;           // Public id of the frame
   double value   = 0.0;         // Single numerical value of the frame

   FrameNext(pass, name, id, value, testerStatistics);   // Receive the frame from the remote agent.

//--- Receive parameter info
   string   parameters[];
   uint     parameterCount;

   FrameInputs(pass, parameters, parameterCount);
   ArrayResize(parameters, parameterCount);

   ProcessFrames(pass, name, id, value, testerStatistics, parameters);
  }

[Alain Verleyen]

Dave Bieleveld #:

I stand corrected. As it turns out, in the OnTester() you a remote agent can not write to db or file. That makes sense, because where would it write to for a remote agent and how to retrieve that data then?

So you have to use frames, which include parameters and the pass for identifying an optimization pass.

The concept is actually not so difficult. With frameadd, you can "send" a value or an array of values to the main, local terminal. Send it from OnTester(). 

In the OnTesterPass() you can catch the sent frame with FrameNext, which contains that pass# and your array. With FrameInput you get all parameters and values of that pass.

That's it.

You can send a lot more than a value or an array.

https://www.mql5.com/en/forum/165457#comment_5077892

[Dave Bieleveld]

Alain Verleyen #:

You can send a lot more than a value or an array.

https://www.mql5.com/en/forum/165457#comment_5077892

Great! That should satisfy most requirements when dealing with remote agents and their results. (I've also tested it with structs (classes not yet), also works.)

There are two very important additions I like to add. It addresses the following quote from the manual:

OnTesterDeinit - Event Handling - MQL5 Reference - Reference on algorithmic/automated trading language for MetaTrader 5

Keep in mind that optimization frames sent by test agents using the FrameAdd() function may come in bundles and take time to deliver. Therefore, not all frames, as well as TesterPass events, may arrive and be processed in OnTesterPass() before the end of optimization. If you want to receive all belated frames in OnTesterDeinit(), place the code block using the FrameNext() function.

1) When storing data in files or sqlite, do not proces frames in OnTesterPass() but in OnTesterDeinit(). OnTesterPass() quite often ends when there are still frames to be received.

2) OnTesterDeinit() does this too but that's why you have to create a process that processes all frames until none left. 

Another reason why you should place it in OnTesterDeinit() is that OnTesterPass() can fire multiple time (for each frame received) and OnTesterDeinit() only once. If you create the loop in OnTesterPass() the loop will fire as much as there are OnTesterPass() events, which creates unnecessary performance impact, locking etc.

The following code has been tested and gives me continuous stable results, meaning no skipped or non-processed frames. The for loop ensures that exactly x frames are processed, where x = number of passes. The number of passes can be calculated with ParameterGetRange().

• Note that some functions may come from my custom private library. Should not be a problem for one to program themselves.
• Note that some of the code is "over generic/abstract". That's because it's also from my private library.
• Note that the only time it doesn't work is when the optimization run is in cache. (I personally made a little console app that monitors the cache dir and deletes any file as soon as it is created)

//+------------------------------------------------------------------+
//| Dave Bieleveld - Process frames sent by other agents             |
//+------------------------------------------------------------------+
void OnTesterDeinit()
  {
   ulong numberOfPasses = ULONG_MAX;

   double testerStatistics[];
   ulong  pass    = 0;           // The number of a pass in the optimization, during which the frame has been added
   string name    = "";          // Public name/frame label
   long   id      = 0;           // Public id of the frame
   double value   = 0.0;         // Single numerical value of the frame

//--- Receive parameter info
   string   parameters[];
   uint     parameterCount;

//--- Iterating through the remaining passes
   for(ulong i = 0; i < numberOfPasses; i++)
     {
      //--- Receive frames
      FrameNext(pass, name, id, value, testerStatistics);

      //--- Receive parameters
      FrameInputs(pass, parameters, parameterCount);
      ArrayResize(parameters, parameterCount);

      //--- Set the actual number of passes
      if(numberOfPasses == ULONG_MAX)
         numberOfPasses = GetNumberOfPasses(parameters, parameterCount);

      //--- Process frame
      int fileHandle = FileOpen(StringFormat("FRAMETEST\\PREDATABASE %I64u_%s.csv", pass, "testType"), FILE_WRITE | FILE_CSV | FILE_COMMON);
      FileWrite(fileHandle, pass);
      FileClose(fileHandle);

      //--- Clean up
      ArrayFree(parameters);
     }
  }

//+------------------------------------------------------------------+
//| Dave Bieleveld - Get number of passes in an optimization.        |
//+------------------------------------------------------------------+
ulong GetNumberOfPasses(string& parameters[], const int parameterCount)
  {
// Premise: We can be sure there is at least 1 pass, and 1 range enabled, else the optimizer won't even start, or have an OnTesterDeinit event.
   ulong passes = 1;
   ParameterRange parameterRange[];
   ArrayResize(parameterRange, parameterCount);

   if(!GetParameterRanges(parameters, parameterCount, parameterRange))
      return -1;

   for(int i = 0; i < parameterCount; i++)
     {
      if(parameterRange[i].Enabled && parameterRange[i].DataType == PARAMETER_DATATYPE_LONG)
        {
         const ulong lPasses = (ulong)MathFloor((parameterRange[i].lStop - parameterRange[i].lStart) / parameterRange[i].lStep); 
         passes *= lPasses;
        }
      else
         if(parameterRange[i].Enabled && parameterRange[i].DataType == PARAMETER_DATATYPE_DOUBLE)
           {
            const ulong dPasses = (ulong)MathFloor((parameterRange[i].dStop - parameterRange[i].dStart) / parameterRange[i].dStep);
            passes *= dPasses;
           }
     }

   ArrayFree(parameterRange);
   return passes + 1;   // Plus 1, because the range is from including -> until including
  }

enum ENUM_PARAMETER_DATATYPE
  {
   PARAMETER_DATATYPE_LONG,
   PARAMETER_DATATYPE_DOUBLE,
  };

struct ParameterRange
  {
   ENUM_PARAMETER_DATATYPE DataType;
   string            Name;
   bool              Enabled;

   //--- Is set when datatype is double
   double            dValue;
   double            dStart;
   double            dStep;
   double            dStop;

   //--- Is set when datatype is long
   long              lValue;
   long              lStart;
   long              lStep;
   long              lStop;
  };

//+------------------------------------------------------------------+
//| Dave Bieleveld - Get all parameter values and ranges.            |
//+------------------------------------------------------------------+
bool GetParameterRanges(string& parameters[], const int parameterCount, ParameterRange& parameterRange[])
  {
   for(int i = 0; i < parameterCount; i++)
     {
      const string parameterName       = StringToParameterName(parameters[i]);
      const bool   isDoubleCandidate   = Contains(parameters[i], "."); //Unstable hack. Doubles can have the value 1 as well as 1.0.

      bool     enabled;
      double   dValue, dStart, dStep, dStop;
      long     lValue, lStart, lStep, lStop;

      if(!isDoubleCandidate && ParameterGetRange(parameterName, enabled, lValue, lStart, lStep, lStop))
        {
         parameterRange[i].DataType = PARAMETER_DATATYPE_LONG;
         parameterRange[i].Name     = parameterName;
         parameterRange[i].Enabled  = enabled;
         parameterRange[i].lValue   = lValue;
         parameterRange[i].lStart   = lStart;
         parameterRange[i].lStep    = lStep;
         parameterRange[i].lStop    = lStop;
        }
      else
         if(isDoubleCandidate && ParameterGetRange(parameterName, enabled, dValue, dStart, dStep, dStop))
           {
            parameterRange[i].DataType = PARAMETER_DATATYPE_DOUBLE;
            parameterRange[i].Name     = parameterName;
            parameterRange[i].Enabled  = enabled;
            parameterRange[i].dValue   = dValue;
            parameterRange[i].dStart   = dStart;
            parameterRange[i].dStep    = dStep;
            parameterRange[i].dStop    = dStop;
           }
         else
            continue; //--- Datatype not supported (like a parameter with datatype string);
     }

   return true;
  }
//+------------------------------------------------------------------+

[Zheng Chen Liang]

Now I understand the workflow is that OnTesterInit() in the main process then tasks distributed to agent processes. Agent process does OnTester() upon a task is finished then pass array or file information by FrameAdd() to the main process. Upon this the main process runs OnTesterPass(). Finally the main process ends with OnTesterDeinit(). This is not clearly expressed in the F1 document for beginners but thank you for the example.

[Imre Durucz]

Based on Getting data frames in terminal (which I hardly understood), I have written the following code that works(!) and writes the tester results in a file (...\Files\output.csv) including the Pass number.

Only works with Slow optimization though (with Fast, the Pass numbers won't match).

You have to choose "Custom max" to optimize by but you can change the criterion in the code below (I have put STAT_RECOVERY_FACTOR in this example).

Basically, you can copy paste the entire code into your EA except the input parameters and it will work as is.

input bool param1;
input long param2;
input double param3;
input string param4;

int handle;

int OnInit(){
   return INIT_SUCCEEDED;
}

double OnTester(){
   FrameAdd(NULL, 1, TesterStatistics(STAT_RECOVERY_FACTOR), NULL);
   return TesterStatistics(STAT_RECOVERY_FACTOR);
}

void OnTesterInit(){
   handle = FileOpen("output.csv", FILE_WRITE | FILE_CSV | FILE_ANSI, ",");
}

void ProcessFileFrames(){
   static ulong framecount = 0;
   ulong   pass;
   string  name;
   long    id;
   double  value;
   uchar   data[];
   string  params[];
   uint    count;
   while(FrameNext(pass, name, id, value, data)){
      if(FrameInputs(pass, params, count)){
         string header, record;
         if(framecount == 0){
            header = "Pass,";
         }
         record = (string)pass + ",";
         for(uint i = 0; i < count; i++){
            string name2value[];
            int n = StringSplit(params[i], '=', name2value);
            if(n == 2){
               long pvalue, pstart, pstep, pstop;
               bool enabled = false;
               if(ParameterGetRange(name2value[0],
                  enabled, pvalue, pstart, pstep, pstop)){
                  if(enabled){
                     if(framecount == 0){
                        header += name2value[0] + ",";
                     }
                     record += name2value[1] + ",";
                  }
               }
            }
         }
         if(framecount == 0){
            FileWriteString(handle, header + "Value\n");
         }
         FileWriteString(handle, record + DoubleToString(value) + "\n");
      }
      framecount++;
   }
}

void OnTesterPass()
{
   ProcessFileFrames();
}

void OnTesterDeinit()
{
   ProcessFileFrames();
   FileClose(handle);
}