Possible Memory Leak when working with Optimization Results with FrameAdd...

New comment
 

I am trying to record the state of an Indicator over the time of a backtest in MQL5.
I am running the optimization on something like 200k params.

I am recording the "Side Changes" of an indicator with a timestamp. So for example if an Indicator crosses from <0 to >0...
So the actual values to be stored are datetime and an int with values of [-1, 0, 1]

I am storing the side changes in my signal class as:

struct SideChange {
   datetime date;
   int side;
}

class member

SideChange m_side_changes[];

on every side change I call this

      datetime date;
      if (Expert_EveryTick) {
         date = TimeCurrent();  // TODO get current bar time
      } else {
         if (!SeriesInfoInteger(m_symbol.Name(),m_period,SERIES_LASTBAR_DATE,date))
         { // If request has failed, print error message:
            Print(__FUNCTION__+" Error when getting time of last bar opening: "+IntegerToString(GetLastError()));
            return(false);
         }
      }

      m_side_changes[m_side_cnt].date = date;
      m_side_changes[m_side_cnt].side = m_side;
      m_side_cnt++;

OnTester calls this

   ArrayResize(m_side_changes, m_side_cnt);
   ResetLastError();
   if(!FrameAdd(m_symbol.Name(),
                     0,  // func not implemented
                     0,  // value in SideChanges array
                     m_side_changes)) {
      Print("Frame add error: ", IntegerToString(GetLastError()));
      return false;
   }
   Print("Added ", m_side_cnt, " side changes to the frame");
   ArrayFree(m_side_changes);
   return true;

OnTesterDeinit calls this

   FrameFirst();
   DatabaseTransactionBegin(db);

//--- go through frames and read data from them
   bool failed=false;
   // while(FrameNext(pass, symbol, func, side, date))
   while(FrameNext(pass, symbol, func, value, sides)) {


      for(int i=0;i<ArraySize(sides);i++) {
         //--- write data to the table
         string request=StringFormat("INSERT INTO SIDES (PASS,DATE,SIDE) "
                                     "VALUES (%u, %d, %d)",
                                     symbol, pass, sides[i].date, sides[i].side);

         //--- execute a query to add a pass to the PASSES table
         if(!DatabaseExecute(db, request)) {
            PrintFormat("Failed to insert pass %d with code %d\nSIDES", pass, IntegerToString(GetLastError()), request);
            failed=true;
            break;
         }

         frame_cnt++;
      }

      if (transaction_limit != -1 && frame_cnt >= transaction_limit) {
         break;
      }
   }

//--- if an error occurred during a transaction, inform of that and complete the work
   if(failed) {
      Print("Transaction failed, error code=", IntegerToString(GetLastError()));
      DatabaseTransactionRollback(db);
   } else {
      DatabaseTransactionCommit(db);
      Print("Transaction done successfully ", frame_cnt, " frames ");
   }

When I run this the Memory on my PC goes up VERY quickly and I reach 40GB in like 10 Minutes.
I noticed that on the first run the memory is somewhat OK but once there is any optimization cache somewhere around the memory goes UP to max out my ram and then my PC crashes due to high memory pressure.

The hight ram usage stays up even after the Optimization process has finished and only gets fried if I close MT5.

I have a Ryzen 2700X and 64GB of RAM so memory should not be a problem.

Where are the Frames allocated?
Should I manually deallocate the frames or the array that FrameNext returns? (
Is some kind of Frames cache using up my memory?
How do I reset this cache?

I tried to delete the cache files but this does not reset the effect.
Tester/cache/*
MQL5/Files/Tester/*

 
It's really hard to answer without a test case to reproduce the issue. Unless someone else experimented something similar which is not my case. Some days ago I worked with 100.000 passes using Frames without a problem, but I probably had less data sent with a frame.
 
Was there a resolution to this? I am having a memory leak issue too. But it's not with Frames. My code also connects to a sqlite db.
New comment
Reason: