Slow optimizations on AMD ryzen 7950x and taking up all RAM

New comment
 

I'm having trouble backtesting and running genetic optimization on this code on every tick as it runs very slowly. My current setup is AMD 7950x with 64gb of DDR5 ram. The computer crashes when i try to run optimizations. I have tried to disable the number of agents to half but now it runs painfully slow. 

Can anyone suggest any code optimizations?

#include <Trade\Trade.mqh>
#include <Trade\PositionInfo.mqh>
#include <Math\Stat\Math.mqh>

//input double rm = 0.1;
input int nPeriod = 1400;
input double threshold = 2.5;
double thresholdbuffer;

input double take_profit_percentage = 70;
input double profit_rolling_trigger = 10;
double take_profit_cutoff;
double max_profit;

double temp_pause; // pause algo if stop losses are hit until market improves again

input double palpha = 205;
input double cutoff = 0.6;

input double take_profit = 0;

string            my_symbol;                               //variable for storing the symbol
ENUM_TIMEFRAMES   my_timeframe;                             //variable for storing the time frame

double myAccountBalance;
double myAccountProfit;
double myAccountEquity;
double myAccountLeverage;

double kfactor; //cointegration factor
double zScore;
double rsiValue;

double Close_bufA[];                             //dynamic array for storing closing values
double Close_bufB[];                             //dynamic array for storing closing values
double RSIBuffer[];
double spread[];

CTrade   m_Trade;                                        //structure for execution of trades
CPositionInfo  m_Position;                              //structure for obtaining information of positions

//+------------------------------------------------------------------+
//| Expert initialization function                                   |
//+------------------------------------------------------------------+
int OnInit()
  {
      my_timeframe=Period();   
      my_symbol=Symbol();
      myAccountBalance = AccountInfoDouble(ACCOUNT_BALANCE); 
      myAccountProfit = AccountInfoDouble(ACCOUNT_PROFIT);
      temp_pause = 0;
      thresholdbuffer = threshold;
      take_profit_cutoff = -1.0 * myAccountBalance;
      max_profit = 0;
      
      ArraySetAsSeries(Close_bufA,true);  
      ArraySetAsSeries(Close_bufB,true);  
      ArrayResize(spread,nPeriod);
      
      //--- create a timer with a 1 minute period
      EventSetTimer(60);
        
      return(INIT_SUCCEEDED);
  }
//+------------------------------------------------------------------+
//| Expert deinitialization function                                 |
//+------------------------------------------------------------------+
void OnDeinit(const int reason)
  {
   ArrayFree(Close_bufA);                                    //free the dynamic array Close_buf of data
   ArrayFree(Close_bufB);                                    //free the dynamic array Close_buf of data
   ArrayFree(RSIBuffer);                                    //free the dynamic array Close_buf of data
   ArrayFree(spread);                                       //free the dynamic array Close_buf of data
  } 
//+------------------------------------------------------------------+
//| Expert tick function                                             |
//+------------------------------------------------------------------+
void OnTick()
  {   
      myAccountEquity = AccountInfoDouble(ACCOUNT_EQUITY);
      myAccountBalance = AccountInfoDouble(ACCOUNT_BALANCE); 
      myAccountProfit = AccountInfoDouble(ACCOUNT_PROFIT);
      myAccountLeverage = AccountInfoInteger(ACCOUNT_LEVERAGE);

      string forexA = "AUDUSD";
      string forexB = "NZDUSD";
      
      int err1=0;                                                    //variable for storing the results of working with the indicator buffer
      int err2=0;                                                    //variable for storing the results of working with the price chart
      
      err1=CopyClose(forexA,my_timeframe,0,nPeriod+1,Close_bufA);      //copy data from the indicator array into the dynamic array iMA_buf for further work with them
      err2=CopyClose(forexB,my_timeframe,0,nPeriod+1,Close_bufB);    //copy the price chart data into the dynamic array Close_buf for further work with them
      if(err1<=1 || err2<=1)                                         //in case of errors
      {
         Print("Failed to copy data from the indicator buffer or price chart buffer");  //then print the relevant error message into the log file
         return;                                                               //and exit the function
      }
      if(ArraySize(Close_bufA) < nPeriod || ArraySize(Close_bufB) < nPeriod)
      {
         Print("Failed to copy data or insufficient data. err1:", err1, " err2:", err2);
         return;
      }
      
      double sum_ab, sum_b2; /*for calculating integration factor k*/
      sum_ab = 0;
      sum_b2 = 0;
      
      for(int i=0; i < nPeriod; i++) {
         sum_ab = sum_ab + Close_bufA[i]*Close_bufB[i];
         sum_b2 = sum_b2 + Close_bufB[i]*Close_bufB[i];
      }
      
      kfactor = sum_ab/sum_b2;
      for(int i=0; i < nPeriod; i++) {
         spread[i] = Close_bufA[i]-kfactor*Close_bufB[i];
      }
      zScore = (spread[0]-MathMean(spread))/MathStandardDeviation(spread);
      //Check zscore to remove temp pause if market recovers
      if((temp_pause < 0 && zScore > 0) || (temp_pause > 0 && zScore < 0) ){
         temp_pause = 0;
      }
      
      // Fetch RSI(14) for daily timeframe
      int rsiHandle = iRSI("AUDNZD", PERIOD_D1, 14, PRICE_CLOSE);
      if (rsiHandle != INVALID_HANDLE){
         if (CopyBuffer(rsiHandle, 0, 0, 1, RSIBuffer) > 0){
            rsiValue = RSIBuffer[0];
         }
         IndicatorRelease(rsiHandle);
      }
      
      if((myAccountProfit > max_profit) && (myAccountProfit > (profit_rolling_trigger/100) * myAccountBalance)){
         max_profit = myAccountProfit;
         take_profit_cutoff = myAccountBalance + (take_profit_percentage/100) * myAccountProfit;
      }

      Comment("K-factor is: ",kfactor, "\n",
               "Z-score is: ",zScore,"\n");            
               
      /*Trading algo*/
      /*If there is no current position*/
      if(!m_Position.Select(forexA) && !m_Position.Select(forexB)){
         if(zScore > threshold && temp_pause == 0 && rsiValue > 70){
            //Short 1 A and Long pA/pB B
            double ratio = Close_bufA[0]/Close_bufB[0];
            //Normalize Volume according to https://www.mql5.com/en/forum/192896
            double shortVol = GetSafeLotSize(forexA, palpha, myAccountLeverage);
            double longVol  = GetSafeLotSize(forexB, ratio * palpha, myAccountLeverage);
            
            Print("Entered position, zScore is :",zScore,",  k-factor is: ", kfactor);
            m_Trade.Sell(shortVol,forexA); 
            m_Trade.Buy(longVol,forexB);
         } 
         if(zScore < -1*threshold && temp_pause == 0 && rsiValue < 30){
            //Long pB/pA A and Short 1 B
            double ratio = Close_bufB[0]/Close_bufA[0];
            //Normalize Volume according to https://www.mql5.com/en/forum/192896
            double longVol  = GetSafeLotSize(forexA, palpha * ratio, myAccountLeverage);
            double shortVol = GetSafeLotSize(forexB, palpha, myAccountLeverage);
            Print("Entered position, zScore is :",zScore,",  k-factor is: ", kfactor);
            m_Trade.Buy(longVol,forexA);
            m_Trade.Sell(shortVol,forexB); 
            
         } 
      }  
      //If there is already a current trading position  
      else if(m_Position.Select(forexA) && m_Position.Select(forexB)) {
      
          // Variables to store position types
          ENUM_POSITION_TYPE posTypeA, posTypeB;
          // Select and check forexA position
          if (m_Position.Select(forexA)) {posTypeA = (ENUM_POSITION_TYPE)PositionGetInteger(POSITION_TYPE);}
          // Select and check forexB position
          if (m_Position.Select(forexB)) {posTypeB = (ENUM_POSITION_TYPE)PositionGetInteger(POSITION_TYPE);}
      
          // Determine if forexA is long and forexB is short, or vice versa
          bool isLongA  = (posTypeA == 0); //0 is buy
          bool isShortA = (posTypeA == 1); //1 is sell
          
          bool isLongB  = (posTypeB == 0); //0 is buy
          bool isShortB = (posTypeB == 1); //1 is sell
      
         //Cut loss conditions
         // Check if forexA is LONG and forexB is SHORT (mean reversion scenario)
          if (isLongA && isShortB && zScore > take_profit) {
              Close_Pair(forexA, forexB); //Mean reversion trigger
              Print("Take profit, Equity: ", myAccountEquity, ", Balance: ", myAccountBalance,", zScore is :",zScore);
          }
      
          // Check if forexA is SHORT and forexB is LONG (mean reversion scenario)
          else if (isShortA && isLongB && zScore < -1.0*take_profit) {
              Close_Pair(forexA, forexB); //Mean reversion trigger
              Print("Take profit, Equity: ", myAccountEquity, ", Balance: ", myAccountBalance,", zScore is :",zScore);
          }
          // Rolling equity take profit, don't need to wait till zscore becomes 0'
          else if (myAccountEquity < take_profit_cutoff) {
              Close_Pair(forexA, forexB); //Mean reversion trigger
              Print("Take profit, Equity rolling limit hit: ", myAccountEquity, ", Balance: ", myAccountBalance,", zScore is :",zScore);
              Print("equity_cutoff_take_profit: ",take_profit_cutoff);
              if(zScore < 0){temp_pause = -1;}
              else if(zScore > 0){temp_pause = 1;}  
          }      
          // Rolling stop-loss: Cut loss if account equity falls too much
          else if (myAccountEquity < cutoff * myAccountBalance) {
              Close_Pair(forexA, forexB);
              Print("Hit Stop loss, closing pair. zScore is :",zScore);
              if(zScore < 0){temp_pause = -1;}
              else if(zScore > 0){temp_pause = 1;}  
          }
      }
  }

void Close_Pair(string forex1,string forex2) {
   PositionSelect(forex1);
   double profit1 = PositionGetDouble(POSITION_PROFIT);
   PositionSelect(forex2); 
   double profit2 = PositionGetDouble(POSITION_PROFIT);
   if(profit2>profit1){
      m_Trade.PositionClose(forex2);
      m_Trade.PositionClose(forex1);
   }
   else {
      m_Trade.PositionClose(forex1);
      m_Trade.PositionClose(forex2);
   }
   take_profit_cutoff = -1.0 * myAccountBalance;
   max_profit = 0;
}

//void OnTimer()
//  {
//      //Evaluate Z score and send message to mobile every minute
//      myAccountEquity = AccountInfoDouble(ACCOUNT_EQUITY);
//      myAccountBalance = AccountInfoDouble(ACCOUNT_BALANCE); 
//      myAccountProfit = AccountInfoDouble(ACCOUNT_PROFIT);
//
//      string forexA = "AUDUSD";
//      string forexB = "NZDUSD";
//      
//      int err1=0;                                             //variable for storing the results of working with the indicator buffer
//      int err2=0;                                             //variable for storing the results of working with the price chart
//      
//      err1=CopyClose(forexA,my_timeframe,0,nPeriod+1,Close_bufA);      //copy data from the indicator array into the dynamic array iMA_buf for further work with them
//      err2=CopyClose(forexB,my_timeframe,0,nPeriod+1,Close_bufB);    //copy the price chart data into the dynamic array Close_buf for further work with them
//      if(err1<0 || err2<0)                                    //in case of errors
//      {
//         Print("Failed to copy data from the indicator buffer or price chart buffer");  //then print the relevant error message into the log file
//         return;                                                               //and exit the function
//      }
//      if(ArraySize(Close_bufA) < nPeriod || ArraySize(Close_bufB) < nPeriod)
//      {
//         Print("Failed to copy data or insufficient data. err1:", err1, " err2:", err2);
//         return;
//      }
//            
//      double sum_ab, sum_b2; /*for calculating integration factor k*/
//      sum_ab = 0;
//      sum_b2 = 0;
//      
//      for(int i=0; i < nPeriod; i++) {
//         sum_ab = sum_ab + Close_bufA[i]*Close_bufB[i];
//         sum_b2 = sum_b2 + Close_bufB[i]*Close_bufB[i];
//      }
//      
//      kfactor = sum_ab/sum_b2;
//         
//      double spread[]; /*for calculating Z-score of spread*/   
//      ArrayResize(spread,nPeriod);
//      
//      for(int i=0; i < nPeriod; i++) {
//         spread[i] = Close_bufA[i]-kfactor*Close_bufB[i];
//      }
//      
//      zScore = (spread[0]-MathMean(spread))/MathStandardDeviation(spread);
//      
//      string message = "forex time interval: " + EnumToString(my_timeframe) + "\n" +
//               "K-factor is: " + kfactor + "\n" +
//               "Z-score is: " + zScore + "\n";
//      SendNotification(message);
//  }
   
double GetSafeLotSize(string symbol, double riskPercentage, double avail_leverage) {
   double freeMargin = AccountInfoDouble(ACCOUNT_FREEMARGIN);
   double lotStep = SymbolInfoDouble(symbol, SYMBOL_VOLUME_STEP);
   double minLot = SymbolInfoDouble(symbol, SYMBOL_VOLUME_MIN);
   double maxLot = SymbolInfoDouble(symbol, SYMBOL_VOLUME_MAX);
   double contractSize = SymbolInfoDouble(symbol, SYMBOL_TRADE_CONTRACT_SIZE);
    
   // Risk as percentage of free margin
   double riskAmount = freeMargin * (riskPercentage / 100.0);
    
   // Approximate lot size based on available risk amount
   double lotSize = riskAmount / (contractSize / avail_leverage);

   // Ensure the lot size follows broker’s constraints
   lotSize = MathMax(minLot, MathMin(maxLot, lotSize));  // Keep within min-max
   lotSize = MathFloor(lotSize / lotStep) * lotStep;     // Normalize to lot steps

   return lotSize;
}  
 
joeytian:

I'm having trouble backtesting and running genetic optimization on this code on every tick as it runs very slowly. My current setup is AMD 7950x with 64gb of DDR5 ram. The computer crashes when i try to run optimizations. I have tried to disable the number of agents to half but now it runs painfully slow. 

Can anyone suggest any code optimizations?

Hi joey, 

on first sight I would ask if it's necessary to run the entire operations on every single tick? An interval approach running every x minutes would speed things up a lot. 

And regarding the agents, it's enough to disable just two of them to keep the computer manageable. I'm using a 5950x with 128 GB RAM and optimize with 30 out of 32 cores. 

Hope this helps to speed things up.

Best regards

Daniel

 
joeytian:

I'm having trouble backtesting and running genetic optimization on this code on every tick as it runs very slowly. My current setup is AMD 7950x with 64gb of DDR5 ram. The computer crashes when i try to run optimizations. I have tried to disable the number of agents to half but now it runs painfully slow. 

Can anyone suggest any code optimizations?

In the editor run Debug => run profiling with hist. data

Read: https://www.mql5.com/en/forum/7144

Example of MQL5 code profiling
Example of MQL5 code profiling
  • 2012.07.13
  • MetaQuotes
  • www.mql5.com
In build 674 we have introduced the new feature of MQL5 code profiling...
  
      int rsiHandle = iRSI("AUDNZD", PERIOD_D1, 14, PRICE_CLOSE);
      if (rsiHandle != INVALID_HANDLE){
         if (CopyBuffer(rsiHandle, 0, 0, 1, RSIBuffer) > 0){
            rsiValue = RSIBuffer[0];
         }
         IndicatorRelease(rsiHandle);
      }

Perhaps you should read the manual, especially the examples.
   How To Ask Questions The Smart Way. (2004)
      How To Interpret Answers.
         RTFM and STFW: How To Tell You've Seriously Screwed Up.

They all (including iCustom) return a handle (an int). You get that in OnInit. In OnTick/OnCalculate/OnStart (after the indicator has updated its buffers), you use the handle, shift and count to get the data.
          Technical Indicators - Reference on algorithmic/automated trading language for MetaTrader 5
          Timeseries and Indicators Access / CopyBuffer - Reference on algorithmic/automated trading language for MetaTrader 5
          How to start with MQL5 - General - MQL5 programming forum - Page 3 #22 (2020)
          How to start with MQL5 - MetaTrader 5 - General - MQL5 programming forum - Page 7 #61 (2020)
          MQL5 for Newbies: Guide to Using Technical Indicators in Expert Advisors - MQL5 Articles (2010)
          How to call indicators in MQL5 - MQL5 Articles (2010)

Stop releasing the indicator; you are creating your own problem.
New comment