Back-test slow - Will this speed it up?

New comment
 

Now, first off, I regard myself as being marginally higher than a novice in MQL4. So that means my knowledge stretches as far as to say that I appreciate the fact that the speed in which the back-test runs is based upon a few variables. I.e. efficiency in the written code, hardware and data (and in some cases, the build of MT4 = some bugs - albeit not in the latest one).

I have a decent PC to say the least. My CPU is a Haswell i5 4670K, SSD's, HDD's, 8 Gigs ram, GTX 780... bla bla bla, the works.

 My question is: will my back-testing run faster (ignoring the efficiency element) if I migrate my platform and data onto my SSD? 

At the moment, 22 runs in optimization mode takes 1 hour and 17 minutes. 

CPU cores look like this:

Highest core is running at approx 40% load with a max of 77% but this fluctuation changes rapidly. Meaning my CPU's are not stressed to max by any means. (video of my cores during backtest: http://screencast.com/t/7s21U2je0 )

 
DomGilberto:

Now, first off, I regard myself as being marginally higher than a novice in MQL4. So that means my knowledge stretches as far as to say that I appreciate the fact that the speed in which the back-test runs is based upon a few variables. I.e. efficiency in the written code, hardware and data (and in some cases, the build of MT4 = some bugs - albeit not in the latest one).

I have a decent PC to say the least. My CPU is a Haswell i5 4670K (24/7 stable OC'd to 4.2GHZ - stress test and benched) SSD's, HDD's, 8 Gigs ram, GTX 780... bla bla bla, the works.

 My question is: will my back-testing run faster (ignoring the efficiency element) if I migrate my platform and data onto my SSD? 

At the moment, 22 runs in optimization mode takes 1 hour and 17 minutes. 

CPU cores look like this:

Highest core is running at approx 40% load with a max of 77% but this fluctuation changes rapidly. Meaning my CPU's are not stressed to max by any means. 

1) are you using TickData (can be more?) or Mt4-Data.

2) SSD won't speead up a lot unless your EA is writing and reading to files as the test data are loaded at the begin of the backtest (as far as I know).

3) Improve your code: a) disable e.g. all the graphical stuff, b) are you always checking the open positions (OrderSelect() ...), ...

 
gooly:

1) are you using TickData (can be more?) or Mt4-Data.

2) SSD won't speead up a lot unless your EA is writing and reading to files as the test data are loaded at the begin of the backtest (as far as I know).

3) Improve your code: a) disable e.g. all the graphical stuff, b) are you always checking the open positions (OrderSelect() ...), ...

I WAS using 1 minute bars from <link removed by moderator> but it's even slower. Instead I am currently using the bars from Forex Tester 2 which I export > import. The quality is less but I can get a faster run through and as I am not scalping but instead trend following, it's not AS crucial. None the less, I'd rather use 1 minute bars from FXCM as I trade with them and the quality is pretty damn good tbh! The 1 minute bars are of incredible quality in comparison to what I've seen else-where for free. When I do comparison straight from live platform at FXCM, they're near enough identical.

3) I thought so about the OrderSelect(). The issue is, I pretty much have to check if last order hit it's stop per tick. I know this may sound dumb, but I need to know immediately if the trade hit it's stop. The only way I can think of doing this is by checking the history through a for loop. If there is a less intensive way of doing it and knowing whether the LAST trade (and only the last trade on this pair) hit it's stop immediately, I am all ears. Else, as I said, I am using for loops under OnTick(): like this:

//+------------------------------------------------------------------+
//| Check History to see if trade hit stop                           |
//+------------------------------------------------------------------+
bool CheckHistory()
{
 RefreshRates();
 
 bool isCloseBySL;
 int lastTrade=-1;
 // Find latest Trade, this assumes, last OrderID is last ticket 
 for(int i=0;i<OrdersHistoryTotal();i++)
      if(OrderSelect(i,SELECT_BY_POS,MODE_HISTORY)) 
          if (OrderTicket()>lastTrade) lastTrade=OrderTicket();      
 
 // Now lets load this last Trade and decide if it hit SL
 if (lastTrade>-1 && OrderSelect(lastTrade,SELECT_BY_TICKET,MODE_HISTORY))
   {
      if( OrderMagicNumber() == MagicNumber1  )
        if(OrderSymbol() == Symbol())
        {
                 

        if( OrderType() == OP_BUY )
            {
             double OCP = OrderClosePrice();
             
             isCloseBySL = MathAbs( OCP - OrderStopLoss() ) < MathAbs( OCP - OrderTakeProfit() );
             if( OrderStopLoss() >= OrderClosePrice() && isCloseBySL )
               {
               HistoryBarTime = OrderCloseTime();
               return(true);
               }
            } 

        if( OrderType() == OP_SELL )
            {
            double OCP = OrderClosePrice();
            
            isCloseBySL = MathAbs( OrderStopLoss() - OCP ) < MathAbs( OrderTakeProfit() - OCP );
            if( OrderClosePrice() >= OrderStopLoss() && isCloseBySL)
               {
               HistoryBarTime = OrderCloseTime();
               return(true);
               }
            } 
        }
   } 
HistoryBarTime = 0; 
return(false); 
}

That is the ONLY thing, that is done on each tick... Everything else is done under the premise that IsNewCandle():

//+------------------------------------------------------------------+   
//| Ensuring its a new candle function                               |
//+------------------------------------------------------------------+   
bool IsNewCandle()
   {
      if(Bar1Time == iTime(NULL, lowTF, 0))  
      return(false);
      
      Bar1Time = iTime(NULL, lowTF, 0);return(true);
   }
 

1) if you are using various tf:

if(Bar1Time == iTime(NULL, lowTF, 0))

then the hd-speed could matter as this accesses the hst-files?

2) you can try to check SL against a variable (global) instead of OrderSelect() ...

 
gooly:

1) if you are using various tf:

then the hd-speed could matter as this accesses the hst-files?

2) you can try to check SL against a variable (global) instead of OrderSelect() ...

Right, well I am using multiple time-frames constantly checking on each IsNewCandle(). I will migrate over to SSD and report back if there was any difference. 

I'll upload a broad speed comparison test video of before and after. 

2) you can try to check SL against a variable (global) instead of OrderSelect() ... 

Problem with this though, is that I cannot be for SURE whether or not the trade was actually closed at stop loss. I assume you mean by monitoring the Bid and Ask to see if it was >= SL level? 

 
Ok - so there is absolutely no difference between whether or not it is on SSD or HDD...

At least that is confirmed now lol!
 
DomGilberto:
Ok - so there is absolutely no difference between whether or not it is on SSD or HDD...

At least that is confirmed now lol!


I tried running with my price data in memory using a RAM disk and it had no appreciable difference.  You can make the biggest gains by looking at your code . . . don't do stuff every tick that you only need to do once per bar,  don't do stuff once every bar that you can do less frequently than once per bar . . .  if you haven't read this thread then it's worth a look to get an idea of what can be done with badly designed code:  https://www.mql5.com/en/forum/144240

Edit:  just seen from your other thread that you have read that thread . . . 

 

Oh that's interesting. What was the last thing you posted in that thread? Regarding strings? I compare strings?

 Basically, to keep things simple without getting too deep into my entire code, I need to know immediately that the last open trade hit it's stop or not. If I do not know this, then CheckForMaTrade() could push out another trade when it should infact wait.

I can't think of a way to not do this other than on each tick...? 

//+------------------------------------------------------------------+
//| Expert tick function                                             |
//+------------------------------------------------------------------+
void OnTick()
 {
  CheckHistory();//Check if last Market Order (this pair) hit stop (HistoryBarTime stores OrderCloseTime())

   if( IsNewCandle() )//Is it a new H1 candle? 
      {  
       if( Bar1Time > HistoryBarTime )//Did the HistoryBarTime come first between these two? 
         {
         CheckForMaTrade();//If yes, then check trade bias string to see if new trade should be placed.
         } 
      }    

   if( OpenOrdersThisPair(Symbol()) > 0 )
      {
      MA_Trail(); 
      Move_To_BreakEven();
      Check_Position_Parameters();
      }  


    if( Bar1Time > HistoryBarTime && OpenOrdersThisPair(Symbol()) == 0 && 
         H1_Buy_Touch == "H1 Buy Touch" && Ask > Stored_BuyPrice + Point )
         {
         OrderEntry(0); // Market Order Buy
         }
         
         
    if(Bar1Time > HistoryBarTime && OpenOrdersThisPair(Symbol()) == 0 && 
        H1_Sell_Touch == "H1 Sell Touch" && Stored_SellPrice > Bid + Point )
         {
         OrderEntry(1);// Market Order Sell
         }
            
 }
 
DomGilberto:

Oh that's interesting. What was the last thing you posted in that thread? Regarding strings? I compare strings?

Had to refresh my memory . . . it was about 18 months ago.

 

Instead of this . . .

void getFisherTransformValue(){
   getFisherTransform_Value="N/A";
   fClose0=iCustom(NULL,Timeframe,"fisher",Length,Price,NumBars,0,FTbar);
   fClose1=iCustom(NULL,Timeframe,"fisher",Length,Price,NumBars,1,FTbar);
   if(fClose0<fClose1){getFisherTransform_Value="BUY";}   //buy
   if(fClose0>fClose1){getFisherTransform_Value="SELL";}} //sell

and . . . 

   if(getHA_Value=="BUY"&&getFisherTransform_Value=="BUY"&&Signl!="BUY"){

 

Do this . . .

#define BUY   0
#define SELL  1
#define NA    2

and . . . 

void getFisherTransformValue()
   {
   getFisherTransform_Value = NA;
   fClose0=iCustom(NULL,Timeframe,"fisher",Length,Price,NumBars,0,FTbar);
   fClose1=iCustom(NULL,Timeframe,"fisher",Length,Price,NumBars,1,FTbar);
   if(fClose0 < fClose1)
      {
      getFisherTransform_Value = BUY;  //buy
      } 
   
   if(fClose0 > fClose1)
      {
      getFisherTransform_Value = SELL;//sell
      }
   } 
  

 and . . .

   if(getHA_Value == SELL && getFisherTransform_Value == SELL && Signl != SELL)
 
DomGilberto:


 Basically, to keep things simple without getting too deep into my entire code, I need to know immediately that the last open trade hit it's stop or not. If I do not know this, then CheckForMaTrade() could push out another trade when it should infact wait.

I can't think of a way to not do this other than on each tick...? 

 

Don't do it all if you are not waiting for an open trade to close . . . do you check for that ?  if you have already determined that the last closed trade did or didn't hit the SL then don't do it again,  do you check for that ?
 
RaptorUK: Do this . . .
#define BUY   0
#define SELL  1
#define NA    2
 Or now with enumerations
enumeration eDirection = { BUY, SELL, NA }
eDirection getFisherTransform_Value = NA;
:
  getFisherTransform_Value = BUY;
12
New comment
Reason: