Closing multiple positions according FIFO rule - page 2

[Miroslav Popov]

"Regarding using a fixed array size with one ArrayResize() call at the end vs. calling ArrayResize() on every iteration"

You are right about that. I'm calling ClosePositionas function frequently. It's better to make the 2D array fixed to let's say 500 and to resize it once.

[JC]

Miroslav_Popov:

You are right about that. I'm calling ClosePositionas function frequently. It's better to make the 2D array fixed to let's say 500 and to resize it once.

One of my personal bugbears... I think that MQL may be faster than you think it is. I've just tried the dynamic array size on a list of 20 open orders. Building the array and sorting it appears to take about 20 microseconds (0.02 milliseconds). Regardless of how fast the fixed array is, I'd regard that as trivially fast, and my very personal preference would be to regard an assumption about the maximum number of orders as unnecessary.

(EDIT: I'd imagine that MQL does chunked allocation of memory for arrays and therefore that, most of the time, ArrayResize() is simply updating a single integer recording the upper bound of the array, not causing a memory re-allocation and memory copy.)

[gordon]

jjc:

One of my personal bugbears... I think that MQL may be faster than you think it is. I've just tried the dynamic array size on a list of 20 open orders. Building the array and sorting it appears to take about 20 microseconds (0.02 milliseconds). Regardless of how fast the fixed array is, I'd regard that as trivially fast, and my very personal preference would be to regard an assumption about the maximum number of orders as unnecessary.

(EDIT: I'd imagine that MQL does chunked allocation of memory for arrays and therefore that, most of the time, ArrayResize() is simply updating a single integer recording the upper bound of the array, not causing a memory re-allocation and memory copy.)

I agree regarding using ArrayResize() for something like an order closing loop which might happen every thousands of ticks (or maybe tens or hundreds of thousands). But if, for example, u use ArrayResize() 20 times per tick (for something other than closing orders) and let's say on average your EA will close after about 10,000 ticks (which is not that much, most brokers have an average volume of about 30 for M1, so that's about ~6 hours of trading), and u optimize over 10,000 passes (which I personally do many times) then the time it takes:

20x10,000x10,000x20x10^-6=40000Sec=~11 Hours (!)

In this case, obviously a fixed array is a better solution. But as I said, an order closing loop does not get called that often, hence I would do the same.

[JC]

gordon:

20x10,000x10,000x20x10^-6=40000Sec=~11 Hours (!)

Leaving all other considerations aside, I think this calculation is assuming that the dynamic array is 20 microseconds slower than the fixed array. I don't know what the actual numbers are, but that's not what my testing says. The dynamic array takes 20 microseconds in my sample. The fixed array could take 19 microseconds.

EDIT: I've got the fixed-array version of the code taking about half the time (though this will obviously vary depending on the number of orders). I think that's therefore a saving of about 6 hours in your calculation. But I dread to think what that 6 hours would be a percentage of. If the code is needing to do the equivalent of 20 of these order-sorts per tick, then it would presumably be doing various other complex things as well, and it's potentially a 6-hour saving on what could be a multi-week test.

[gordon]

jjc:

Leaving all other considerations aside, I think this calculation is assuming that the dynamic array is 20 microseconds slower than the fixed array. I don't know what the actual numbers are, but that's not what my testing says. The dynamic array takes 20 microseconds in my sample. The fixed array could take 19 microseconds.

EDIT: I've got the fixed-array version of the code taking about half the time (though this will obviously vary depending on the number of orders). I think that's therefore a saving of about 6 hours in your calculation. But I dread to think what that 6 hours would be a percentage of. If the code is needing to do the equivalent of 20 of these order-sorts per tick, then it would presumably be doing various other complex things as well, and it's potentially a 6-hour saving on what could be a multi-week test.

Well, I think ultimately it depends on the specific EA and optimization circumstances. In some cases u might be right and the the time saved would be negligible relative to the total time, in others maybe not. Bottom line, I think both designs have advantages/disadvantages and should be used according to the specific EA and circumstances. But I agree that for the majority of cases your design would probably be preferable (I usually use dynamic arrays in my code as well...).

I do remember one very good example where a dynamic array affects performance to a point where it shouldn't be used... Don't know if u read the article HTML Walkthrough Using MQL4. The author writes a script to process HTML files (sort of like an extremely simplified semi-parser). His script has 2 bottlenecks when trying to process thousands of HTML tags - the first is recursive calls and the second is overuse of ArrayResize():

In the process of the script operation, calling function ArrayResize() tens, hundreds of thousands, or even millions of times will result in huge time wasting. Each dynamical resizing an array requires some time to allocate a new area of the necessary size in the PC memory and to copy the content of the old array into this new one. If we allocate an array of a rather large size in advance, we will be able to essentially reduce the time taken by these excessive operations.

BTW, MT4 Tester has an interesting bug where any usage of recursive calls will cause a memory buildup in the Tester. This memory is NOT freed when u press the 'Stop' button! Hence, optimizer (which simply does many tests), causes MT4 to choke the system and crash very fast. The strange thing is that memory is built up, regardless of if the recursive function is called or not.

[JC]

gordon:

Well, I think ultimately it depends on the specific EA and optimization circumstances.

I'm not for a second disagreeing with you there. There's pretty much nothing which can be advocated as "always do this". For example, in some previous topic or other, I've done some testing which shows that StringConcatenate() is slower than the + operator for any sort of text lengths which are likely to be encountered in the course of any normal EA operation.

Don't know if u read the article HTML Walkthrough Using MQL4.

I haven't read this before, but - from a very, very quick glance - the need to resize the array is escaping me. I can't see why you don't start search #2 from the end position for search #1, without modifying the array contents. But I'm probably missing something - I've only skim-read the article for a few seconds.

[gordon]

jjc:

I've done some testing which shows that StringConcatenate() is slower than the + operator for any sort of text lengths which are likely to be encountered in the course of any normal EA operation.

Interesting. I'll have a look at it. I am always hunting for ways to make my experts faster since they go through extensive optimizations...

Regarding the subject of dynamic/static array sizes... I think we have exhausted the subject. Was an interesting discussion.

[idht]

I modified the script and tested it using a demo account at an American broker.

It's for MetaTrader4. I added a dimension for volume to collect all info in one pass. Obviously, disclaimer: I dont know who wrote it if it does not work as you expected, lol.

#define INDEX_TIME 0
#define INDEX_TICKET 1
#define INDEX_LOTS 2

void ClosePositions(string symbol)
{//close all positions in the time order, complies with the FIFO rules
    int orderstotal = OrdersTotal();
    int posCounter      = 0;
    long ordticket[100][3];
    
    for (int i = 0; i < orderstotal; i++) 
    {
        if( !OrderSelect(i, SELECT_BY_POS, MODE_TRADES)) continue;//MODE_TRADES (default)- order selected from trading pool(opened and pending),
        
        if ( OrderSymbol() != symbol) continue; //OrderMagicNumber() != Expert_Magic || I dont need this
            
        int orderType = OrderType();
        if (orderType != OP_BUY && orderType != OP_SELL) continue;//is it to SKIP PENDING?
        
        ordticket[posCounter][INDEX_TIME] = OrderOpenTime();          // casting of datetime -> int, should work fine...
        ordticket[posCounter][INDEX_TICKET] = OrderTicket();
        ordticket[posCounter][INDEX_LOTS] = (long) ( 100 * OrderLots() ); //0.01 =>1 , 0.1 => 10
        posCounter++;
    }
    
    ArrayResize(ordticket,posCounter);   // we have to remove rest of array otherwise it's contents will be sorted as well
    ArraySort(ordticket);                // sorts ascending by first dimension -> by OrderOpenTime()

    for (int i = 0; i < posCounter; i++) 
    { 
        int ticket = (int) (ordticket[i][INDEX_TICKET]);
        double lots = (double) (ordticket[i][INDEX_LOTS]/100.0);
        Print( "Closing Position Ticket:" + IntegerToString( ticket ) + " Lots:" + DoubleToStr(lots) );
 
        if(! OrderClose(ticket,lots,Bid,3,Red) )
         {
            PrintFormat("%s %OrderClose error %d",symbol, GetLastError());  // if unable to send the request, output the error code
         } 
    }
}