MT5 and speed in action - page 20

New comment
 
fxsaber:

There may be a faster option. But one step to the left in the condition of what needs to be counted, and the logic may have to change considerably. Not easy, in general.

CHashMap<ulong, ulong> DealsIn;  // По PositionID возвращает DealIn.


It's not caching, it's indexing. Here is caching (part of code):

class DealInfo {
public:
  datetime Closed;
  ulong Ticket;
  ulong Position;
  string Symbol;
  long Type;
  long Reason;
  double Volume;
  double Price;
  double Profit;
  double Swap;
  long Magic;
};

class DealsComparer : public IComparer<DealInfo*> {
  int Compare(DealInfo* x, DealInfo* y) {
    int res = (int)(x.Closed - y.Closed);
    if (res == 0) {
      res = (int)(x.Ticket - y.Ticket);
    }
    
    return res;
  }
};

CArrayList<DealInfo*> dealsHistory;

inline bool UpdateDeals(CArrayList<DealInfo*> &deals, datetime &lastUpdated) {
  DealInfo* dealsToAdd[];
  DealsComparer comparer;
  int toAdd = 0;
  DealInfo* deal;
  
  if (!HistorySelect(lastUpdated, TimeLocal() + 12*3600)) {
    return false;
  }

  for (int i = 0, total = HistoryDealsTotal(); i < total; i++) {
    DealInfo tmp;
    ulong ticket = HistoryDealGetTicket(i);
    if (ticket == 0) continue;
    
    datetime dt = (datetime)HistoryDealGetInteger(ticket, DEAL_TIME);
    if (lastUpdated < dt) {
      lastUpdated = dt;
    }

    if (HistoryDealGetInteger(ticket, DEAL_ENTRY) != DEAL_ENTRY_OUT) continue;

    ulong reason = HistoryDealGetInteger(ticket, DEAL_REASON);

    tmp.Ticket = ticket;
    tmp.Closed = dt;
    int idx = deals.BinarySearch(&tmp, &comparer);
    if (idx >= 0 && deals.TryGetValue(idx, deal) && deal != nullptr && deal.Ticket == ticket)
      continue;

    deal = new DealInfo;
    deal.Ticket = ticket;
    deal.Closed = dt;
    deal.Position = HistoryDealGetInteger(ticket, DEAL_POSITION_ID);
    deal.Symbol = HistoryDealGetString(ticket, DEAL_SYMBOL);
    deal.Type = HistoryDealGetInteger(ticket, DEAL_TYPE);
    deal.Reason = HistoryDealGetInteger(ticket, DEAL_REASON);
    deal.Volume = HistoryDealGetDouble(ticket, DEAL_VOLUME);
    deal.Price = HistoryDealGetDouble(ticket, DEAL_PRICE);
    deal.Swap = HistoryDealGetDouble(ticket, DEAL_SWAP);
    deal.Profit = HistoryDealGetDouble(ticket, DEAL_PROFIT);
    deal.Magic = HistoryDealGetInteger(ticket, DEAL_MAGIC);
    
    ArrayResize(dealsToAdd, toAdd + 1, total);
    dealsToAdd[toAdd++] = deal;
  }
  
  if (toAdd > 0) {
    deals.AddRange(dealsToAdd);
    deals.Sort(&comparer);
  }
  
  return (toAdd > 0);
}

The code was written hastily and there is a lot to refine, given the frequent ArrayResize, but it exactly updates the cache, sorted by Closed. If you want to search later, use your own index. But you only have to update a small part each time.
I don't remember why"12*3600" is there, I don't think all deals were issued to me.

 
Andrey Pogoreltsev:

It's not caching, it's an index.

Please read carefully.

Forum on trading, automated trading systems and trading strategy testing

MT5 and Speed in Action

fxsaber, 2020.08.28 21:10

Pure MQL5 is 100x slower than partial (only HistorySelectByPosition) caching.

Architecturally do full caching - you need to think carefully. There are many pitfalls.

Here is caching (part of code):

The code was written hastily and has some things to improve considering frequent ArrayResize, but it updates cache sorted by Closed. You want to search later, use your own index. But you only have to update a small part each time.

This is just an example of a head-on history saving without the real-world gimmicks. Even in MT4Orders partial caching is done with a five second margin...

I don't remember why"12*3600" is there, I don't think all trades were issued to me.

Always set INT_MAX.

 
fxsaber:

Architecturally, doing full caching takes a lot of thinking. There are a lot of pitfalls.

There's really nothing complicated there. You can do a sampling with hours to spare, for example, for all orders that might appear in history retroactively (which is very strange, by the way).

This is just an example of head-on history saving without any real-time gimmicks. Even in MT4Orders partial caching is done with a five second margin...

Always set INT_MAX.

I don't understand the point of the shift. As if there is a current timestamp, I want to get relative to it and it's strange that you have to specify a time that doesn't exist. I kind of want a logical explanation.

My cache, by the way, works on real accounts with 10k+ trades.

And the main brakes in the code so far are the network functions.

 
Just passing by, I thought I would summarise what has already been hinted at here. If you need superfast performance at any cost, it is achieved traditionally - at the expense of other resources consumption, e.g. memory. In other words, it is not necessary to run a loop through the entire history every time when it is required to calculate some aggregate value (can be not just the total duration), and calculate all in advance in the "array" (or other optimized data structure) and then only augment. And a request for a value by position identifier (or whatever parameter, by which the breakdown is done) will degenerate into a memory cell reference. In the most general form - it's something like OLAP (although in my implementation is not finished online update), but for a particular problem can be done and simpler.
 

In the latest beta 2588, the HistorySelect feature is very well cached and almost always (except for the first time) is free.

We are likely to make a number of other improvements by the release.



As I explained earlier, in MT5 there is no extra cost in automatically creating market snapshots for each EA before each event as it was done in MT4. This reduces delays and allows the robots to run faster. Each developer asks exactly what he needs.

Therefore, you should clearly understand that the approach of "calling HistorySelect on the whole history, and then immediately make another selection of HistorySelectByPosition" will kill the previously created caches of history. This is a shot in the foot.


After the release we will start a big work on adding new more efficient MQL5 functions and open native order/trade data structures, so we can simplify and speed up algotrading.

 
Renat Fatkhullin:

In the latest 2588 beta, the HistorySelect function is very well cached and almost always (except the first time) is free.

#include <fxsaber\Benchmark.mqh> // https://c.mql5.com/3/321/Benchmark.mqh

input int inAlertTime = 1; // Нижний порог в миллисекундах

#define _B2(A) _B(A, inAlertTime)
#define  ALERT(A) Alert(#A + " = " + (string)(A))

void OnInit()
{
  if (HistorySelect(0, INT_MAX))
  {
    ALERT(HistoryDealsTotal());
    ALERT(HistoryOrdersTotal());
  }
}

void OnTick()
{
  static int i = 0;
  
  ALERT(i++);
  
  _B2(HistorySelect(TimeCurrent(), INT_MAX));
  _B2(HistorySelect(0, INT_MAX));
}


Result.

2020.09.01 22:56:46.089 Test6 (EURAUD,M1)       Alert: HistoryDealsTotal() = 9435
2020.09.01 22:56:46.089 Test6 (EURAUD,M1)       Alert: HistoryOrdersTotal() = 12529
2020.09.01 22:56:46.575 Test6 (EURAUD,M1)       Alert: i++ = 0
2020.09.01 22:56:46.579 Test6 (EURAUD,M1)       Alert: Time[Test6.mq5 25: HistorySelect(0,INT_MAX)] = 3 ms.
2020.09.01 22:56:47.424 Test6 (EURAUD,M1)       Alert: i++ = 1
2020.09.01 22:56:47.428 Test6 (EURAUD,M1)       Alert: Time[Test6.mq5 25: HistorySelect(0,INT_MAX)] = 3 ms.
2020.09.01 22:56:47.765 Test6 (EURAUD,M1)       Alert: i++ = 2
2020.09.01 22:56:47.768 Test6 (EURAUD,M1)       Alert: Time[Test6.mq5 25: HistorySelect(0,INT_MAX)] = 3 ms.
2020.09.01 22:56:47.902 Test6 (EURAUD,M1)       Alert: i++ = 3
2020.09.01 22:56:47.906 Test6 (EURAUD,M1)       Alert: Time[Test6.mq5 25: HistorySelect(0,INT_MAX)] = 3 ms.
2020.09.01 22:56:48.453 Test6 (EURAUD,M1)       Alert: i++ = 4
2020.09.01 22:56:48.456 Test6 (EURAUD,M1)       Alert: Time[Test6.mq5 25: HistorySelect(0,INT_MAX)] = 3 ms.
2020.09.01 22:56:48.516 Test6 (EURAUD,M1)       Alert: i++ = 5
2020.09.01 22:56:48.521 Test6 (EURAUD,M1)       Alert: Time[Test6.mq5 25: HistorySelect(0,INT_MAX)] = 4 ms.


On every tick there is a problem.


ZY installed Win10, LatencyMon shows that everything is fine.

 
At the moment I see that in 99% of cases only HistorySelect(0, INT_MAX) should be used. Try not to use other options.
 
Renat Fatkhullin:

After the release we will start a lot of work to add new more efficient MQL5 functions and open up native order/trade data structures so that algorithmic trading can be simplified and accelerated.

MqlDeal, MqlOrder and MqlPosition would be great. It might even become simpler.

 
fxsaber:
At the moment, I see that in 99% of cases we should only use HistorySelect(0, INT_MAX). Try not to use the other options.

If I have hundreds of thousands of orders in my history, will it also be faster than taking the last minute history?

So do I have to go through all of this history? It does not make sense.

 
Dmi3:

If I have hundreds of thousands of orders in my history, is it also faster than taking the last minute history?

Left only 0-INT_MAX variant in combat robots. Stopped noticing the lags.

1...131415161718192021222324252627...94
New comment
Reason: