Multi-Symbol Tick-event framework.

New comment
 

I wanted to start a thread to discuss and develop an idea that I've been conceptualizing; a method to eliminate perpetual loops when working with multiple symbols in one EA. The idea is rather simple.. 

  1. EA has a list of symbols to use for signal calculations.
  2. EA opens a chart for each symbol. 
  3. EA applies a template to each chart with an Event Handler Expert(EHE)
  4. EA registers itself to a list of receivers for tick event on that symbol and saves to file.
  5. EHE loads a list of subscribed EA to update from file.
  6. EHE pushes a new-tick event to all registered charts using EventCustom

This could also be a two way communication in order to manage asynchronous orders

  1. EA wants to place a basket of trades.
  2. EA sends trade details via EventCustom to EHE
  3. EHEs place trades in an asynchronous fashion. 

So my questions are:

  • Does anything like this already exist? 
  • Does anyone have any suggestions to make this better?

 

Why is EventChartCustom is so slow? I performed an experiment that measures the latency (ping) of EventChartCustom and my avg ping is 10ms! Can somebody else test and verify?


Ping logic: The "sender" generates a custom chart event and passes it's chart id to the "receiver", and the receiver instantly generates a custom event in return. 

Sender

ulong g_time;

void OnTick()
{
   for(long ch=ChartFirst();ch>=0;ch=ChartNext(ch))
   {
      if(ch == ChartID())
         continue;
      g_time = GetMicrosecondCount();
      if(!EventChartCustom(ch,EVENT_ONTICK,ChartID(),0.0,_Symbol))
         __ERROR__
      else 
         break;
   }
}
//+------------------------------------------------------------------+
void OnChartEvent(const int id,
                  const long &lparam,
                  const double &dparam,
                  const string &sparam)
{
   static ulong ping=0, max=0, min=LONG_MAX, cnt=0, avg=0;
   if(id == CHARTEVENT_CUSTOM+EVENT_ONTICK)
   {
      ping = GetMicrosecondCount()-g_time;
      if(ping > max)
         max = ping;
      if(ping < min)
         min = ping;
      avg+=ping;
      cnt++;
      
      printf("ping = %d, min = %d, max = %d, avg = %d",
             ping, min, max, avg/cnt);
   }
}

Reciever

void OnChartEvent(const int id,
                  const long &lparam,
                  const double &dparam,
                  const string &sparam)
{
   if(id == CHARTEVENT_CUSTOM+EVENT_ONTICK)
      if(!EventChartCustom(lparam,1,0,0.0,""))
         __ERROR__
}

NOTE: This is designed to work with a max of two open charts; one sender and one receiver. 

Files:
Receiver.mq4  2 kb
Sender.mq4  3 kb
 
Old idea.
 
Alain Verleyen:
Old idea.

Thanks for the article, I'll give it a read. Any idea why EventChartCustom is so slow?

 
Ok, I was able to narrow down the delay to something happening behind the scenes once OnTick() is finished, and whatever it is - it's tying up the expert thread for almost 10ms per tick. 
 
nicholishen:

Why is EventChartCustom is so slow? I performed an experiment that measures the latency (ping) of EventChartCustom and my avg ping is 10ms! Can somebody else test and verify?

hello,

it looks normal and it shall be because of the system timer. IIRC there should be a function in Windows API where we can set it to other values. Another way to test it would be to open a site with video, where it should be set to low values or, there are some tools like "Timer Tool". But TBH I do not like it when it goes below 15-10ms, as the gains are negligible, for no good reason usually

 
Demos Stogios:

hello,

it looks normal and it shall be because of the system timer. IIRC there should be a function in Windows API where we can set it to other values. Another way to test it would be to open a site with video, where it should be set to low values or, there are some tools like "Timer Tool". But TBH I do not like it when it goes below 15-10ms, as the gains are negligible, for no good reason usually

There is no timer in this code.
 
nicholishen:
Ok, I was able to narrow down the delay to something happening behind the scenes once OnTick() is finished, and whatever it is - it's tying up the expert thread for almost 10ms per tick. 

It's correlated to number of symbols in Market Watch. Custom events have low priority I guess.

2018.02.08 13:26:04.404    Sender USDCHF,H1: cnt = 2201, ping = 3055, min = 45, max = 16323, avg = 1300

 
Alain Verleyen:
There is no timer in this code.

But there are timers in the OS. Normally, when we are creating events, timers etc. (in the Windows API, where MT is obviously being done), we are committing  entries in a system queue. Of course, someone can bundle as much as many events as wanted and should expect that, given enough CPU power, will be executed in a single system cycle

Edit: Instead of printf, put that in the code where we can see it in real time

Alert("ping = ", ping, " min = ", min, " max = ", max, " avg = ", avg/cnt);

In my machine, I can see my global system timer is low and I get an average ping of 150microseconds

 
Alain Verleyen:

It's correlated to number of symbols in Market Watch. Custom events have low priority I guess.

2018.02.08 13:26:04.404    Sender USDCHF,H1: cnt = 2201, ping = 3055, min = 45, max = 16323, avg = 1300

I tested your theory and it did in fact reduce it a bit, however, I tested a feedback loop where EventCustoms call back to each-other and time averages a reasonable 50 micros. The take-away is that there's some major overhead in the behind-the-scenes post-processing of the OnTick function, therefore, if the goal is low latency asynchronous order processing then it's best to select the chart of a symbol with the lowest tick volume in order to reduce the chance of OnTick induced latency. 

 
Demos Stogios:

But there are timers in the OS. Normally, when we are creating events, timers etc. (in the Windows API, where MT is obviously being done), we are committing  entries in a system queue. Of course, someone can bundle as much as many events as wanted and should expect that, given enough CPU power, will be executed in a single system cycle. But, we have to wait for that to happen

The issue is the behind the scenes processing that the terminal does whenever OnTick is called which holds the thread until it is completed. So lets say a chart event comes in while OnTick is being called, what happens is OnTick called -> OnChartEvent wait for OnTick finish -> OnTick returns -> terminal does stuff -> now OnChartEvent is called.  

123
New comment
Reason: