HELP: processing High Frequency data

New comment
 

I want to develop an EA to process live high frequency data (including ticks and order book events). but in first step I encountered problems in extracting data.

for start I tried to get tick data and save them to a file to check quality of data . here is the code:

 

 datetime dtm = TimeTradeServer();
 string tick[7];
  int i=0;

  string tickfile_handle;
   string tickfilename = Symbol()+"_tick_"+TimeToString(dtm,TIME_DATE)+".txt";


//----------------------------------------------------------------------------
  void OnInit()
  {
 tickfile_handle=FileOpen(tickfilename,FILE_WRITE|FILE_TXT|FILE_READ);
  }
//----------------------------------------------------------------------------


  void OnTick()
{

   MqlTick last_tick;
 
 
  if (SymbolInfoTick(Symbol(),last_tick))
  {
     datetime stm = TimeTradeServer();
  
      i=i+1;
  
      tick[0]=i;
      tick[1]=TimeToString(stm,TIME_SECONDS);
      tick[2]=TimeToString(last_tick.time,TIME_SECONDS);
      tick[3]=last_tick.last;
      tick[4]=last_tick.volume;
      tick[5]=last_tick.ask;
      tick[6]=last_tick.bid;
  


    
    FileSeek(tickfile_handle,0,SEEK_END);
     FileWriteArray(tickfile_handle,tick,0,WHOLE_ARRAY);

   }
   
 }
//---------------------------------------------------------------------------


void OnDeinit()
  {

     FileClose(tickfile_handle);
  
  }

 

       after running the EA for one trading day I opened the text file and compared it to data that I had got from exchange website. there are tow problems:

1. when the transaction traffic is high and there are more than 2 tick events in a second, the EA could save only one or tow events and deny the rest.

 I think I will encounter same problem using OnBookEvent handler to get book data . 

 if this is because of CPU or network limits to process data in such frequency I have an idea to solve first problem . using OnTimer event handler and in every second get last one second tick events (remember I need to process high frequency live data).

but I dont know how and I dont know if this solution works for book event, because I think despite of tick data, server dont save order book event so there is no historical data to extract for last second  .

 

2. I need only data when transaction occurs not when quote changes , but OnTick event handler consider both. how can I write a code to distinguish and ignore quote changes data.


.

 
haqiqi.afshin:

I want to develop an EA to process live high frequency data (including ticks and order book events). but in first step I encountered problems in extracting data.

for start I tried to get tick data and save them to a file to check quality of data . here is the code:

after running the EA for one trading day I opened the text file and compared it to data that I had got from exchange website. there are tow problems:

1. when the transaction traffic is high and there are more than 2 tick events in a second, the EA could save only one or tow events and deny the rest.

 I think I will encounter same problem using OnBookEvent handler to get book data . 

 if this is because of CPU or network limits to process data in such frequency I have an idea to solve first problem . using OnTimer event handler and in every second get last one second tick events (remember I need to process high frequency live data).

but I dont know how and I dont know if this solution works for book event, because I think despite of tick data, server dont save order book event so there is no historical data to extract for last second  .

2. I need only data when transaction occurs not when quote changes , but OnTick event handler consider both. how can I write a code to distinguish and ignore quote changes data..

Hello haqiqi.afshin,

please take a look at this post on the Portuguese section of the forum. It's directly related to this situation. The only problem: it's in Portuguese.

Maybe it's the situation you're facing and you might come to the same conclusions as those on the related topic.

I hope it helps.

 

haqiqi.afshin:

...

2. I need only data when transaction occurs not when quote changes , but OnTick event handler consider both. how can I write a code to distinguish and ignore quote changes data.

Isn't a quote's change always resulting from a transaction ? Probably it depends of what you are calling a transaction.
 

I use this to record ticks (well the actual code records ticks from several symbols at the same time).

So you will loose only a few ticks every when the file writes once in an hour (or less)

struct OneTick
  {
   MqlTick           tick;
  };
//+------------------------------------------------------------------+
//|                                                                  |
//+------------------------------------------------------------------+
struct TickArray
  {
   OneTick           ticks[20000];
  };
//+------------------------------------------------------------------+
//|                                                                  |
//+------------------------------------------------------------------+
struct TickContainer
  {
   TickArray         allticks;
   int               ticksrecorded;
  };
//+------------------------------------------------------------------+
//|                                                                  |
//+------------------------------------------------------------------+
class CMultiTick
  {

public:
   int               file_handle;
   TickContainer     bticks;

   CMultiTick() 
     {
      file_handle = INVALID_HANDLE;
     }

   void CMultiTick:: ~CMultiTick()
     {
      SetFile(INVALID_HANDLE);
     }

   void  CMultiTick::AddTick(const MqlTick &theTick)
     {
      AddTick(bticks,theTick);
     }

   void  CMultiTick::AddTick(TickContainer &container,const MqlTick &theTick)
     {
         container.allticks.ticks[container.ticksrecorded].tick=theTick;
         if (container.ticksrecorded==20000) 
            WriteTicksToFile(); 
         else
            container.ticksrecorded++;
     }

   void  CMultiTick::WriteTicksToFile()
     {        
         if (file_handle!=INVALID_HANDLE)
         {
              int anz = bticks.ticksrecorded;
              if (anz>0)
              {
                 FileWriteStruct(file_handle,bticks.allticks,sizeof(OneTick)*anz);
                  bticks.ticksrecorded=0;
                  Print(anz," ticks written to file. ",sizeof(OneTick)*anz);
              }
         }
    }



   void  CMultiTick::SetFile(int fhandle)
     {
      if(file_handle!=INVALID_HANDLE)
         FileClose(file_handle);
      file_handle=fhandle;
     }
  };
 
ugo58:

I use this to record ticks (well the actual code records ticks from several symbols at the same time).

So you will loose only a few ticks every when the file writes once in an hour (or less)

You will ALWAYS lose ticks, independently of your code. See link provided by Malacarne.
 
angevoyageur:
You will ALWAYS lose ticks, independently of your code. See link provided by Malacarne.

I know but I checked that I got all ticks to make synthetic bars which had the same value than the ones from the broker.

Anyway I recorded 21 symbols at the same time and it worked quite well for me.

 
Malacarne:

Hello haqiqi.afshin,

please take a look at this post on the Portuguese section of the forum. It's directly related to this situation. The only problem: it's in Portuguese.

Maybe it's the situation you're facing and you might come to the same conclusions as those on the related topic.

I hope it helps.

Hi Malacane 

thank you.I read whole posts in that topic , it helped a lot (thanks google translate ;)  ) . I hoped to find a solution to get all ticks perfectly. but apparently Its not easy. I use meta trader for stock market that is centralized . I compared volumes indicator "tick volume " with exchange "tick volume" data and they were equal. does this indicator count every ticks? . if so this indicator does not loose any tick !!!

 or maybe it get the data separately .if there is not a way to get whole tick by OnTick Event handler I,m going to find a way to modify data by whatever volumes indicator uses for its source. 

 thanks so much.  

 
haqiqi.afshin:

Hi Malacane 

thank you.I read whole posts in that topic , it helped a lot (thanks google translate ;)  ) . I hoped to find a solution to get all ticks perfectly. but apparently Its not easy. I use meta trader for stock market that is centralized . I compared volumes indicator "tick volume " with exchange "tick volume" data and they were equal. does this indicator count every ticks? . if so this indicator does not loose any tick !!!

 or maybe it get the data separately .if there is not a way to get whole tick by OnTick Event handler I,m going to find a way to modify data by whatever volumes indicator uses for its source. 

 thanks so much.  

Volume is only a volume (tick or real), of course you get the right volume. But you will ALWAYS lose some ticks with MT5, it's not possible to get all the data of each tick.
 
angevoyageur:
Isn't a quote's change always resulting from a transaction ? Probably it depends of what you are calling a transaction.

I think that was not clear. let me say this way : when new order arrives or someone modify or cancel his order the quote changes when a market order arrives quote changes too.
 So I want to filter ticks resulted by market order . I tried to compare every ticks with tick before it . if ask and bid were equal then its a market order. it works but looses some data .(ticks that are saved by last code and are market order)

 datetime dtm = TimeTradeServer();
 string tick[7];
  int i=0;

  string tickfile_handle;
   string tickfilename = Symbol()+"_tick_"+TimeToString(dtm,TIME_DATE)+".txt";

MqlTick befortick;


int OnInit()
  {
   tickfile_handle=FileOpen(tickfilename,FILE_WRITE|FILE_TXT|FILE_READ);
   SymbolInfoTick(Symbol(),befortick);
   return(INIT_SUCCEEDED);
  }

void OnTick()
  {

   MqlTick last_tick;
   if(SymbolInfoTick(Symbol(),last_tick))
     {
        if(last_tick.ask==befortick.ask && last_tick.bid==befortick.bid)
        {
          datetime stm = TimeTradeServer();
  
          i=i+1;
  
          tick[0]=i;
          tick[1]=TimeToString(stm,TIME_SECONDS);
          tick[2]=TimeToString(last_tick.time,TIME_SECONDS);
          tick[3]=last_tick.last;
          tick[4]=last_tick.volume;
          tick[5]=last_tick.ask;
          tick[6]=last_tick.bid;
  


    
          FileSeek(tickfile_handle,0,SEEK_END);
          FileWriteArray(tickfile_handle,tick,0,WHOLE_ARRAY);

        }  
     }


      befortick=last_tick;

  }


 
 
haqiqi.afshin:

I think that was not clear. let me say this way : when new order arrives or someone modify or cancel his order the quote changes when a market order arrives quote changes too.
 So I want to filter ticks resulted by market order . I tried to compare every ticks with tick before it . if ask and bid were equal then its a market order. it works but looses some data .(ticks that are saved by last code and are market order)

Independently of the tick's loss, I don't think what you wrote is right. A market order can result in a change of price (bid or ask) if the volume of this order is greater than the available volume at market price.
New comment
Reason: