Best way to mine tick data from 70+ instruments simultaneously?

To add comments, please log in or register
nadiawicket
963
nadiawicket  

What is the best (no chokes 28 simultaneous forever) way to mine both the Bid and Ask data provided by my broker on MT4 and store it in a file for further analysis? I don't need timestamps (are a plus though if possible with not lagging the thing to not working status) or anything that has to do with candles/ high lows (candle high lows closes and opens really not necessary/ will need to edit out). I'm not looking for History center bar stuff, just the Bid and Ask quotes for all 70 instruments from a single MT4 terminal.

I was hoping there was a TickFile somewhere in MT4 for every instrument that just had this data.

Wine chokes up when write to file more than 10 files more or less. I have to admit I am no expert coder however my previous experience with write to file function on more than 2-3 things is a choke fest on Wine Debian Stretch XFCE with pretty simple stuff, there was not much room for error.

I have access to a VPS so windows solution is no problem as long as we can be sure it will work out for 28 simultaneous instruments non stop 24 seven forever.


Should this code be enough:

//+------------------------------------------------------------------+
//|                                     DLabs_TickData_ToCSV_MT4.mq4 |
//|                                   Copyright 2018, Darwinex Labs. |
//|                         https://blog.darwinex.com/category/labs/ |
//+------------------------------------------------------------------+
#property copyright "Copyright 2018, Darwinex Labs."
#property link      "https://blog.darwinex.com/category/labs/"
#property version   "1.00"
#property strict
#property indicator_chart_window

// Variables
int csv_io_hnd;
MqlTick tick_struct;

//+------------------------------------------------------------------+
//| Custom indicator initialization function                         |
//+------------------------------------------------------------------+
int OnInit()
  {
//--- indicator buffers mapping
   
   // Create CSV file handle in WRITE mode.
   csv_io_hnd = FileOpen(Symbol() + "_TickData.csv", FILE_CSV|FILE_READ|FILE_WRITE|FILE_REWRITE, ',');
   
   // If creation successful, write CSV header, else throw error
   if(csv_io_hnd > 0)
   {
      if(FileSize(csv_io_hnd) <= 5)
         FileWrite(csv_io_hnd, "time_milliseconds", "bid", "ask", "spread");
         
      // Move to end of file (if it's being written to again)
      FileSeek(csv_io_hnd, 0, SEEK_END);
   }
   else
      Alert("ERROR: Opening/Creating CSV file!");
//---
   return(INIT_SUCCEEDED);
  }
//+------------------------------------------------------------------+
//| Custom indicator iteration function                              |
//+------------------------------------------------------------------+
int OnCalculate(const int rates_total,
                const int prev_calculated,
                const datetime &time[],
                const double &open[],
                const double &high[],
                const double &low[],
                const double &close[],
                const long &tick_volume[],
                const long &volume[],
                const int &spread[])
  {
//---
   
   // If CSV file handle is open, write time, bid, ask and spread to it.
   if(csv_io_hnd > 0)
   {
      if(SymbolInfoTick(Symbol(), tick_struct))
      {
         Comment("\n[Darwinex Labs] Tick Data | Bid: " + DoubleToString(tick_struct.bid, 5) 
         + " | Ask: " + DoubleToString(tick_struct.ask, 5) 
         + " | Spread: " + StringFormat("%.05f", NormalizeDouble(MathAbs(tick_struct.bid - tick_struct.ask), 5))
         + "\n\n* Writing tick data to \\MQL4\\Files\\" + Symbol() + "_TickData.csv ..."
         + "\n(please remove the indicator from this chart to access CSV under \\MQL4\\Files.)"
         );
         FileWrite(csv_io_hnd, tick_struct.time_msc, tick_struct.bid,
                       tick_struct.ask, StringFormat("%.05f", NormalizeDouble(MathAbs(tick_struct.bid - tick_struct.ask), 5)));
      } 
      else
         Print("ERROR: SymbolInfoTick() failed to validate tick."); 
   }
   
   
//--- return value of prev_calculated for next call
   return(rates_total);
  }
//+------------------------------------------------------------------+

void OnDeinit(const int reason)
{  
   // Close CSV file handle if currently open, before exiting.
   if(csv_io_hnd > 0)
      FileClose(csv_io_hnd);
      
   // Clear chart comments
   Comment("");
}

Or does this look better:

//------------------------------------------------------------------
#property copyright "www.forex-tsd.com"
#property link      "www.forex-tsd.com"
//------------------------------------------------------------------
#property indicator_chart_window

//
//
//
//
//

extern string FileName   = "saveticksmethodTicks.csv";
extern bool   SaveVolume = false; 
int fileHandle;

//+------------------------------------------------------------------+
//|                                                                  |
//+------------------------------------------------------------------+
//
//

int init()   { fileHandle = FileOpen(Symbol()+" - "+FileName,FILE_WRITE|FILE_SHARE_READ|FILE_CSV); return(0); }
int deinit() {              FileClose(fileHandle);                                 return(0); }
int start()
{
   if (fileHandle <0)
   {
      static bool alerted = false;
         if (!alerted)
         {
            Alert("File : "+Symbol()+" - "+FileName+" could not be opened"); alerted = true;
         }
         return(0);
   }         
   if (FileSize(fileHandle) == 0)
         FileWriteString(fileHandle,"date and time,bid,ask,volume\n");
         if (SaveVolume)
               FileWriteString(fileHandle,TimeToStr(TimeCurrent(),TIME_DATE|TIME_SECONDS)+","+DoubleToStr(Bid,Digits)+","+DoubleToStr(Ask,Digits)+","+DoubleToStr(Volume[0],0)+"\n");
         else  FileWriteString(fileHandle,TimeToStr(TimeCurrent(),TIME_DATE|TIME_SECONDS)+","+DoubleToStr(Bid,Digits)+","+DoubleToStr(Ask,Digits)+"\n");
         FileFlush(fileHandle);
   return(0);
}

You know any better way?

Also provided is a zip I mined from https://github.com/rosasurfer that includes a solution that apparently used to work on previous builds by using the "mt4 console" for mining the data through some PHP; is there anything like this for current build?

Any insight is more than welcome,

I dont mind windows with a paid vps  running, however a linux command line solution that just works would be ideal so anything like that around at all? My broker requires a 50k deposit for their API; I wont just hand someone 50k especially just for the API at this point in time.

Files:
nadiawicket
963
nadiawicket  
Or is
//+------------------------------------------------------------------+
//|                                             PriceQtIndicator.mq4 |
//|                                                             linx |
//|                             https://login.mql5.com/en/users/linx |
//+------------------------------------------------------------------+
#property copyright "linx"
#property link      "https://login.mql5.com/en/users/linx"

string symbol;
int handle;

#property indicator_chart_window
//+------------------------------------------------------------------+
//| Custom indicator initialization function                         |
//+------------------------------------------------------------------+
int init()
  {
//---- indicators
   symbol = Symbol();
   
   handle=FileOpen(symbol+"_tick.csv", FILE_CSV|FILE_WRITE,',');
   if (handle>0)
      FileWrite(handle, "datatime", "open", "high", "low", "close", "volume");
   else
      Alert("Failed to open data file. Please check if you have write priviledge!");

//----
   return(0);
  }
//+------------------------------------------------------------------+
//| Custom indicator deinitialization function                       |
//+------------------------------------------------------------------+
int deinit()
  {
//----
   if (handle>0)
      FileClose(handle);
//----
   return(0);
  }
//+------------------------------------------------------------------+
//| Custom indicator iteration function                              |
//+------------------------------------------------------------------+
int start()
  {
   //int    counted_bars=IndicatorCounted();
//----
   if (handle>0) {
      FileWrite(handle,iTime(symbol,PERIOD_M1,0), iOpen(symbol,PERIOD_M1,0),
                       iHigh(symbol,PERIOD_M1,0), iLow(symbol,PERIOD_M1,0),
                       iClose(symbol,PERIOD_M1,0), iVolume(symbol,PERIOD_M1,0));
   }
//----
   return(0);
  }
//+------------------------------------------------------------------+

The best?

I am in the process of tweaking/ testing them all out on debian and also on windows vps on 70+ simultaneous instruments, so any info or opinions is highly appreciated.

Stanislav Korotky
23220
Stanislav Korotky  

Try to profile your code and eliminate "heavy" unnecessary places (such as comments, excessive functions call, etc). Don't use indicator as MQL program type, chose expert.

The best solution would be use a broker with parallel MT5 support and download ticks history from server.

nadiawicket
963
nadiawicket  
Stanislav Korotky:

Try to profile your code and eliminate "heavy" unnecessary places (such as comments, excessive functions call, etc). Don't use indicator as MQL program type, chose expert.

The best solution would be use a broker with parallel MT5 support and download ticks history from server.

1. Why use an Expert Advisor over an Indicator for this?

2. How would you "download ticks history from server"? What exactly do you mean by that?

kypa
1079
kypa  

https://www.mql5.com/en/docs/series/copyticks

mt5 only

EA runs in a separate process and has (way) more asynchronous-ish functions.

Documentation on MQL5: Timeseries and Indicators Access / CopyTicks
Documentation on MQL5: Timeseries and Indicators Access / CopyTicks
  • www.mql5.com
[in]  The number of requested ticks. If the 'from' and 'count' parameters are not specified, all available recent ticks (but not more than 2000) will be written to ticks_array[]. The first call of CopyTicks() initiates synchronization of the symbol's tick database stored on the hard disk. If the local database does not provide all the requested...
nadiawicket
963
nadiawicket  
kypa:

https://www.mql5.com/en/docs/series/copyticks

mt5 only

EA runs in a separate process and has (way) more asynchronous-ish functions.

Which of these provided codes looks better for you?

To add comments, please log in or register