Tick data in MT5
is there someone (especially the experienced mods) who can comment on this topic?
Thanks!
You have not provided any information that helps recreate or demonstrate your issue, so there is little that can be said.
One would expect:
- A code sample to allow one to recreate the issue or that demonstrates how you were collecting the tick data.
- Sample log output of the data, to make comparison to internally stored data.
- Screenshot of the Symbols tick data to compare with the logged data.
- General metrics and details of your comparison, such symbol, time of day where it was most prevalent, and any other relevant information.
thanks a lot for your reply.
My questions are more generic and not directly code-related. What I am not sure is on how "correct/realistic" or reliable (I think that's the best word to describe it) the available tick data within MT5 (of course, dependent on broker) are on a daily basis. I am wondering on how good they are, maybe you have some information on that.
You have not provided any information that helps recreate or demonstrate your issue, so there is little that can be said.
One would expect:
- A code sample to allow one to recreate the issue or that demonstrates how you were collecting the tick data.
- Sample log output of the data, to make comparison to internally stored data.
- Screenshot of the Symbols tick data to compare with the logged data.
- General metrics and details of your comparison, such symbol, time of day where it was most prevalent, and any other relevant information.
To your questions: Basically, the (very) slight difference is independent on symbol or time of day, it occurs on major forex symbols as well as on major indices. Sometimes, there are one or two specific ticks missing at 00:30 o'clock, sometimes at 11:00 am.
Thank you in advance for your feedback!
It can be clearly seen from your words, that your issue is specific and not generic.
Your primary question:
"I am thinking about the reason, is it because the daily available tick data of MT5 is incorrect or are the recorded ticks not recording everything?"
Answer:
Please refer to my post #2.
Your additional question:
"How reliable/realistic are the daily ticks available in MT5?"
Answer:
Several EAs of mine do active tick data analysis, and I have no issues neither live nor in back-testing.
However, your definition of "reliability" can only be considered within the context of your issue, and for that please refer to my post #2
I can only guess that you are not collecting or processing the tick data correctly.
thank you for your fast reply.
With your latest post, I do have two questions:
- Are the available daily ticks, available in MT5, reliable? In other words: Are those ticks complete and full, according to the ticks which occurred on the day? Or is there some "gap"?
- Maybe the tick data is not recorded correctly, therefore, here my current code to copy all ticks within the OnTick function:
MqlTick tick_array[]; bool success=false; string path="Data"+"//"+"file.txt"; int copied=CopyTicks(_Symbol,tick_array,COPY_TICKS_ALL,0,10); if(copied!=-1) { if(!GetLastError()==0) PrintFormat("%s: Ticks are not synchronized, error=%d",_Symbol,copied,_LastError); else success = true; } else PrintFormat("Failed CopyTicks(%s), Error=",_Symbol,GetLastError()); //--- Saving ticks in file if(success) { int ticks=ArraySize(tick_array); for(int i=0;i<=ticks-1;i++) { // If tick time_msc has only 2 or even 1 or 0 digits, then add 0 before (e.g. 60 to 060) time = (string)tick_array[i].time + "." + (string)(tick_array[i].time_msc%1000); if(StringLen((string)(tick_array[i].time_msc%1000)) < 3) time = (string)tick_array[i].time + ".0" + (string)(tick_array[i].time_msc%1000); if(StringLen((string)(tick_array[i].time_msc%1000)) < 2) time = (string)tick_array[i].time + ".00" + (string)(tick_array[i].time_msc%1000); if(StringLen((string)(tick_array[i].time_msc%1000)) < 1) time = (string)tick_array[i].time + ".000"; int file_handle=FileOpen(path,FILE_READ|FILE_WRITE|FILE_TXT|FILE_ANSI,';'); if(file_handle!=INVALID_HANDLE) { // Adding new ticks only if tick is not available in file yet while(!FileIsEnding(file_handle)) last_row = FileReadString(file_handle); u_sep=StringGetCharacter(sep,0); int k=StringSplit(last_row,u_sep,result); if(k>0) { string_date = result[0]; string_bid = result[1]; string_ask = result[2]; } // if(time > string_date || (time == string_date && (StringToDouble(string_ask) != tick_array[i].ask || StringToDouble(string_bid) != tick_array[i].bid))) { FileSeek(file_handle,0,SEEK_END); FileWrite(file_handle, time, tick_array[i].bid,tick_array[i].ask,tick_array[i].last,tick_array[i].volume,tick_array[i].flags); FileClose(file_handle); } FileClose(file_handle); } else Print("Failed to open the file, error ",GetLastError()); } }
As stated above, the code checks the last 10 ticks in the OnTick functio . If there is a new tick, the code checks the last line of the file.txt. If the current tick date is newer, then the EA saves this specific tick incl. its information. The same applies for ticks which occur at the same time, but if bid or ask are different.
Do you see the reason why there is a very minor difference in the MT5 recorded ticks and the recorded ticks on this EA?
Thanks!
Already answered in post #4. I personally have no issues, and I actively use 3 different brokers and one prop firm.
The method you use in code is not a reliable method of collecting ticks. It can easily skip ticks (or even duplicate them).
The OnTick() event handler in EAs, will skip new tick events while the handler is still running, and during that time, a cache of only 10 ticks (as per your code) may not be enough to cover it, especially during periods of high volatility.
Also, the millisecond timestamp is not unique. You can easily have several ticks per millisecond all with the same timestamp.
The following code of mine is already more than a year old (my current code is a little more complex), but the following serves as an example to show how complex it can be to make sure one does not miss ticks.
Forum on trading, automated trading systems and testing trading strategies
Fernando Carreiro, 2023.09.04 15:57
It is somewhat complex and can easily be a couple hundred lines of code.
Here is a skeleton code example for the tick processing and timing of a seconds based time-frame. It does however, not build the OHLC bar data. In simply uses the open mid price "(Ask + Bid)/2".
You will have to extend the functionality if you want to build the OHLC data as well.
//--------------------------------------------------------------------------------------------------------------------- #property copyright "Copyright \x00A9 2022, Fernando M. I. Carreiro, All rights reserved" #property link "https://www.mql5.com/en/users/FMIC" //--------------------------------------------------------------------------------------------------------------------- //--- Parameter settings input group "Parameters: Tick data processing" input uint i_nTimeframeSeconds = 15, // Time-frame in seconds i_nTickHistoryDays = 5; // Days to backtrack tick history // enough to cover long weekends //--- Macro definitions // Define macro for invalid parameter values #define MCheckParameter( _condition, _text ) if( _condition ) \ { Print( "Error: Invalid parameter - ", _text ); return INIT_PARAMETERS_INCORRECT; } // Define macros for time masking #define MDayMinutes ( 24 * 60 ) #define MDaySeconds ( 24 * 60 * 60 ) #define MTimeOnly( _vardatetime ) ( int( ( _vardatetime ) % MDaySeconds ) ) #define MDateOnly( _vardatetime ) ( int( ( _vardatetime ) / MDaySeconds ) ) #define MDateTime( _vardate, _vartime ) ( datetime( ( _vardate ) * MDaySeconds + ( _vartime ) ) ) //--- Structures and classes // Tick data processor structure struct STickProcessor { // Declare member variables long m_nTicksTimeMsc; // Next search time (milliseconds) int m_nTicksTimeCount, // Next search duplicate time counter m_iTicksIndex, // Index of current tick data m_iTicksLast, // Index of last valid tick data m_nTicksCount; // Count of available ticks in the cached data MqlTick m_oTicks[]; // Tick data cache string m_sSymbol; // Symbol name for tick data bool m_bRealtime; // Real-time processing flag // Default deconstructor ~STickProcessor( void ) { ArrayFree( m_oTicks ); }; // Default constructor STickProcessor( void ) : m_nTicksTimeMsc( WRONG_VALUE ), m_nTicksTimeCount( WRONG_VALUE ), m_iTicksIndex( WRONG_VALUE ), m_iTicksLast( WRONG_VALUE ), m_nTicksCount( WRONG_VALUE ), m_bRealtime( false ), m_sSymbol( "" ) {}; // Initialiser bool Init( string sSymbol, long nTicksTimeMsc ) { if( ( nTicksTimeMsc > 0 ) && ( sSymbol.Length() > 0 ) ) { m_nTicksTimeMsc = nTicksTimeMsc; m_nTicksTimeCount = 0; m_iTicksIndex = m_iTicksLast = m_nTicksCount = WRONG_VALUE; m_bRealtime = false; m_sSymbol = sSymbol; return true; }; return false; // Failed to initialise }; // Verify validaty of current tick data state bool IsValid( void ) { return ( m_nTicksCount > 0 ) && ( m_iTicksIndex >= 0 ) && ( m_iTicksIndex <= m_iTicksLast ); }; // Update Real-time status void RealTimeUpdate( void ) { if( m_iTicksIndex == m_iTicksLast ) m_bRealtime = m_oTicks[ m_iTicksIndex ].time >= TimeCurrent(); }; // Advance to next tick bool Next( void ) { if( ( m_nTicksCount > 0 ) && ( m_iTicksIndex < m_iTicksLast ) ) { m_iTicksIndex++; // Next tick if( !m_bRealtime ) RealTimeUpdate(); // Update Real-time status return true; }; if( m_nTicksTimeMsc > 0 ) { // Acquire new data for the cache ResetLastError(); m_nTicksCount = _StopFlag ? 0 : CopyTicks( m_sSymbol, m_oTicks, COPY_TICKS_ALL, m_nTicksTimeMsc ); if( m_nTicksCount > m_nTicksTimeCount ) { // Count final ticks as there may be more in the future with the same timestamp int iTicksFinal = m_nTicksCount - 1; m_iTicksLast = iTicksFinal; m_nTicksTimeMsc = m_oTicks[ m_iTicksLast ].time_msc; while( ( iTicksFinal >= 0 ) && ( m_oTicks[ iTicksFinal ].time_msc >= m_nTicksTimeMsc ) ) iTicksFinal--; m_iTicksIndex = m_nTicksTimeCount; // Dismiss initial duplicate ticks m_nTicksTimeCount = m_iTicksLast - iTicksFinal; if( !m_bRealtime ) RealTimeUpdate(); // Update Real-time status return m_iTicksIndex <= m_iTicksLast; // Succeeded in acquiring tick data }; // Report any errors while acquiring the tick data if( ( m_nTicksCount == WRONG_VALUE ) || ( _LastError != ERR_SUCCESS ) ) PrintFormat( "Error %d: Unable to acquire tick data for symbol %s", _LastError, m_sSymbol ); // Reset everything m_iTicksIndex = m_iTicksLast = m_nTicksCount = WRONG_VALUE; }; return false; // Failed to acquire tick data }; // Get current tick time - Attention! For the sake of efficiency, validity is not verified datetime Time( void ) { return m_oTicks[ m_iTicksIndex ].time; }; // Get current mid price - Attention! For the sake of efficiency, validity is not verified double MidPrice( void ) { return 0.5 * ( m_oTicks[ m_iTicksIndex ].ask + m_oTicks[ m_iTicksIndex ].bid ); }; }; //--- Global variable declarations STickProcessor g_oTickProcessor; // Tick data processor structure object //--- Event handling functions // Initialisation event handler int OnInit( void ) { // Validate inputs and calculate parameter variables MCheckParameter( i_nTickHistoryDays < 1, "days to backtrack tick history" ); MCheckParameter( i_nTimeframeSeconds < 1, "time-frame in seconds" ); return INIT_SUCCEEDED; // Successful initialisation of indicator }; // Tick event handler void OnTick( void ) { // Detect first tick static bool bFirstTick = true; if( bFirstTick ) { // Initialise tick processing object with symbol and backtracking time g_oTickProcessor.Init( _Symbol, ( TimeCurrent() - i_nTickHistoryDays * MDaySeconds ) * 1000 ); bFirstTick = false; // First tick condition complete }; // Process tick data while( g_oTickProcessor.Next() ) { // Detect a change in timestamp (new bar in specified time-frame) static datetime dtTimeStampCurrent = WRONG_VALUE; datetime dtTimeStampPrevious = dtTimeStampCurrent; dtTimeStampCurrent = g_oTickProcessor.Time() / i_nTimeframeSeconds; bool bTimeStampNew = dtTimeStampCurrent != dtTimeStampPrevious; // Process opening state for new time (second based time-frame) if( bTimeStampNew ) { // Build data ... // ... do something with data ... // Check if we processing data in real-time if( g_oTickProcessor.m_bRealtime ) { // ... carry out trading operations ... }; }; }; }; //---------------------------------------------------------------------------------------------------------------------
Already answered in post #4. I personally have no issues, and I actively use 3 different brokers and one prop firm.
The method you use in code is not a reliable method of collecting ticks. It can easily skip ticks (or even duplicate them).
The OnTick() event handler in EAs, will skip new tick events while the handler is still running, and during that time, a cache of only 10 ticks (as per your code) may not be enough to cover it, especially during periods of high volatility.
Also, the millisecond timestamp is not unique. You can easily have several ticks per millisecond all with the same timestamp.
The following code of mine is already more than a year old (my current code is a little more complex), but the following serves as an example to show how complex it can be to make sure one does not miss ticks.
Dear Fernando,
thanks a lot for your fast reply. Your explanation makes sense, also thank you for sharing your code on how you prevent any ticks missing. More complex, yes, but well thought through.
Thank you for your support, highly appreciate that!
- Free trading apps
- Over 8,000 signals for copying
- Economic news for exploring financial markets
You agree to website policy and terms of use
I have written a code which is recording live tick data.
However, when comparing the recorded tick data with the tick data available in the broker's MT5 (View - Symbols - Ticks), there are a little less ticks which I have recorded.
Question: I am thinking about the reason, is it because the daily available tick data of MT5 is incorrect or are the recorded ticks not recording everything?
Note that I am checking the last 10 ticks at each new tick, and if there was any change and I am recording only new ticks. Therefore, reason #2 seems not plausible to me.
Additional question to that: How reliable/realistic are the daily ticks available in MT5?
Thanks in advance for your feedback.