Asynchronous loading of charts - does MetaQuotes really know how many problems this creates?

New comment
 

I need to create an extra post for this topic, because it cannot be done with a simple code sample and it´s a real major problem, which cannot be handled, not in EAs, nor in indicators, but it must be discussed with the developers of MetaTrader. 

What´s the issue?

In the past it was like that - in MT4 at least - that when a timeframe was changed and/or a chart was opened, that all bars were accessible immediately. Thats why functions like iMA() just worked as they did. But in the meanwhile (I don´t know exactly since when) the chart data is loaded asynchronously. This means: MetaTrader provides just a part of the bars, beginning from the right side. An example: EURUSD consists of 100.000 bars in history, bar zero is the oldest, bar 99.999 the current. But after the timeframe was changed, only bars 90.000 - 99.999 are accessible. After some time - nobody knows when - MT adds the bars 80.000 - 89.999, later 70.000 - 79.999 and so on. So far so "good", but the consequences are:

EAs:

1. Variant A - when an EA is active and this EA accesses bar zero or any bar prior such bars which are accessible - it gets no values 

2. Variant B - the EA freezes for seconds, minutes or it crashes the entire MT environment. (I already posted code some months ago, in which one could see, that this happens. Some developers could confirm this, some could not. But its fact)

3. Variant C - the EA waits for data during CopyRates() etc. is used until the data is accessible - never worked for me, such a loop never found its end 


Indicators: 

1. The main function of indicators is OnCalculate() which is executed from the most left bar to the last. In this case of asynchronous loading in, let's say, 10 steps like described, the OnCalculate() is executed 10 times and calculates all available data 10 times. In fact this means, that in such a case the calculation time is 10+20+30+40+50+60+70+80+90+100 = 550% of the original time. In other words: It takes at least five times longer to calculate, the more steps, the longer it takes. I watched already more than 50 steps! Now imagine what happens, when a user uses a complex indicator and starts MetaTrader with 10 charts opened. Yes, it seems that the indicator is crashing - and of course it does from time to time. But actually it's not the indicator which crashes, it´s MetaTrader. 

2. Not every indicator deals with with bars data synchronously, but accesses also bars from other timeframes. Such an indicator has the same problems as an EA, but the major issue here is, that in such case one single indicator will crash all other indicators, because (as far as I know) all indicators are executed in one common task.


The following code snippet detects such asynchronous loadings:

         //+------------------------------------------------------------------+
         //| Event handler for OnCalculate()                                  |
         //+------------------------------------------------------------------+
      public:
         int OnCalculate( const int rates_total,
                         const int prev_calculated,
                         const datetime &time[],
                         const double &open[],
                         const double &high[],
                         const double &low[],
                         const double &close[],
                         const long &tick_volume[],
                         const long &volume[],
                         const int &spread[])
            {
               //--- Check if there was a connection reset or async loading
               static int rates_total_last=0; 
               static int _BarsCount=-1;
               if (rates_total_last>prev_calculated)
                  {
                  if (_BarsCount==rates_total)
                     return rates_total;
                  Comment("Re-Connection detected, recalculating ...");
                  //--- Internal reset
                  }
               //--- Store bar count   
               _BarsCount=rates_total;
               //--- Remember for internal reset check
               rates_total_last=rates_total;
               //---
               return rates_total;
            }

And produces this result. As we see, there were already much more async loadings than described in the example and therefore in this live-sample, which I just created on the fly when writing this post, we talk about 24 steps and because of that - about a calculation time with a factor or 12! Twelve!


Simply stated: This functionality is a very serious problem, which should be solved/changed as soon as possible, because in the very end, it makes any EA and any indicator to become somehow "unreliable". No matter how clean the code is, the only reason to deal with this is: Exit the EA, but sometimes (Variant B of EAs), there is no way out.

In my opinion, this functionality should be removed entirely. Any user and any developer should not have a problem to wait some seconds until the chart is displayed, but all of them have a problem with crashes and unpredictable behavior of EAs and indicators. 


MetaQuotes, could you please provide a statement?

And, if you do not agree, could you provide a developers access to the databases where the bars data is stored, or is there already such? 


Thanks in advance

 
I can confirm this issue!
Should definitely be reviewed by stuff!
 
Doerk Hilger:

I need to create an extra post for this topic, because it cannot be done with a simple code sample and it´s a real major problem, which cannot be handled, not in EAs, nor in indicators, but it must be discussed with the developers of MetaTrader.

I do not agree.

The problem will only be in very primitive code that does not take into asynchrony. Consider this asynchrony in your indicators and EA, always control which bar is first, which last and there will be no problems.
This is done to save time, which is so important for trading.

 
Geess:

I do not agree.

The problem will only be in very primitive code that does not take into asynchrony. Consider this asynchrony in your indicators and EA, always control which bar is first, which last and there will be no problems.
This is done to save time, which is so important for trading.


In "very primitive code" ... ? :) Made my day ;) 

 
Doerk Hilger: What´s the issue?

In the past it was like that - in MT4 at least - that when a timeframe was changed and/or a chart was opened, that all bars were accessible immediately...

Obviously our German readers are mislead and don't know MetaTrader very well.

At least since MT build 224 (10 years ago) history loading works like this: Data pumping (loading of price history) happens in a separate background thread and in blocks of 512 bars each. New bars arrive first. As Indicator::start() is called multiple times during data pumping the available chart history will still contain gaps. Each time Indicator::start() is called IndicatorCounted() will return 0 (zero = all bars changed). At the first call of Indicator::start() after data pumping IndicatorCounted() still returns 0 (zero = all bars changed).

ps: Substitute Indicator:start() with Indicator::OnCalculate() and IndicatorCounted() with the matching parameter to get behaviour for builds >= 600.

pps:
(1) How to detect the finishing of data pumping in an indicator. Check IndicatorCounted() or the according parameter of OnCalculate().
(2) How to detect the finishing of data pumping in an expert. You can't. Compare the number of bars to the number of bars at the previous tick. If the difference is greater than 1 assume IndicatorCounted() = 0 (all bars changed).

 
alphatrading:

Obviously our German readers are mislead and don't know MetaTrader very well.

Please avoid any personal opinions and generalizations, this does not help anybody.

alphatrading:

At least since MT build 224 (10 years ago) history loading works like this: (...)

Can you provide a source for this? Cause it´s not what I have in mind. 

alphatrading:

pps: 

(1) How to detect the finishing of data pumping in an indicator. Check IndicatorCounted() or the according parameter of OnCalculate().
(2) How to detect the finishing of data pumping in an expert. You can't. Compare the number of bars to the number of bars at the previous tick. If the difference is greater than 1 assume IndicatorCounted() = 0 (all bars changed).

1 - as you see in the code, it´s done like that using bars_calculated, it´s not the reason. Bars() is also changed permanently. Furhtermore, none of these reasons even if Im wrong could avoid a total recalculation when the connection has changed, which is also caused by MT sometimes.

2 - Yes, just as you write: There is no reason. Translated: It´s all just about hope when the expert is started. Lucky when you receive the quote you requested, bad if not. And regarding ticks: Some EAs and indicators don´t deal with ticks, and IndicatorCounted() is not available in MT5 experts. It´s necessary to have a solution which works always. 

And finally, this is a proven issue, the indicator-calculation goes wrong anyway. Just put a standard bollinger-bands or whatever on your chart and just leave it open. Come back later, maybe very much later, and you will recognize, that the curves are somewhere in the chart, but not there where they should be. 

 

And to keep it all a little simpler:

When an expert or indicator asks for a price of a specific bar, this price should be returned - always, no matter what is organized in the background - and a function like CopyRates() should never freeze the EA, the indicator or the whole system. Same as for iMA() etc., an expert needs this value when it´s asking, not when functions in the background are done with anything. It´s a matter of priority. We talk about apps that deal with real money, not about any funny games, we need reliability.

If MetaQuotes is not willing to change the architecture, which I could also understand, there should at least be solutions, such as an event or a function which tells when this background-loading-process is done. This would at least be a workaround for quote-requests. And if this is also not wanted for whatever reason, we should have read-access to the database. And if we get none of these, whats the reason then? Taking the quotes from somewhere else? ... not really, but this is where it forces us to go, at least me. 

But also for integrated indicator-functions such iMA() - at the moment they just can be used in standard-environments, but never out of the box. In experts you deal with the same problem: Maybe data is there, maybe not and no function tells you when it´s ready. I already created an own class which provides calculating and buffering of indicator values, because all the rest was always either too slow or too unreliable. And by the way, this is what I am doing really too often: Building workarounds. It´s a huge waste of time. 

 
Please refer to https://www.mql5.com/en/docs/constants/tradingconstants/enum_series_info_integer

You might want to try this:
SERIES_SYNCHRONIZED


Good luck.
 
Doerk Hilger:

Please avoid any personal opinions and generalizations, this does not help anybody.

Sorry for that. Couldn't help myself :-)


Can you provide a source for this? Cause it´s not what I have in mind. 

That's from my experience as I'm around long enough. Write an indicator and on each call of OnCalculate() log the relevant data. In MT5 the number 512 may differ but the fact stays the same: For each loaded chunk the indicator is called once.


...Furhtermore, none of these reasons even if Im wrong could avoid a total recalculation when the connection has changed, which is also caused by MT sometimes.

A changing connection is a different ball park. Anyway, on connection change the full history changes and all bars are signaled as changed. No issues with recalculation.


...There is no reason. Translated: It´s all just about hope when the expert is started...

Don't agree. You don't need fancy parameters or constants to initialize an expert. The current number of bars in the chart is enough to emulate the non-existing IndicatorCounted(). However, for calls of built-in technical indicators like iMA() you are right. Only way here is to make sure history is loaded when you attach the EA.


...Just put a standard bollinger-bands or whatever on your chart and just leave it open. Come back later, maybe very much later, and you will recognize, that the curves are somewhere in the chart, but not there where they should be. 

If a custom indicator shows such behaviour the start bar calculation is wrong. Unfortunately most of the examples provided by MetaQuotes don't show how to correctly calculate the first (i.e. oldest) bar to update. Most provided examples are just wrong. An experienced developer can easily figure out how to do it correctly. I can't /won't show for MT5 as that's not my playground. Showing it for MT4 would be misplaced.


Good luck
 
alphatrading:

Don't agree. You don't need fancy parameters or constants to initialize an expert. The current number of bars in the chart is enough to emulate the non-existing IndicatorCounted(). (...)

I am tired of polemic answers, sorry.

One should not always take ones own situation as a benchmark when someone describes a problem. From time to time, one should rather assume that a questioner knows exactly what he is doing, in other words it doesn't help if one claims that this is not a problem just because one can handle it oneself in the own tasks. Other developers, other tasks.

------------------

------------------

I can reduce all this to one single question:

1. how can I basically retrieve the required quote data at any time from any timeframe without any misbehavior?

If there is no answer, and no perfect workaround, it´s something that should be improved, because I am not the only ones who needs this. I am also in contact with other developers, not only in my team, also others who totally agree on this and who are also tired of creating workarounds, but also tired posting and describing problems like these here and who think it´s a waste of time, cause issues like these are usually ignored by MQ.

However, this is why addressed the topic mainly to MetaQuotes directly, to help them to understand that there are circumstances/tasks in which this behavior results in massive, serious problems. And ours is not the only one. If they are willing to help here, they will help many other programmers as well, also all my partners and finally all our common clients. 

 
Doerk Hilger:

I am tired of polemic answers, sorry.

One should not...

What do you know about "one"? I'm tired of your attitude.

Good luck. You will need it.
123
New comment
Reason: