Download MetaTrader 5

Refreshing offline charts - background, performance and solution ideas

To add comments, please log in or register
JHawk
55
JHawk  

Hello everyone :-)

Now this is my first question after I read so much now I still don't understand, how indicator data are retrieved and from where. And  how static indiactor retrieval  can be optimized.

Maybe what I want is a bit special. I explain:

- Most of my indicators are static in the sense as they will never be optimized or changed with parameters. As simple well know sample you may look at daily pivot point levels.

- I know many people use many symbols and also test different parameters on the used indicators. I trade maybe 2 symbols and I simply use very very little "moving" indicators like maybe the standard bollingers or Keltener Channel with EMA but thats it.


So what I stumbled over is the refreshing of offline charts which - as I learned now - calls oncalculate with prev_calculated = 0.

That had the effect, that I even had problems to simply change charts with the mouse click while my gaming notebook I use started cooking....


And simply to say: All this to recalculate my - static - indicators!

And even for online charts it is a performance problem, because most of my indicators are never looking back, but are always calculated with asSeries = false.


A note: I use and trade with MT4 because my broker uses it. But I tried to solve the problems also in MT5 in making a custom symbol.


So is there any chance in any Metatrader to have quotes and indicator data ready for all charts one uses in a global manner?

So I don't have to use any Copy mechanism like CopyBuffer from MT5 or iCustom from MT4 and mainly reach the effect that the data are not recalculated?

I am one step before saving my static data out as CSV or may as new symbol history to reach this. But that is also one step before leaving MT and using any other System like C or C#.


Take the daily pivot point levels as a sample. They never change if you look back. How can I calculate them ONCE and forever and have them  - in the best case - always at hand in memory?

Many thanks in advance for all answers.

JHawk

Stanislav Korotky
23361
Stanislav Korotky  
It seems you have some problems in your design or use case. Indicators are normally calculated once (especially if they "static"), and then they can be calculated in optimized manner - only on new missing bars - unless you drop previous calculations intentionally as you described. Just don't do this. Also it's not clear why do you use offline charts. If you mean ticks and not complete refresh of an offline chart, then it's performed in MT4 by WinAPI call (you may see how it's used here).
JHawk
55
JHawk  
Stanislav Korotky:
It seems you have some problems in your design or use case. Indicators are normally calculated once (especially if they "static"), and then they can be calculated in optimized manner - only on new missing bars - unless you drop previous calculations intentionally as you described. Just don't do this. Also it's not clear why do you use offline charts. If you mean ticks and not complete refresh of an offline chart, then it's performed in MT4 by WinAPI call (you may see how it's used here).

Hi Stanislav,

first many thanks for your quick answer. And what coincidence I just today downloaded your code and read your great article "MQL's OOP notes: Singleton, Command queue, and Template method patterns in simple order manager". Havent read all but I already learned a lot from it. Thank you for that too.

Now, what I use is regression channels in many forms and many different starting points. you can reuse historical values for that only to a certain extend.If the ending point of the regression channel changes with new bars you have to calculate the whole channel.

I already optimized the calculation in saving the intermediate sums in supporting arrays.

I managed to get down to 40% of the orginal elapsed time  as the profiler showed me. But anyway the calculation has its cost.


You are right, on normal charts I dont have that problem. I have the problem on FXBlue Barchanger Charts which should do not much different than periodconverter to restrict the charts to certain time ranges (market times).

But maybe -  as I write - that is the problem. As I understand in these two lines of the example you gave ...


if(EmulateOnLineChart && MT4InternalMsg == 0)
        MT4InternalMsg = RegisterWindowMessageA("MetaTrader4_Internal_Message");

    if(hwnd != 0) if(PostMessageA(hwnd, WM_COMMAND, 0x822c, 0) == 0) hwnd = 0;
    if(hwnd != 0 && MT4InternalMsg != 0) PostMessageA(hwnd, MT4InternalMsg, 2, 1);



I understand that the second one with the

MT4InternalMsg

simulates the tick. But the first one

WM_COMMAND

is nothing else but calling refresh from the menu?

So I would have to reinvent the BarChanger basing on the periodconverter without the first line? (because FXBlue is not open source)

Am I right with these assumptions?


Greetings

JHawk

(hope my english is OK)

Stanislav Korotky
23361
Stanislav Korotky  
JHawk:

You are right, on normal charts I dont have that problem. I have the problem on FXBlue Barchanger Charts which should do not much different than periodconverter to restrict the charts to certain time ranges (market times).

But maybe -  as I write - that is the problem. As I understand in these two lines of the example you gave ...


I understand that the second one with the

simulates the tick. But the first one

is nothing else but calling refresh from the menu?

So I would have to reinvent the BarChanger basing on the periodconverter without the first line? (because FXBlue is not open source)

Am I right with these assumptions?

Well, this works like a charm in the renko charts - without lagging. Unfortunately I don't know how FXBlue charts work. The only one weird assumption which I could make is that the renko code updates another window, while yours maybe attepts to refresh itself.

Anyway, I don't see the reason why to refresh entire chart. If your code "sees" new bars for calculation regression channel, just do it on updated position without refresh.

PS. Can you try your indicator on different offline charts, for example on this renko? This way you can deduce what part is the bottleneck.

JHawk
55
JHawk  
Stanislav Korotky:

Well, this works like a charm in the renko charts - without lagging. Unfortunately I don't know how FXBlue charts work. The only one weird assumption which I could make is that the renko code updates another window, while yours maybe attepts to refresh itself.

Anyway, I don't see the reason why to refresh entire chart. If your code "sees" new bars for calculation regression channel, just do it on updated position without refresh.

The BarChanger is an indicator that produces an offline chart like the RenkoLiveChart does. This indicator uses refresh like RenkoLiveChart on the offline chart.

You understand my situation if you imagine putting an indicator on the RenkoChart. That indicator then gets a refresh every new bar and you would have my problem.


Yet your answers helped me think in new directions :-)


I thank you very much for your support and good will!

I try to find a solution and maybe can give something back here when I successful.


Greetings

JHawk

JHawk
55
JHawk  
Stanislav Korotky: PS. Can you try your indicator on different offline charts, for example on this renko? This way you can deduce what part is the bottleneck.


As I understand it with offline charts is that there is no other way but refresh to get new data into the chart, that is loading data from the history files.

The

PostMessageA(hwnd, MT4InternalMsg, 2, 1);


is just the eventdriver for the tick event with the aim to have an OnTick()-Event in offline charts and that for a expert to be able to work.

---

All the problem I have here is that my CFD provider gives me 24 hour quotes. That means for Dow Jones for example 6.5 hours real data (Eastern Standard Time 9.30 am to 4 p.m.) and 17.5 hours of junk data (anything what my provider thinks it would be worth).

Thats what the BarChanger is for. it filters out those not-market-quotes and gives me a something that is near the dow jones.

If I could restrict the time range on my quote FOR A VIEW like a chart, that would be fantastic and probably not only for me.

BUT most programs - and think Metatrader too - if they allow such a restriction, its not for a view like a chart but the instrument like the Symbol in MT. Other have this possibility too.

But that is like wearing sunlasses in the night. (could be funny though ...)

That means: its not possible to track a trade after hours.

I never understand people using moving averages on this kind of 24 Hour quotes. Or gaps for example just do not exist.

Sorry. That was off topic. But it explains my need. And I know I am not alone because thats why FXBlue made this MT-Indicator and sells it as addon to brokers.

Now I wish a good and relaxed night.


JHawk

Stanislav Korotky
23361
Stanislav Korotky  

Well, then I can sugget 2 methods to try.

If you have a sources code which initiates the refresh, you could probably "mark" the artificial refreshes with a global flag (in global variables or something other) and in the indicators omit every new tick when the flag is on, then changing a counter for this flag from every indicator. When the counter becomes equal to a number of your processing indicators - remove the flag.

Another way is to not use IndicatorCounted in your indicators at all. Instead check for number of bars and datetime of the latest bar. When they changed for more than 1 single bar - make refresh. Otherwise process ticks only.

alphatrading
82
alphatrading  

It is a tricky issue and a limitation of offline charts. A bullet-proof workaround is possible but not for the faint-hearted. First we need to clarify what happens in offline charts opposite to regular charts.

If the number of bars changes in regular charts (by refresh or by arrival of a new tick which triggers BarOpen) the terminal shifts timeseries and applied indicator buffers by the required amount and updates the return value of IndicatorCounted() accordingly. Now user-land code can use this value and update only the modified bars. This works efficient and is quick.

However, if the number of bars changes in an offline chart (by refresh only) the automatic shifting of indicator buffers does not happen. Instead IndicatorCounted() always returns all bars as invalid. From the point of view of the terminal this is correct because the terminal reloaded all bars from history and cannot know that we might have changed only a few bars at the beginning.

There are 2 issues now. (1) Indicator buffers have been calculated and are filled with valid data but are not shifted. (2) User-land code has no information about the amount of shifting that would be required to save already calculated data.


The following code will show how to handle both issues. The lenghty logic is required to handle all possible edge cases. One could reduce the lines of code but readability would suffer. In most of the cases the loops are NOT executed. If they are they return so quickly that you will not notice a difference to regular charts.

/**
 * Main function
 *
 * @return int - error status
 */
int onTick() {
   // original Valid/ChangedBars values for regular charts (might be modified in offline charts)
   int ValidBars   = IndicatorCounted();
   int ChangedBars = Bars - ValidBars;
   int ShiftedBars = 0;

(1) Introduce a variable ShiftedBars holding the amount of bars to shift indicator buffers (skipped by the terminal in offline charts):

   // manually calculate Valid/Changed/ShiftedBars in offline charts: IndicatorCounted() will always return all bars as modified
   static int      last_bars = -1;
   static datetime last_startBarOpenTime, last_endBarOpenTime;

   if (!ValidBars) /*&&*/ if (!IsConnected()) {                         // detect offline charts

      // initialisation
      if (last_bars == -1) {
         ChangedBars = Bars;                                            // first timeseries access
      }

      // number of bars didn't change
      else if (Bars == last_bars) {
         if (Time[Bars-1] == last_endBarOpenTime) {                     // oldest bar still the same
            ChangedBars = 1;
         }
         else {                                                         // oldest bar modified => bars at the end have been dropped
            if (Time[0] == last_startBarOpenTime) {                     // new bars inserted into gap: ambiguous => mark all bars as invalid
               ChangedBars = Bars;
            }
            else {                                                      // new bars added at the beginning: look-up Bar[last_startBarOpenTime]
               for (int i=1; i < Bars; i++) {
                  if (Time[i] == last_startBarOpenTime) break;
               }
               ShiftedBars = i;
               ChangedBars = i+1;                                       // also invalidate Bar[last_startBarOpenTime] to simulate ChangedBars=2 on BarOpen
            }
         }
      }

      // number of bars changed (must have been increased)
      else {
         if (Time[Bars-1] == last_endBarOpenTime) {                     // oldest bar still the same
            if (Time[0] == last_startBarOpenTime) {                     // new bars inserted into gap: ambiguous => mark all bars as invalid
               ChangedBars = Bars;
            }
            else {                                                      // new bars added at the beginning: look-up Bar[last_startBarOpenTime]
               for (i=1; i < Bars; i++) {
                  if (Time[i] == last_startBarOpenTime) break;
               }
               ShiftedBars = i;
               ChangedBars = i+1;                                       // also invalidate Bar[last_startBarOpenTime] to simulate ChangedBars=2 on BarOpen
            }
         }
         else {                                                         // oldest bar is modified
            if (Time[Bars-1] < last_endBarOpenTime) {                   // new bars added at the end: invalidate all bars
               ChangedBars = Bars;
            }
            else {                                                      // bars dropped at the end
               if (Time[0] == last_startBarOpenTime) {                  // new bars inserted into gap: ambiguous => mark all bars as invalid
                  ChangedBars = Bars;
               }
               else {                                                   // new bars added at the beginning: look-up Bar[last_startBarOpenTime]
                  for (i=1; i < Bars; i++) {
                     if (Time[i] == last_startBarOpenTime) break;
                  }
                  ShiftedBars = i;
                  ChangedBars = i+1;                                    // also invalidate Bar[last_startBarOpenTime] to simulate ChangedBars=2 on BarOpen
               }
            }
         }
      }
   }
   last_bars             = Bars;
   last_startBarOpenTime = Time[0];
   last_endBarOpenTime   = Time[Bars-1];
   ValidBars             = Bars - ChangedBars;                          // re-define ValidBars

Now we have 3 variables ValidBars, ChangedBars and ShiftedBars holding the correct values in regular and in offline charts. In regular charts ShiftedBars will always be 0.

(2) In offline charts the indicator buffers need to be shifted by us (skipped by the terminal):

   // manually shift indicator buffers in offline charts to not lose already calculated data
   if (ShiftedBars > 0) {
      ShiftIndicatorBuffer(bufferMA,        Bars, ShiftedBars, EMPTY_VALUE);
      ShiftIndicatorBuffer(bufferTrend,     Bars, ShiftedBars,           0);
      ShiftIndicatorBuffer(bufferUpTrend,   Bars, ShiftedBars, EMPTY_VALUE);
      ShiftIndicatorBuffer(bufferDownTrend, Bars, ShiftedBars, EMPTY_VALUE);
   }
Continue with regular indicator re-calculation:
   // re-calculate invalid bars
   for (int bar=ChangedBars; bar >= 0; bar--) {
      ...
   }
}

Final missing piece is the function ShiftIndicatorBuffer(). While it is possible to implement it in MQL for performance reasons it is strongly recommended to move the code to a DLL where memory can be moved directly without using loops:

/**
 * Shift indicator buffer values by an amount of bars to the end. Old bars at the end will be discarded.
 *
 * @param  double buffer[]   - MQL indicator buffer
 * @param  int    bufferSize - size of the indicator buffer in bars
 * @param  int    bars       - amount of bars to shift the buffer
 * @param  double emptyValue - initialization value for emptied elements at buffer start
 *
 * @return BOOL - success status
 */
BOOL WINAPI ShiftIndicatorBuffer(double buffer[], int bufferSize, int bars, double emptyValue) {
   if (buffer && (uint)buffer < MIN_VALID_POINTER) return(error(ERR_INVALID_PARAMETER, "invalid parameter buffer = 0x%p (not a valid pointer)", buffer));
   if (bufferSize < 0)                             return(error(ERR_INVALID_PARAMETER, "invalid parameter bufferSize = %d", bufferSize));
   if (bars < 0)                                   return(error(ERR_INVALID_PARAMETER, "invalid parameter bars = %d", bars));
   if (!bufferSize || !bars) return(TRUE);

   MoveMemory((void*)&buffer[0], &buffer[bars], (bufferSize-bars)*sizeof(buffer[0]));

   for (int i=bufferSize-bars; i < bufferSize; i++) {
      buffer[i] = emptyValue;
   }
   return(TRUE);
   #pragma EXPANDER_EXPORT
}


MQL import statement:
#import "Expander.dll"
   bool   ShiftIndicatorBuffer(double buffer[], int bufferSize, int bars, double emptyValue);
#import


The solution works flawless under all observed circumstances (as to my knowledge and my use of it over the last years). If you hit refresh in an offline chart your indicators will only re-calculate the new arrived bars and keep existing data.


Original sources:

ShiftedBars calculation: https://github.com/rosasurfer/mt4-mql/blob/9efa10dd4a603ce0d78d4d528f0a0c3474ac74d2/mql4/include/core/indicator.mqh#L169

ShiftIndicatorBuffer():   https://github.com/rosasurfer/mt4-expander/blob/47dbb1a57439b233117f7b141c6b4b010f39e57a/src/util/helper.cpp#L215

the actual shifting:        https://github.com/rosasurfer/mt4-mql/blob/9efa10dd4a603ce0d78d4d528f0a0c3474ac74d2/mql4/indicators/ALMA.mq4#L294

JHawk
55
JHawk  
alphatrading:

It is a tricky issue and a limitation of offline charts. A bullet-proof workaround is possible but not for the faint-hearted. First we need to clarify what happens in offline charts opposite to regular charts.

...


Now look at this: What a great answer!


Many many thanks for your huge work.

I have to digest it.... and it will surely help.


German: Vielen vielen Dank :-) Ich muss das erst mal verdauen. Aber es hilft massiv, selbst wenn ich nicht den gleichen Weg gehe. Aber ich verstehe das Ganze jetzt viel besser. Und es hilft auch zu wissen, dass ich mit meiner Anforderung nicht alleine bin.

Grüße aus der Nähe von Frankfurt


JHawk

Stanislav Korotky
23361
Stanislav Korotky  
alphatrading:

Final missing piece is the function ShiftIndicatorBuffer(). While it is possible to implement it in MQL for performance reasons it is strongly recommended to move the code to a DLL where memory can be moved directly without using loops:


MQL import statement:


Why the custom DLL? Why not to use imports for memmove and memset from the system kernel DLL directly? Then ShiftIndicatorBuffer can be coded in MQL.

alphatrading
82
alphatrading  
Stanislav Korotky:

Why the custom DLL? Why not to use imports for memmove and memset from the system kernel DLL directly? Then ShiftIndicatorBuffer can be coded in MQL.

Because (1) the DLL ensures more strict error checking and (2) the signature

bool   ShiftIndicatorBuffer(double buffer[], int bufferSize, int bars, double emptyValue);

is much easier to understand and to use then memmove/memset. I don't want to put system calls and potentially dangerous pointer arythmetics into MQL code. Last but not least, maintenance is an important factor, too.

Of course you are free to run a loop in MQL and skip the DLL alltogether. This is just one of many possibilities to do it.

12
To add comments, please log in or register