Is it possible to work with live ticks in MQL5 indicators? - page 3

[Fernando Carreiro]

Samuel Manoel De Souza #: I was to say that, but i though the documentation was enough to proof the point when it says "  If the queue already contains the NewTick event or this event is in the processing stage, then the new NewTick event is not added to mql5 application queue. "

@Dominik Egert is correct in stating that the NewTick event and Calculate event are different events. So, your reference does not prove the point.

I originally thought that the OnCalculate worked the same way as OnTick, but now I am starting to wonder about it.

[Samuel Manoel De Souza]

Dominik Egert #:

This passage is related to OnTick event handling, not OnCalculate.

OnCalculate, as I will show with code, is called for each tick arrived, and these calls are queued and processed one by one, without skipping any ticks.

It says, all events

[Fernando Carreiro]

Samuel Manoel De Souza #: It says, all events

No, it does not say "all" events. it specifically mentions that only the NewTick, ChartEvent and Timer events are the only ones that don't queue up until the previous one has been processed.

"A program gets events only from the chart it is running on. All events are handled one after another in the order of their receipt. If the queue already contains the NewTick event or this event is in the processing stage, then the new NewTick event is not added to mql5 application queue. Similarly, if the ChartEvent is already in an mql5 program queue or such an event is being handled, then a new event of this type is not placed into a queue. Timer event handling is processed in the same way – if the Timer event is already in the queue or is being handled, no new timer event is set into a queue."

Obviously, this can be open for different interpretations, but I'm beginning to side on @Dominik Egert's interpretation that the Calculate event does in fact trigger every single event, unlike these three.

[Samuel Manoel De Souza]

Fernando Carreiro #:

@Dominik Egert is correct in stating that the NewTick event and Calculate event are different events. So, your reference does not prove the point.

I originally thought that the OnCalculate worked the same way as OnTick, but now I am starting to wonder about it.

There are diferences, but none of them will be called for every tick. I did test in 2022.

And the point is that is not effcicient to call it tick by tick, due to the indicators not run in a separete thread.

[Samuel Manoel De Souza]

Fernando Carreiro #:

No, it does not say "all" events. it specifically mentions that only the NewTick, ChartEvent and Timer events are the only ones that not queue up until the previous one has been processed.

"A program gets events only from the chart it is running on. All events are handled one after another in the order of their receipt. If the queue already contains the NewTick event or this event is in the processing stage, then the new NewTick event is not added to mql5 application queue. Similarly, if the ChartEvent is already in an mql5 program queue or such an event is being handled, then a new event of this type is not placed into a queue. Timer event handling is processed in the same way – if the Timer event is already in the queue or is being handled, no new timer event is set into a queue."

Obviously, this can be open for different interpretations, but I'm beginning to side on @Dominik Egert's interpretation that the Calculate event does in fact trigger every single event, unlike these three.

do not ignore the part i've marked in red.

[Fernando Carreiro]

Samuel Manoel De Souza #: There are diferences, but none of them will be called for every tick. I did test in 2022.

And the point is that is not effcicient to call it tick by tick, due to the indicators not run in a separete thread.

Yes, I too did test this a long time ago, but not recently.

It is currently the weekend and most markets are closed, but I did test @Dominik Egert's test indicator on cryptos, and it did seem to generate even more events than even the tick volume could account for.

So currently, I'm reevaluating my opinion, that it in fact might be processing every event and no losing any ticks. Once the markets open up again, I will test it in more volatile situations to verify.

[Fernando Carreiro]

Samuel Manoel De Souza #:do not ignore the part i've marked in red.

I am in fact ignoring it on purpose because that section is referring to how the events are sequenced.

It has nothing to do with the events being ignored if another one is being processed.

It refers to the default action and not the exceptions.

[Fernando Carreiro]

There is also a more complete explanation in the documentation which refers to the OnTick() handler only ...

NewTick

The NewTick event is generated if there are new quotes, it is processed by OnTick() of Expert Advisors attached. In case when OnTick function for the previous quote is being processed when a new quote is received, the new quote will be ignored by an Expert Advisor, because the corresponding event will not enqueued.

All new quotes that are received while the program is running are ignored until the OnTick() is completed. After that the function will run only after a new quote is received. The NewTick event is generated irrespective of whether automated trade is allowed or not ("Allow/prohibit Auto trading" button). The prohibition of automated trading denotes only that sending of trade requests from an Expert Advisor is not allowed, while the Expert Advisor keeps working.

The prohibition of automated trading by pressing the appropriate button will not stop the current execution of the OnTick() function.

[Fernando Carreiro]

We just have to run more tests when the Markets are open and come up with a more definitive answer to the issue.

[Dominik Egert]

Here is a more in depth testing indicator. - I guess, it would be possible to integrate more testing and debug output, but for a proof, this should serve the needs.

//+------------------------------------------------------------------+
//| Custom indicator initialization function                         |
//+------------------------------------------------------------------+
#property indicator_plots 0


// Global static variables
static int      seconds_initial_delay       = 30;
static int      seconds_bulk_delay          = 30;
static int      milliseconds_max_live_delay = 2500;
static datetime current_candle              = NULL;
static int      call_counter                = NULL;
static bool     wait_for_sync               = true;
static bool     dequeue_ticks               = true;

int OnInit()
{
    // Init random generator
    MathSrand(GetTickCount());

    // Return 
    return(INIT_SUCCEEDED);
}



void OnDeinit(const int reason)
{
    // Return
    return;
}



int OnCalculate(const int rates_total,
                const int prev_calculated,
                const datetime &time[],
                const double &open[],
                const double &high[],
                const double &low[],
                const double &close[],
                const long &tick_volume[],
                const long &volume[],
                const int &spread[])
{
    // Local init
    const int index = rates_total - 1;
    MqlTick current_tick;

    // Call tracking
    SymbolInfoTick(_Symbol, current_tick);
    printf("****** OnCalculate() ******\n Tick millisecond: %llu", current_tick.time_msc);


    // Sync execution to new period
    if(wait_for_sync)
    {
        if(current_candle == NULL)
        { 
            printf("Waiting for sync: %s", TimeToString(time[index] + PeriodSeconds(), TIME_DATE|TIME_MINUTES|TIME_SECONDS));
            current_candle = time[index]; 
            return(1);
        }
        else if(current_candle < time[index])
        { 
            wait_for_sync   = false; 
            current_candle  = NULL;
        }
        else
        { return(1); }
    }

    // First set of calculation
    const int _rates_total = (rates_total - (rates_total/10));

    // Initial calulation simulator
    if(prev_calculated < (_rates_total - 1))
    {
        // Circumvent terminal complaining about slow indicator
        const double    time_pre_period     = ((double)seconds_initial_delay) / ((double)_rates_total);
        const int       periods_per_second  = (int)(1.0 / time_pre_period) + 1;

        // Execution delay
        const ulong wait_timer = GetTickCount() + 1000;
        while(wait_timer > GetTickCount());

        // Print details
        printf("Stepping interval: %s tick_volume: %llu", TimeToString(time[index], TIME_DATE|TIME_MINUTES|TIME_SECONDS), tick_volume[index]);

        // Return
        return(MathMin((prev_calculated + periods_per_second), rates_total - 2));
    }

    // Bulk delay 
    if(prev_calculated < (rates_total - 1))
    {
        // Print details
        printf("Bulk interval: %s tick_volume: %llu", TimeToString(time[index], TIME_DATE|TIME_MINUTES|TIME_SECONDS), tick_volume[index]);

        // Execution delay
        const ulong wait_timer = GetTickCount() + (seconds_bulk_delay * 1000);
        while(wait_timer > GetTickCount());

        // Return
        return(rates_total);
    }

    // Dequeue all waiting ticks
    if(dequeue_ticks)
    {
        if(current_candle == NULL)
        { 
            printf("Dequeuing waiting ticks arrived in bulk interval: Current tick volume: %llu", tick_volume[index]);
            current_candle = time[index]; 
            call_counter++;
            return(rates_total);
        }
        else if(current_candle < time[index])
        {
            printf("All backlog ticks processed: Backlog size was: %i; Current tick volume: %llu", call_counter, tick_volume[index]);
            dequeue_ticks   = false;
            current_candle  = NULL;
            call_counter    = NULL;
        }
        else
        { 
            call_counter++;
            return(rates_total); 
        }
    }

    // New period
    call_counter = (current_candle < time[index]) ? 1 : (call_counter + 1);
    current_candle = time[index];

    // Live mode
    check_ticks(time[index]);

    // Execution delay
    ulong wait_timer = GetTickCount() + (MathRand() % milliseconds_max_live_delay);
    while(wait_timer > GetTickCount());

    // Print current call details
    printf("Period Time: %s; Tick Volume: %llu; Call counter: %i", TimeToString(time[index], TIME_DATE|TIME_MINUTES|TIME_SECONDS), tick_volume[index], call_counter); 

    // Return
    return(rates_total);
}


void check_ticks(const datetime period_now)
{
    // Static init
    static int  call_log_counter    = NULL;
    static long last_tick_timestamp = NULL;

    // Local init
    ResetLastError();
    MqlTick ticks[];
    const int tick_count = CopyTicks(_Symbol, ticks, COPY_TICKS_ALL, (period_now - PeriodSeconds()) * 1000, NULL);

    // Check result
    if(tick_count <= NULL)
    { 
        printf("Error receiving tick data: %i", _LastError);
        return; 
    }

    // Check call sequence
    if( (tick_count > 1)
     && (last_tick_timestamp > NULL) )
    {
        // Search for last timestamp
        int find_last_ptr = tick_count - 1;
        while( (find_last_ptr > NULL)
            && (ticks[find_last_ptr].time_msc > last_tick_timestamp) )
        { find_last_ptr--; }

        const int ticks_skipped = (tick_count - find_last_ptr) - 1;
        call_log_counter += (ticks_skipped > 1) ? ticks_skipped : -(call_log_counter > NULL);
        printf("Ticks received since last call: %i; last tick: %llu; Call backlog counter: %i", ticks_skipped, last_tick_timestamp, call_log_counter);

        ArrayPrint(ticks, _Digits, NULL, find_last_ptr++, 1, ARRAYPRINT_HEADER|ARRAYPRINT_LIMIT|ARRAYPRINT_ALIGN);
        for(; find_last_ptr < tick_count; find_last_ptr++)
        { ArrayPrint(ticks, _Digits, NULL, find_last_ptr, 1, ARRAYPRINT_ALIGN); }
    }

    // Save last tick timestamp
    last_tick_timestamp = ticks[tick_count - 1].time_msc;

    // Return
    return;
}

EDIT:

I have added a tick checking function to see if calls to OnCalculate are actually in sequence with the ticks received from the market. - I guess this will not be possible to convert to MQL4. Therefore I have added it as a function.

And fixed a bug...

EDIT2:

After testing the indicator on live markets (BTCUSD), I noticed some strange behaviours, and had to fix quite some issues. - Currently I dont know if this is working properly, as resutls I am getting are weird, to say the least. - But for sure, OnCalculate is called much more often than tick_volume is counted up, and CopyTicksRange is somewhat unreliable for copying current tick data. At least I thought. So I switched to CopyTicks and used that, now things get interesting.

OnCalculate seems to be called for every price change, and it seems not all ticks are reflected in CopyTicks result array. But I would guess, best is to see the results for yourself, and try to understand... - I am not even able to express exactly what is going on.