CopyTicks() Inconsistent Behavior

Andriy Moraru  

Hello all!

I can't make sense of why the CopyTicks() returns what it returns.

I use this code:

    MqlTick ticks_array[];
    int n = CopyTicks(Symbol(), ticks_array, COPY_TICKS_ALL, 0, N);
    if (n != N)
    {
        Print("Waiting for ticks...");
        return prev_calculated;
    }
    Print("Copied ", n, " ticks. First tick: ", TimeToString(ticks_array[0].time), ". Last tick: ", TimeToString(ticks_array[n - 1].time), ".");
When N = 100000, the result is the following:
2022.08.22 16:51:16.763    PNF (BTCUSD,H1)    Copied 100000 ticks. First tick: 2022.08.22 14:30. Last tick: 2022.08.22 16:51.

Which makes sense - it copied ticks from the current time till whatever time is for the 100000th tick.

But when I run it with N = 200000, I get the following:

2022.08.22 16:49:37.701    PNF (BTCUSD,H1)    Copied 200000 ticks. First tick: 2022.08.22 00:00. Last tick: 2022.08.22 04:16.

For some reason, it requests ticks not starting from the current time, but from 04:16 of today and arrives exactly at 00:00 for the 200000th tick.

What gives?

The help file says that calling CopyTicks() with from = 0, will copy count ticks from the current time. Or do I misunderstand what it says?

Any help will be appreciated!

Lorentzos Roussos  
Andriy Moraru:

Hello all!

I can't make sense of why the CopyTicks() returns what it returns.

I use this code:

When N = 100000, the result is the following:

Which makes sense - it copied ticks from the current time till whatever time is for the 100000th tick.

But when I run it with N = 200000, I get the following:

For some reason, it requests ticks not starting from the current time, but from 04:16 of today and arrives exactly at 00:00 for the 200000th tick.

What gives?

The help file says that calling CopyTicks() with from = 0, will copy count ticks from the current time. Or do I misunderstand what it says?

Any help will be appreciated!

Building a Point and figure chart ? :)

I was fighting with the ticks mechanism recently too so , there are some issues , on our end not MT5.

The system will querry the server for any date that is not in the downloaded tick data .

Try step loading small batches of ticks based on what the symbol has already (on its rates array ,on the native timeframe).

The following produces inverted ticks which you will need to sort based on time_msc and , it temporarily stores the batch on ticks[] which you then need to add to your 

collection array . It was done this way because i were developing an ask chart so i had to scrub any non TICK_FLAG_ASK ticks.

Now once the ticks you want are downloaded they will be almost instant to access from consecutive loads .

You can (should) also setup date limits (when reached stop loading) , and or tick amount limits .

int bars_step=400;//bars step
int step_timeout=2;//Step timeout in seconds
//+------------------------------------------------------------------+
//| Expert initialization function                                   |
//+------------------------------------------------------------------+
bool has_timer=false;
MqlTick ticks[];
datetime step_from,step_to;
int OnInit()
  {
//--- create timer
   //EventSetTimer(60);
   ArrayFree(ticks);
   step_from=0;step_to=0;
   has_timer=EventSetMillisecondTimer(300);
   if(!has_timer){return(INIT_FAILED);}
//---
   return(INIT_SUCCEEDED);
  }
//+------------------------------------------------------------------+
//| Expert deinitialization function                                 |
//+------------------------------------------------------------------+
void OnDeinit(const int reason)
  {
//--- destroy timer
   //EventKillTimer();
   if(has_timer){EventKillTimer();}
   ArrayFree(ticks);
  }
//+------------------------------------------------------------------+
//| Expert tick function                                             |
//+------------------------------------------------------------------+
void OnTick()
  {
//---
   
  }
//+------------------------------------------------------------------+
//| Timer function                                                   |
//+------------------------------------------------------------------+
void OnTimer()
  {
        /*
        okay so first download fails with 4401 
        then at some point when the range is ready its available
        */
//---
  EventKillTimer();
  //load ticks process 
    //copy ticks from the first bar 
      int barz_total=Bars(_Symbol,_Period);
      int maxbarsinchart=(int)TerminalInfoInteger(TERMINAL_MAXBARS);
      if(barz_total>maxbarsinchart){barz_total=maxbarsinchart;}
      //stepwise load
      if(barz_total>0){ 
      //first step 
        if(step_from==0&&step_to==0){
        step_to=iTime(_Symbol,_Period,0);
        step_from=(datetime)(step_to-bars_step*PeriodSeconds(_Period));
        }
      //request 
        ResetLastError();
        ArrayFree(ticks);
        //Print("Request "+TimeToString(step_from,TIME_DATE|TIME_MINUTES|TIME_SECONDS)+" - "+TimeToString(step_to,TIME_DATE|TIME_MINUTES|TIME_SECONDS));
        int ticks_total=CopyTicksRange(_Symbol,ticks,COPY_TICKS_INFO,((ulong)step_from*1000),((ulong)step_to*1000));
        //some delay occurs here 
          int error=GetLastError();
          //if not yet ready 
            if(error==4401&&ticks_total==-1){
            EventSetTimer(step_timeout);
            Print("wait for sync");
            }
          //if ready skip to next part
            else if(error==0&&ticks_total>=0){
            Print(TimeToString(step_from,TIME_DATE|TIME_MINUTES|TIME_SECONDS)+" - "+TimeToString(step_to,TIME_DATE|TIME_MINUTES|TIME_SECONDS)+" ("+IntegerToString(ticks_total)+")ticks");
            step_to=step_from;
            step_from=(datetime)(step_to-bars_step*PeriodSeconds(_Period));
            if(ticks_total==0){
            Print("Finished ");ExpertRemove();
            }else{EventSetMillisecondTimer(40);}
            }
            else{Print("Unknown error "+IntegerToString(error)+" ticks("+IntegerToString(ticks_total)+")");ExpertRemove();}
      }else{Print("No bars in symbol");}
      
  }
Andriy Moraru  
Lorentzos Roussos #:

Building a Point and figure chart ? :)

I was fighting with the ticks mechanism recently too so , there are some issues , on our end not MT5.

The system will querry the server for any date that is not in the downloaded tick data .

Try step loading small batches of ticks based on what the symbol has already (on its rates array ,on the native timeframe).

The following produces inverted ticks which you will need to sort based on time_msc and , it temporarily stores the batch on ticks[] which you then need to add to your 

collection array . It was done this way because i were developing an ask chart so i had to scrub any non TICK_FLAG_ASK ticks.

Now once the ticks you want are downloaded they will be almost instant to access from consecutive loads .

You can (should) also setup date limits (when reached stop loading) , and or tick amount limits .

So, you suggest doing the copying with CopyTicksRange() rather than with CopyTicks() and in small portions, right?

Why tie it to the current chart's period? Aren't ticks completely independent of other chart history?

Enrique Dangeroux  
Andriy Moraru #:

So, you suggest doing the copying with CopyTicksRange() rather than with CopyTicks() and in small portions, right?

Why tie it to the current chart's period? Aren't ticks completely independent of other chart history?

Personally i do not see any usage for CopyTicks as a fixed number of ticks messes with my mind.

I use CopyticksRange for both indicator and combat advisor and never had any problem, also no problem loading all ticks at once(millions of them) instead of batches, i also do not see why you'd ever want to do that. 

There is one thing to account for and that is ticks not downloaded, this can be handled similar to what you would do with bars not downloaded yet.

Lorentzos Roussos  
Andriy Moraru #:

So, you suggest doing the copying with CopyTicksRange() rather than with CopyTicks() and in small portions, right?

Why tie it to the current chart's period? Aren't ticks completely independent of other chart history?

Yes .  

You can stop polling the server (or the history) once a specific date in the past has been reached , or , an amount of ticks .

This way you can also provide feedback on the progress of the "tick collection" instead of just telling the user to wait for an unspecified amount of time.

If you go from the past to the future in your requests , then you can use the chart history as a guide for the first date you are going to request.

So if the default chart starts from 2020 your users won't expect to see 2019 anyway.

Andriy Moraru  
Enrique Dangeroux #:

Personally i do not see any usage for CopyTicks as a fixed number of ticks messes with my mind.

I use CopyticksRange for both indicator and combat advisor and never had any problem, also no problem loading all ticks at once(millions of them) instead of batches, i also do not see why you'd ever want to do that. 

There is one thing to account for and that is ticks not downloaded, this can be handled similar to what you would do with bars not downloaded yet.

I see... The reason I wanted to use CopyTicks rather than CopyTicksRange is finer control over how many ticks actually get copied. With date ranges, there can be any number of ticks in between.

I will try the proposed solution with CopyTicksRange and see if it works for me.

However, the question about CopyTicks's inconsistency still stands. Why does it behave the way it does? Is it a bug or rather some undocumented feature?

Enrique Dangeroux  
Andriy Moraru #:

I see... The reason I wanted to use CopyTicks rather than CopyTicksRange is finer control over how many ticks actually get copied. With date ranges, there can be any number of ticks in between.

I will try the proposed solution with CopyTicksRange and see if it works for me.

However, the question about CopyTicks's inconsistency still stands. Why does it behave the way it does? Is it a bug or rather some undocumented feature?

I do not use it, so i can not tell. Also note that the cache for copyticks is  limited and can be a slow call.

The question is, why do you want a fixed number of ticks? The only reason i can think of is analysing the last x number of ticks. Other then that if you want to copy to a already existing pile of ticks you have to manange the synchronisation yourself.

With Range copy you copy, split them up in desired chunks for example if you want to have 2000 tick bars. To update copy ticks from last tick time to current time and feed it to the chunker.

Fernando Carreiro  

Here is some food for thought ...

I prefer using CopyTicks instead of CopyTicksRange even when I want is a time range.

As the OP has pointed out, I want to have control of how many ticks I read at a time so that I do not have to have a very large array to handle. Instead, I read the time range in chunks.

I seed the first read of CopyTicks with the starting time in milliseconds of the required time range, and process the batch of returned ticks, but filter out the last millisecond timestamp of the array. I don’t process any ticks belonging to the last timestamp. I then use that timestamp as a seed point for the next CopyTicks.

It sometimes happens that you get two (or more) consecutive ticks with the same millisecond timestamp, but with different quote data. That is why I don’t process any ticks with the last timestamp of the array and only process them in the next round. This prevents me from accidentality duplicating these runs of ticks when using that timestamp for the next batch.

This repetition continues until the end of the time range is reached or no more ticks are available.

Doing it it this way may seem more complex, but gives you more control over the RAM usage, and it may even be faster to process then using CopyTicksRange , but I have never timed it.

EDIT: If you want to use this method for reading/processing ALL available tick data, then seed the first call with a "from" value of 1 and not 0. This is important!

Andriy Moraru  

Fernando Carreiro #:

EDIT: If you want to use this method for reading/processing ALL available tick data, then seed the first call with a "from" value of 1 and not 0. This is important!

Why seeding from 1 and not from 0 is important?

Fernando Carreiro  
Andriy Moraru #: Why seeding from 1 and not from 0 is important?

Because if you use "0" instead of "1" , it will fetch the data starting with the most recent tick data, going back "count" ticks, instead of starting from the oldest tick going forward.

This is also stated in the documentation:

"from [in]   The date from which you want to request ticks. In milliseconds since 1970.01.01. If from=0, the last count ticks will be returned."

This can also be seen in the example code given in the documentation:

//--- Requesting the tick history since 1970.01.01 00:00.001 (parameter from=1 ms)
      int received=CopyTicks(_Symbol,tick_array,COPY_TICKS_ALL,1,getticks);
Andriy Moraru  
Fernando Carreiro #:

Because if you use "0" instead of "1" , it will fetch the data starting with the most recent tick data, going back "x" ticks, instead of starting from the oldest tick going forward.

This is also stated in the documentation:

"from [in]   The date from which you want to request ticks. In milliseconds since 1970.01.01. If from=0, the last count ticks will be returned."

This can also be seen in the example code given in the documentation:

Oh, I see. Thank you!
Reason: