Mysterious stock indicator, another bug found in the new version of Pokemon Go - Trading Systems

fxsaber 2016.09.27 17:00 #171

fxsaber:
If one needs ONLY ribbon (COPY_TICKS_TRADE - time_msc, last, volume and flags), this solution is completely suitable - no bugs detected.

No bugs found! Another bug revealed

Forum on trading, automated trading systems and testing trading strategies

Mysterious stock indicator

fxsaber, 2016.09.27 18:32

All the brakes seem to be on when the input from CopyTicks is not zero.

Seems to be a very crooked implementation of CopyTicks in this mode, even if ticks since last call are requested. Seems like it should fly, but it doesn't.

Errors, bugs, questions MetaTrader 5 Strategy Tester: From theory to practice

Renat Fatkhullin 2016.09.27 23:27 #172

Wait for this week's beta, where we've made a number of improvements in ticks and tumblr updates.

fxsaber 2016.09.28 11:26 #173

Renat Fatkhullin:
Wait for this week's beta, where we've made a number of improvements to the ticks and tumblr updates.

1432 - lots of bugs resolved. Thank you!

fxsaber 2016.09.29 07:39 #174

fxsaber:
1432 - Many bugs solved. Thank you!

But not all.

If the added history is compared to the actual history, there are discrepancies in COPY_TICKS_ALL mode (TRADE and INFO - no problem). EA

#include <TypeToBytes.mqh> // https://www.mql5.com/ru/code/16280

string GetTickFlag( uint tickflag )
{
  string flag = "";

#define  TICKFLAG_MACRO(A) flag += ((bool)(tickflag & TICK_FLAG_##A)) ? " TICK_FLAG_" + #A : "";
  TICKFLAG_MACRO(BID)
  TICKFLAG_MACRO(ASK)
  TICKFLAG_MACRO(LAST)
  TICKFLAG_MACRO(VOLUME)
  TICKFLAG_MACRO(BUY)
  TICKFLAG_MACRO(SELL)
#undef  TICKFLAG_MACRO

  if (flag == "")
    flag = " FLAG_UNKNOWN (" + (string)tickflag + ")";
     
  return(flag);
}

#define  TOSTRING(A) " " + #A + " = " + (string)Tick.A

string TickToString( const MqlTick &Tick )
{
  return(TOSTRING(time) + "." + (string)IntegerToString(Tick.time_msc %1000, 3, '0') +
         TOSTRING(bid) + TOSTRING(ask) + TOSTRING(last)+ TOSTRING(volume) + GetTickFlag(Tick.flags));
}

// Дописывает свежие тики после предыдущего запуска
int AddFreshTicks( MqlTick &Ticks[], const string Symb = NULL, const uint flags = COPY_TICKS_ALL )
{
  int Res = 0;
  const int Amount = ArraySize(Ticks);
  
  MqlTick NewTicks[];  
  const int NewAmount = (Amount == 0) ? CopyTicks((Symb == NULL)? Symbol() : Symb, NewTicks, flags, (ulong)(TimeCurrent() - 100) * 1000) :
                                        CopyTicks((Symb == NULL)? Symbol() : Symb, NewTicks, flags, Ticks[Amount - 1].time_msc);
  
  if (NewAmount > 0)
  {
    if (Amount > 0)
    {
      // Взяли крайнее время из предыдущей истории
      const long LastTime = Ticks[Amount - 1].time_msc;
      
      int Count = 1;
      
      // Находим (Count) в предыдушей истории количество тиков со временем LastTime
      for (int i = Amount - 2; i >= 0; i--)
      {
        if (Ticks[i].time_msc < LastTime)
          break;
          
        Count++;
      }

      if ((Count < Amount) && (Count < NewAmount))      
        Res = ArrayCopy(Ticks, NewTicks, Amount, Count);
    }
    else
      Res = ArrayCopy(Ticks, NewTicks);
  }
  
  return(Res);
}

#define  TOSTRING2(A) #A + " = " + (string)(A) + " "

template <typename T>
bool ArrayEqual( const T &Array1[], const T &Array2[] )
{
  const int Amount = MathMin(ArraySize(Array1), ArraySize(Array2));
  bool Res = (Amount > 0);

  if (Res)
    for (int i = 0; i < Amount; i++)
      if (_R(Array1[i]) != Array2[i]) // https://www.mql5.com/ru/code/16280
      {
        Print(TOSTRING2(i) + TOSTRING2(ArraySize(Array1)) +TOSTRING2(ArraySize(Array2)));
        Print(TOSTRING2(TickToString(Array1[i])) + "\n" + TOSTRING2(TickToString(Array2[i])) + "\n");
        
        Res = false;

        break;
      }

  return(Res);
}

void OnTick( void )
{
 static MqlTick PrevTicks[];
  
  // Дописываем свежие тики после предыдущего вызова
  AddFreshTicks(PrevTicks, _Symbol, COPY_TICKS_ALL);
  
  MqlTick Ticks[];
  
  if (ArraySize(PrevTicks) > 0)
  {
    // Взяли историю тиков
    Print(CopyTicks(_Symbol, Ticks, COPY_TICKS_ALL, PrevTicks[0].time_msc, 100000));
    
    // Проверка на совпадение собираемой истории с самой историей
    Print(ArrayEqual(Ticks, PrevTicks) ? "Equal" : "Not Equal");
  }
}

Result

2016.09.29 10:36:23.722 Test10 (Si-12.16,M1)    Not Equal
2016.09.29 10:36:23.722 Test10 (Si-12.16,M1)    
2016.09.29 10:36:23.722 Test10 (Si-12.16,M1)    TickToString(Array2[i]) =  time = 2016.09.29 10:36:20.547 bid = 64353.0 ask = 64354.0 last = 64353.0 volume = 4 TICK_FLAG_BID 
2016.09.29 10:36:23.722 Test10 (Si-12.16,M1)    TickToString(Array1[i]) =  time = 2016.09.29 10:36:20.546 bid = 64353.0 ask = 64354.0 last = 64353.0 volume = 1 TICK_FLAG_LAST TICK_FLAG_VOLUME TICK_FLAG_SELL 
2016.09.29 10:36:23.722 Test10 (Si-12.16,M1)    i = 57 ArraySize(Array1) = 59 ArraySize(Array2) = 58 
2016.09.29 10:36:23.722 Test10 (Si-12.16,M1)    59

fxsaber 2016.09.29 08:15 #175

fxsaber:

But not all.

If the added history is compared to the actual history, there are discrepancies in COPY_TICKS_ALL mode (TRADE and INFO - no problem). EA

Result

I logged the code above and found out the reasons. If CopyTicks (from > 0) receives ticks up to the freshest one, it may miss some.

Example.

Requested ticks with from = 2016.09.29 11:05:55.564. Got three ticks in reply

2016.09.29 11:05:58.724 Test10 (Si-12.16,M1)    2:  time = 2016.09.29 11:05:55.580 bid = 64380.0 ask = 64382.0 last = 64381.0 volume = 4 TICK_FLAG_BID TICK_FLAG_ASK
2016.09.29 11:05:58.724 Test10 (Si-12.16,M1)    1:  time = 2016.09.29 11:05:55.576 bid = 64379.0 ask = 64381.0 last = 64381.0 volume = 4 TICK_FLAG_LAST TICK_FLAG_VOLUME TICK_FLAG_BUY
2016.09.29 11:05:58.724 Test10 (Si-12.16,M1)    0:  time = 2016.09.29 11:05:55.564 bid = 64379.0 ask = 64381.0 last = 64380.0 volume = 1 TICK_FLAG_BID TICK_FLAG_ASK

Some time later I've requested the tick history from afar and got a tick, which CopyTicks missed before

2016.09.29 11:05:58.732 Test10 (Si-12.16,M1)    time = 2016.09.29 11:05:55.579 bid = 64380.0 ask = 64382.0 last = 64381.0 volume = 16 TICK_FLAG_LAST TICK_FLAG_VOLUME TICK_FLAG_BUY

Such a bug!

Seems to be some kind of conflict of writing to and reading from the tick database in parallel.

MetaTrader 5 Python User Services , new functionality Custom symbols. Errors, bugs,

fxsaber 2016.09.30 13:10 #176

Another bug, now in all modes COPY_TICKS_*

Forum on trading, automated trading systems and testing trading strategies

Mysterious stock indicator

fxsaber, 2016.09.30 15:09

Was able to localise one of the bugs causing divergence in the indicator. It's about CopyTicks again.

It turns out that if we collect tick history in parts, it may not coincide with the real history. The Expert Advisor shows it

#include <TypeToBytes.mqh> // https://www.mql5.com/ru/code/16280

long LastTime = 0; // time_msc-время последнего тика (самого свежего), полученного из истории
int Count = 0;     // Количество тиков в последенем запросе, у которых time_msc == LastTime

// Возвращает следующие тики (после предыдущего вызова)
int GetFreshTicks( MqlTick &Ticks[], const uint flags = COPY_TICKS_TRADE, const uint count = 100000 )
{
  int Res = 0;

  MqlTick NewTicks[];
  const int NewAmount = CopyTicks(Symbol(), NewTicks, flags, LastTime, count);

  if ((NewAmount > 0) && (Count < NewAmount))
  {
    Res = ArrayCopy(Ticks, NewTicks, 0, Count);

    // Взяли крайнее время из текущей истории
    LastTime = Ticks[Res - 1].time_msc;
    Count = 1;

    // Находим (Count) в текущей истории количество тиков со временем LastTime
    for (int i = Res - 2; i >= 0; i--)
    {
      if (Ticks[i].time_msc < LastTime)
        break;

      Count++;
    }
  }
  
  return(ArrayResize(Ticks, Res));
}

// Сравнение двух массивов
template <typename T>
bool ArrayEqual( const T &Array1[], const T &Array2[] )
{
  const int Amount = MathMin(ArraySize(Array1), ArraySize(Array2));
  bool Res = (Amount > 0);

  if (Res)
    for (int i = 0; i < Amount; i++)
      if (_R(Array1[i]) != Array2[i]) // https://www.mql5.com/ru/code/16280
      {
        Res = false;
        
        ExpertRemove();

        break;
      }

  return(Res);
}

void OnTick()
{
  // возьмем тики с начала утренней сессии
  Count = 0;
  LastTime = (TimeCurrent() - (TimeCurrent() % (24 * 3600))) * 1000;
  
  MqlTick Ticks[];    // История, собранная по частям
  MqlTick NewTicks[]; // массив для следующей части тиков
  
  // Собираем историю по частям  
  while (GetFreshTicks(NewTicks, COPY_TICKS_TRADE, 100000) > 0)
    ArrayCopy(Ticks, NewTicks, ArraySize(Ticks));
    
  if (ArraySize(Ticks) > 0)    
  {
    // Взяли ВСЮ историю тиков
    Print(CopyTicks(_Symbol, NewTicks, COPY_TICKS_TRADE, Ticks[0].time_msc, 10000000)); // 10000000 - большое число, чтобы все выкачать.
    
    // Проверка на совпадение собранной по частям истории с самой историей
    Print(ArrayEqual(NewTicks, Ticks) ? "Equal" : "Not Equal");
  }    
}

The result

2016.09.30 16:02:54.661 Test (Si-12.16,M1)      Not Equal
2016.09.30 16:02:54.661 Test (Si-12.16,M1)      ExpertRemove() function called
2016.09.30 16:02:54.621 Test (Si-12.16,M1)      333740
2016.09.30 16:02:54.121 Test (Si-12.16,M1)      Equal
2016.09.30 16:02:54.071 Test (Si-12.16,M1)      333736
2016.09.30 16:02:53.791 Test (Si-12.16,M1)      Equal
2016.09.30 16:02:53.741 Test (Si-12.16,M1)      333723

This EA also shows a weak bug. I found out that the history collected in parts may be missing some parts that last several minutes. Only a concise and clear reproduction in the form of code has not been invented. And there is no point in posting complex ones, because no one will even look.

In general, there is no way to defeat bugs in CopyTicks. And note that the Expert Advisor works in tape mode (COPY_TICKS_TRADE). So, it cannot even work with ribbon.

I'm puzzled... EAs don't problem with ibarshift getting Increasing lot size

[Deleted] 2016.09.30 14:46 #177

fxsaber:
Another bug, now in all modes COPY_TICKS_*

Have you tried getting ticks both starting from a certain point and a certain, e.g. fixed number?

From the code, it looks like a certain number (100000) from the last moment. And what if you get just N ticks. Will there be skips too?

Let me tell you right away, I haven't experimented with ticks much myself yet...

CopyTicksRange () does not Questions from Beginners MQL4 Libraries: Tester Control

fxsaber 2016.09.30 14:58 #178

Alexey Kozitsyn:

Have you tried getting ticks both starting from a certain point and a certain, e.g. fixed number?

From the code, it looks like a certain number (100000) from the last moment. And what if you get just N ticks. Will there be skips too?

Let me tell you right away, I haven't experimented with ticks much myself yet...

Tried it.

Today they are promising a new build for the demo. So we have to wait.

[Deleted] 2016.09.30 15:10 #179

fxsaber:

Tried it.

Today they promise a new build for demo. That's why I have to wait.

I hope, CopyTicks() will be corrected.

By the way, ever since CopyTicks() appeared I asked developers to add overloading function like in other Copy...() functions. They told me they will. И... silence...

fxsaber 2016.09.30 16:08 #180

Alexey Kozitsyn:

By the way, ever since CopyTicks() appeared, I asked developers to add overloading function, like for other Copy...() functions. They told me they will. И... silence...

You may add your own overloads.

Testing 'CopyTicks' - page 18