Libraries: CBitBuffer Class - Data Serialization in MQL5

[Automated-Trading]

CBitBuffer Class - Data Serialization in MQL5:

A class for reading and writing individual bits or bit sequences to and from a buffer.

Author: amrali

[amrali]

Update 8 Oct 2025

Efficient compression/decompression of ticks into bytes buffer, utilizing Delta + VLQ encoding.

TicksShort.mqh

• Alternative implementation of 'TicksShort' tick compression library by @fxsaber.

TicksShortLite.mqh

• Standalone / lightweight version (no #include dependencies)
• Ideal for embedding in small projects or performance tests

[fxsaber]

amrali #:

Efficient compression/decompression of ticks into bytes buffer, utilizing Delta + VLQ encoding.

For practical use, the ability to perform sequential decompression and compression is lacking.

Please consider adding this functionality.

bool Decompress( const uchar &TickIn[], int &PosFrom, MqlTick &TickOut );
bool Compress( const MqlTick &TickIn, uchar &TickIn[] );

[amrali]

fxsaber #:

For practical use, the ability to perform sequential decompression and compression is lacking.

Please consider adding this functionality.

There are many other ideas:

• extend this to include volume, last price and flags fields too (compress full tick)
• stream compression — e.g. append compressed chunks continuously to the same file while keeping it readable later (useful for tick logging in real time)
• Higher compresssion: using delta-of-delta encoding for timestamps, and run-length encoding (RLE) for the flags. 

But, I prefer to keep it simple and lite.

[fxsaber]

amrali #:

But, I prefer to keep it simple and lite.

There is no need to save flags - they are obtained by comparing two adjacent ticks.

I need the minimum functionality I described above. I use it in two solutions: the first and second.

I'll try to figure out the source code and add the necessary functionality. Thank you.

[amrali]

fxsaber #:

For practical use, the ability to perform sequential decompression and compression is lacking.

Please consider adding this functionality.

Could you please explain what do you mean by sequential decompression, give an example.

[fxsaber]

amrali #:
Could you please explain what do you mean by sequential decompression, give an example.

#include <fxsaber\TicksShort\TicksShort.mqh> // https://www.mql5.com/ru/code/61126

void OnStart()
{  
  MqlTick Ticks1[];  
  const int Size = CopyTicks(_Symbol, Ticks1, COPY_TICKS_ALL, 0, 5);
  ArrayPrint(Ticks1);  

  TICK_SHORT Ticks2[];
  ArrayResize(Ticks2, Size << 1);
  
  TICKS_SHORT TicksShort;
  int Amount = 0;  
  
  // Element-by-element compression.
  for (int i = 0; i < Size; i++)
    if (!TicksShort.Compress(Ticks1[i], Ticks2[Amount++]))
      TicksShort.Compress(Ticks1[i], Ticks2[Amount++]);
      
  MqlTick Ticks3[];
  ArrayResize(Ticks3, Size);

  TicksShort.Reset(TicksShort.GetDigits());

  // Elemental decompression.
  for (int i = 0, j = 0; i < Size; i++)
    if (!TicksShort.Decompress(Ticks2[j++], Ticks3[i]))
      TicksShort.Decompress(Ticks2[j++], Ticks3[i]);
  
  ArrayPrint(Ticks3);
  
  MqlTick Ticks4[];
  ArrayResize(Ticks4, Size);

  TicksShort.Reset(TicksShort.GetDigits());

  // Elemental decompression (alternative version).
  for (int i = 0, Count = 0; i < Amount; i++)
    if (TicksShort.Decompress(Ticks2[i], Ticks4[Count]))
      Count++;

  ArrayPrint(Ticks4);  
}

It's necessary to be able to write additional ticks to the uchar-array and read ticks sequentially from the uchar-array.

In the example I do this with the TICK_SHORT-array.

[fxsaber]

I really liked the idea and implementation! Please make the following changes.

      struct HEADER
      {
       int Amount;
       
       ulong LastTime;
      };
      
      union HEADER_UNION
      {
       HEADER Header;
       
       uchar Array[sizeof(HEADER)];
      } Header = {{nticks, ticks[nticks - 1].time_msc}};
      
      int pos = ::ArrayCopy(buf, Header.Array);
/*      
      int pos = 0;

      VLQWrite(buf, pos, nticks);
*/      
      VLQWrite(buf, pos, digits);

This will allow you to modify these two values ​​in the future without affecting the rest of the data.

[amrali]

fxsaber #:
I really liked the idea and implementation! Please make the following changes.

This will allow you to modify these two values ​​in the future without affecting the rest of the data.

If you want the uchar buffer to be filtered by time, you should add a header (i.e, metadata) per each block appended to the file (multiple streams).

   struct TickBlockHeader
   {
      ulong first_time;
      ulong last_time;
      uint  count;
      uint  size;
   };
   
   //==============================================================
   // Write compressed block to file
   //==============================================================
   bool AppendBlock(const string file, const MqlTick &ticks[])
   {
      int hfile = FileOpen(file, FILE_READ | FILE_WRITE | FILE_BIN);
      if(hfile == INVALID_HANDLE)
         return(false);

      uchar cbuf[];
      Compress(ticks, cbuf);
      uint sz = ArraySize(cbuf);

      TickBlockHeader hdr;
      hdr.first_time = ticks[0].time_msc;
      hdr.last_time  = ticks[ArraySize(ticks) - 1].time_msc;
      hdr.count      = ArraySize(ticks);
      hdr.size       = sz;

      FileSeek(hfile, 0, SEEK_END);
      FileWriteStruct(hfile, hdr);
      FileWriteArray(hfile, cbuf, 0, sz);
      FileClose(hfile);
      return(true);
   }

You will get independent blocks with time range headers into a single compressed bin file.

This allows partial decompression of specific time ranges from file

storage.ReadRange("ticks.bin", from_time, to_time, MqlTick &out_ticks[]);

You can apply a similar technique to filter a compressed ticks (uchar) array in memory: 

StructToCharArray(hdr, out_buf);

[fxsaber]

amrali #:

If you want the uchar buffer

#include <amrali\CBitBuffer\TicksShort\TicksShortLite.mqh> // https://www.mql5.com/en/code/61728

TICKS_SHORT TicksShort;
uchar TicksOut[];

void OnTick()
{
  MqlTick Tick;
  
  if (SymbolInfoTick(_Symbol, Tick))
    TicksShort.Compress(Tick, TicksOut);
}

void OnDeinit( const int ) { FileSave(__FILE__, TicksOut, FILE_COMMON); }

[amrali]

fxsaber #:

  if (SymbolInfoTick(_Symbol, Tick))
    TicksShort.Compress(Tick, TicksOut);

Delta encoding for MQL ticks means storing only the differences (deltas) between consecutive tick values instead of the full numbers.

For example, instead of saving each tick’s full Time_msc, Bid, and Ask, we record how much each changed from the previous tick.

Since market data changes slightly between ticks, these deltas are usually small, allowing them to be stored with fewer bytes (especially when combined with variable-length encoding).

Delta encoding = compressing tick data by saving changes, not absolute values.

Each stored tick is dependant on the previous tick.

To log ticks in real time as posted in the example code, collect ticks in a temp MqlTick[] array. Flush to the compressed tick storage for example every 5000-10000 ticks.

      storage.AppendBlock("ticks.bin", inTicks);

Edit: You can modify the 'CTickStorage' class code to adopt memory-based instead of file-based storage (stream compression).