Libraries: CBitBuffer Class - Data Serialization in MQL5 - page 2

New comment
 
amrali #:
Delta encoding for MQL ticks means storing only the differences (deltas) between consecutive tick values instead of the full numbers.
You can't help but know that the original library has been using delta encoding from the very beginning. And that doesn't stop it from working. 
#include <fxsaber\TicksShort\TicksShort.mqh> // https://www.mql5.com/ru/code/61126

TICKS_SHORT TicksShort2;
TICK_SHORT TicksOut2[];

void OnTick()
{
  MqlTick Tick;
  
  if (SymbolInfoTick(_Symbol, Tick) &&
      !TicksShort2.Compress(Tick, TicksOut2[ArrayResize(TicksOut2, ArraySize(TicksOut2) + 1, 1e6) - 1]))
    TicksShort2.Compress(Tick, TicksOut2[ArrayResize(TicksOut2, ArraySize(TicksOut2) + 1, 1e6) - 1]);
}

void OnDeinit( const int ) { FileSave(__FILE__, TicksOut2, FILE_COMMON); }
 
fxsaber #:
You can't help but know that the original library has been using delta encoding from the very beginning. And that doesn't stop it from working.

The code is a demonstration of the high-compression capabilities of CBitBuffer. It is not intended to be a replica of the original library.

Sure, there're differences here and there. 

 
Excellent code!
 
fxsaber #:
For practical use, the ability to perform sequential decompression and compression is lacking.

Please consider adding this functionality.

TICKS_STREAM.mq5 (in attachments)

Variable-Length Encoding (TicksShortLite.mqh) - https://www.mql5.com/en/code/61728

Pros:
  High compression ratio (VLQ)
  Allows reading ticks sequentially (tick-by-tick) from compressed memory or file without full decompression
  Supporting time range queries.
  Ideal for streaming analysis (e.g., backtesting tick-by-tick)

Cons:
  No random access or append ticks.

TICKS_STREAM (EURUSD,H1)      Compressed 705200 ticks.
TICKS_STREAM (EURUSD,H1)      Decoded 705200 ticks successfully
TICKS_STREAM (EURUSD,H1)      from_time: 2025.09.24 19:10.506
TICKS_STREAM (EURUSD,H1)      to_time  : 2025.09.29 15:36.71
TICKS_STREAM (EURUSD,H1)      Read 176301 ticks in time range.


Fixed-Length Encoding (fxsaber's TicksShort.mqh) - https://www.mql5.com/ru/code/61126

Pros:
  Jump directly to any tick index (fast seek).
  Append easily.
  Decompress only one record.

Cons:
  No VLQ compression efficiency.
  Slightly larger compressed files.

Files:
TICKS_STREAM.mq5  6 kb
 

To implement faster range queries (from time, to_time) in TIICKS_STREAM, the encoding scheme has to be changed so that instead of encoding into a single block:

1. we encode ticks into chunks of certain size (eg 5000-10000 ticks/chunk) or time range (one week or month) and write a small header containing: [numer of ticks in chunk, start time, end time, chunk size in bytes].
2. Build an index of those headers, where each entry in the index has the same header information + byte offset into the compressed buffer.
3. Write the index as a single stream of bytes at the end of the compressed data buffer or into a separate index file.
For time range queries (decoding): 
1. we use linear or better binary search on the headers' index to get the start and end block indices for the from_time and to_time.
2. Decompress only ticks spanning these blocks with some filtering of the excess ticks from start and end blocks.
3. Additionally: an LRU (least recently used) cache can be built in memory to cache for example the last 10 used blocks in decompressed form.
So at the end we can build our version of CopyTicksRange() on the compressed buffer.
 
amrali #:

  Allows reading ticks sequentially (tick-by-tick) from compressed memory or file without full decompression

You can do the same thing by recording tick by tick.


Is a zigzag necessary for a time delta?

//         VLQWrite(buf, pos, ZigZagEncode(t - prev_time)); // ZigZag + VLQ encode each delta
         VLQWrite(buf, pos, t - prev_time);
 
fxsaber #:

You can do the same thing by recording tick by tick.

I showed you how it can be done with VLQ. Feel free to modify the code if you're planning to use it. I know how to do it myself. I don't have plans to extend the demo example on the codebase. 
Thanks 😊. 

 
fxsaber #:
Is a zigzag necessary for a time delta?
For time differences that are guaranteed to be non-negative, using VLQ encoding directly is the optimal choice, as it is both simple and efficient. ZigZag encoding is only an necessary extra step when you need to handle integers that can be both positive and negative. I suppose the author used ZigZag to ensure the data could still be used correctly in the occasional exceptional case where a time difference is negative?
 
fxsaber #:

Is a zigzag necessary for a time delta?

Not strictly necessary, but acts as a guard against bad programming:

   MqlTick ticks[];
   ArraySetAsSeries(ticks, true);
   CopyTicks(Symbol(), ticks);

Zigzag can be removed for time delta to gain extra performance. However, the gain is negligible.

12
New comment