Discussing the article: "Zero-Copy Tick Streaming (Part 2) : Reading MetaTrader 5 Ticks into Python and Benchmarking the Result"

[MetaQuotes]

Check out the new article: Zero-Copy Tick Streaming (Part 2) : Reading MetaTrader 5 Ticks into Python and Benchmarking the Result.

We build a Python reader that attaches to the MetaTrader 5 shared memory segment, reproduces the Arrow columnar layout, and constructs ArrowArray/ArrowSchema via ctypes so pyarrow imports ticks without copying. The code honors the three-way schema contract and the seqlock generation counter to read consistent batches. A standalone benchmark verifies performance against a legacy per-field struct.unpack path with real numbers.

Moving live tick data from MetaTrader 5 into Python without unnecessary copying or per-tick object creation requires more than shared memory. The Python reader must interpret the existing memory layout correctly, access it safely while MetaTrader 5 continues writing, and expose the data in a form that analytical libraries can use directly.

Part 1 of this series built the MetaTrader 5 half of that bridge: an Expert Advisor that writes live ticks directly into shared memory using Apache Arrow's columnar physical layout. A lock-free seqlock protects the data so that a fast writer and a slower reader can operate without exposing inconsistent snapshots. This implementation was verified independently using a diagnostic script that wrote real tick history through the bridge and printed the resulting values. The Python reader, however, had not yet been implemented.

This article completes the Python side. We reconstruct the actual ArrowArray and ArrowSchema C structures from the Apache Arrow specification using ctypes, point their buffer fields directly to offsets within a memory-mapped file, and pass them to pyarrow. The result is a RecordBatch created without copying, parsing, or boxing individual tick values into Python objects. We then verify the performance advantage with a self-contained, runnable benchmark that compares zero-copy import against a conventional struct.unpack loop across increasing batch sizes.

Author: Adedayo David Gbadebo