Zero-Copy Tick Streaming (Part 2) : Reading MetaTrader 5 Ticks into Python and Benchmarking the Result

Introduction

Moving live tick data from MetaTrader 5 into Python without unnecessary copying or per-tick object creation requires more than shared memory. The Python reader must interpret the existing memory layout correctly, access it safely while MetaTrader 5 continues writing, and expose the data in a form that analytical libraries can use directly.

Part 1 of this series built the MetaTrader 5 half of that bridge: an Expert Advisor that writes live ticks directly into shared memory using Apache Arrow's columnar physical layout. A lock-free seqlock protects the data so that a fast writer and a slower reader can operate without exposing inconsistent snapshots. This implementation was verified independently using a diagnostic script that wrote real tick history through the bridge and printed the resulting values. The Python reader, however, had not yet been implemented.

This article completes the Python side. We reconstruct the actual ArrowArray and ArrowSchema C structures from the Apache Arrow specification using ctypes, point their buffer fields directly to offsets within a memory-mapped file, and pass them to pyarrow. The result is a RecordBatch created without copying, parsing, or boxing individual tick values into Python objects. We then verify the performance advantage with a self-contained, runnable benchmark that compares zero-copy import against a conventional struct.unpack loop across increasing batch sizes.

Readers should complete Part 1 before proceeding. This article assumes familiarity with the shared memory layout, the three-way schema contract, and the seqlock generation counter. The next section briefly revisits these concepts, but their full explanation, the MetaTrader 5-side implementation, and the diagnostic self-test script are provided in Part 1. Readers unfamiliar with terms such as "slot," "generation," or "column block" should consult that article first.

Two Python files implement and test this half of the bridge. arrow_tick_reader.py contains the reader that attaches to shared memory, implements the reader side of the seqlock, and produces a zero-copy RecordBatch. benchmark_compare.py provides an independent benchmarking harness that does not require MetaTrader 5 to be running. It reproduces the exact memory layout locally and compares zero-copy import with a legacy field-by-field unpacking loop at progressively larger batch sizes.

Contents

1. Quick recap: the layout and seqlock contract from Part 1
2. Reading zero-copy in Python
3. Benchmarking against the legacy struct bridge
4. Edge cases and pitfalls
5. Validating the full pipeline
6. Conclusion

Quick recap: the layout and seqlock contract from Part 1

Everything in this article leans on three decisions made in Part 1. It's worth having them fresh before diving into ctypes code, so here's the condensed version rather than a re-derivation:

The columnar layout. Shared memory holds two double-buffered "slots." Each slot is a fixed 64-byte header (generation, batch sequence, tick count, column count, timestamp, capacity) followed by six column blocks - one each for time_msc, bid, ask, last, volume_real, and flags. Every column block is a validity bitmap plus a flat 8-byte-per-element data buffer, both padded to 64 bytes. No pointer is ever written into shared memory - every offset is a pure function of a single capacity value, which is why this article's Python code can reconstruct the identical layout independently.

The three-way schema contract. An ARROW_TICK_COLUMNS = 6 constant on the MetaTrader 5 side, the actual number of columns WriteBatch() writes, and the column_count field decoded from the header must all agree. Part 1's EA refuses to start on a mismatch; this article's reader checks the same constant before importing anything. On the Python side, this contract also includes COLUMN_FORMATS, COLUMN_NAMES, and COLUMN_IS_INT64 at the top of arrow_tick_reader.py - they encode the column order, Arrow type codes, and int64-vs-float64 split that the MetaTrader 5 side assumes but read_batch() does not independently verify; changing one without the other is the same contract violation as a column-count mismatch, just not one this code currently catches.

The seqlock generation counter. Each slot header carries a generation field. Odd means a writer is mid-publish and the slot is unsafe to read; even means it's stable. The write sequence is odd-flag, write columns, even-flag - so a reader only ever has to retry, never block. This only works if the payload writes genuinely land in memory before the even flag does, and the reader's read of that flag genuinely happens before it reads the payload - on x86 this falls out of the platform's own strong memory ordering, but it's worth naming explicitly: a seqlock is a memory-ordering trick, not just a counter convention, and porting it to a weaker-ordering platform would need explicit acquire/release barriers around generation.

End-to-End Architecture: MetaTrader 5 to Python Zero-Copy Import

Fig. 1. The full pipeline this two-part series builds. Part 1 covered everything up to and including the shared memory segment; this article covers everything from there to the right.

With that in hand, the rest of this article is about turning those raw bytes into something pyarrow, pandas, and polars can use - without ever materializing an intermediate copy. One more thing worth carrying over from Part 1: because every offset in the layout above is a pure function of capacity and nothing else, this article's Python code never needs to ask MetaTrader 5 anything at import time beyond "what capacity was this mapping created with." That single number, plus the mapping name and the three-way schema contract from the previous section, is what this bridge runs on - capacity drives every byte offset, while the schema contract keeps both sides agreeing on what those bytes mean.

Reading zero-copy in Python

What it implements. The ArrowTickReader class - the mirror image of Part 1's CArrowTickBridge. It never allocates a buffer for tick data; it only ever computes offsets into an mmap it already has open and builds small ctypes structs pointing at them.

How it operates. __init__ attaches to the named mapping and computes the same layout constants Part 1's MetaTrader 5 side used; latest_slot() and _consistent_header() implement the seqlock reader half; read_batch() ties it together into one pyarrow.RecordBatch.

Why this approach. Every alternative - copying bytes into a Python bytes object first, or using struct.unpack - reintroduces exactly the per-tick allocation cost this bridge exists to avoid. Building the C structs directly is more code than a naive read, but it's the only way to get pyarrow to treat the mmap's memory as its own.

This is the section that actually earns the word "zero-copy," so it's worth walking through carefully. The Arrow C Data Interface expects two real C structs - not JSON, not a custom protocol, the literal structs from the Apache Arrow specification - with fields that are raw pointers. We define their layout with ctypes:

#+-------------------------------------------------------------------+
#| ArrowSchema / ArrowArray                                          |
#| C Data Interface structs, per the Apache Arrow spec.              |
#| Field order and types are fixed by the spec, not by us.           |
#+-------------------------------------------------------------------+
class ArrowArray(ctypes.Structure):
    pass

ArrowArray._fields_ = [
    ("length", ctypes.c_int64),
    ("null_count", ctypes.c_int64),
    ("offset", ctypes.c_int64),
    ("n_buffers", ctypes.c_int64),
    ("n_children", ctypes.c_int64),
    ("buffers", ctypes.POINTER(ctypes.c_void_p)),
    ("children", ctypes.POINTER(ctypes.POINTER(ArrowArray))),
    ("dictionary", ctypes.POINTER(ArrowArray)),
    ("release", ctypes.c_void_p),
    ("private_data", ctypes.c_void_p),
]

Before any of that gets used, __init__ has to attach to the same named segment Part 1's EA created and compute this process's own base address for it - the one piece of state every other method in the class depends on:

#+-------------------------------------------------------------------+
#| __init__                                                          |
#| Attaches to the named mapping and computes this process's         |
#| own base address for it.                                          |
#+-------------------------------------------------------------------+
def __init__(self, mapping_name: str, capacity: int):
    self.mapping_name = mapping_name
    self.capacity = capacity
    self.layout = layout_for_capacity(capacity)
    # On Windows, tagname with fileno=-1 attaches to an existing named
    # mapping if one already exists under that name, joining the segment
    # Part 1's EA created - but if no mapping exists yet under that name,
    # this call silently creates a new, empty one instead of failing; see
    # Edge Cases for why read order matters here.
    self.mm = mmap.mmap(-1, self.layout["total_size"], tagname=mapping_name)
    self.base_addr = ctypes.addressof(ctypes.c_char.from_buffer(self.mm))
    self._pins = []  # keeps imported batches' backing structs alive

Reading a slot's header safely means implementing the reader half of the seqlock recapped in “Quick recap: the layout and seqlock contract from Part 1” above - retry while the generation is odd, and re-check it hasn't changed after a clean read:

#+-------------------------------------------------------------------+
#| _consistent_header                                                |
#| Seqlock read: retry until generation is even and unchanged        |
#| across the read. Mirrors the write-side odd/even flip in          |
#| CArrowTickBridge::WriteBatch().                                   |
#+-------------------------------------------------------------------+
def _consistent_header(self, slot_index, retries=50):
    for _ in range(retries):
        h1 = self._read_header(slot_index)
        if h1.generation % 2 != 0:
            time.sleep(0)          # writer mid-publish, yield and retry
            continue
        h2 = self._read_header(slot_index)
        if h1.generation == h2.generation:
            return h1
    raise TimeoutError(f"Slot {slot_index} did not stabilise after {retries} retries")

Because the writer always targets whichever slot it didn't just publish to, the reader has to pick the correct one before it can even ask "is this stable" - latest_slot() reads both headers and keeps the one with the higher batch_seq:

#+-------------------------------------------------------------------+
#| latest_slot                                                       |
#| Picks whichever of the two slots has the higher stable            |
#| batch_seq.                                                        |
#+-------------------------------------------------------------------+
def latest_slot(self) -> int:
    h0 = self._consistent_header(0)
    h1 = self._consistent_header(1)
    return 0 if h0.batch_seq >= h1.batch_seq else 1

One edge case this doesn't handle: batch_seq wrap-around. The comparison above assumes the numerically larger value is always the newer one, which holds for the lifetime of any realistic MetaTrader 5 session, but a process that ran long enough to overflow a 64-bit counter - or a writer that resets it - could make an older batch look numerically larger. Not a practical concern at today's tick rates, but worth a comment if this pattern is ever ported to a counter narrower than 64 bits.

The offsets used throughout all come from a small pure function that mirrors Part 1's Init() field for field - this is the one function in the whole project that most needs to stay byte-identical between the two languages, since every downstream pointer depends on it:

#+-------------------------------------------------------------------+
#| layout_for_capacity                                               |
#| Reproduces the exact byte layout ArrowBufferWriter.mqh            |
#| computed on the MT5 side, from capacity alone.                    |
#+-------------------------------------------------------------------+
def layout_for_capacity(capacity: int):
    validity_bytes = _align64((capacity + 7) // 8)
    data_bytes = _align64(capacity * ELEM_SIZE)
    column_block_bytes = validity_bytes + data_bytes
    slot_size = HEADER_SIZE + ARROW_TICK_COLUMNS * column_block_bytes
    total_size = slot_size * 2
    return {"validity_bytes": validity_bytes, "data_bytes": data_bytes,
            "column_block_bytes": column_block_bytes,
            "slot_size": slot_size, "total_size": total_size}

A schema mismatch here would be a silent data-corruption bug rather than a crash, so read_batch() checks the decoded header's column_count against the same ARROW_TICK_COLUMNS constant from Part 1's contract before importing anything, then walks the six columns building one ArrowArray per column at a time:

#+-------------------------------------------------------------------+
#| read_batch                                                        |
#| Assembles a full six-column zero-copy RecordBatch from            |
#| whichever slot is currently stable and newest.                    |
#+-------------------------------------------------------------------+
def read_batch(self) -> pa.RecordBatch:
    slot = self.latest_slot()
    hdr = self._consistent_header(slot)
    if hdr.column_count != ARROW_TICK_COLUMNS:
        raise RuntimeError(
            f"Schema contract violated: header says column_count="
            f"{hdr.column_count}, reader expects {ARROW_TICK_COLUMNS}")
    if hdr.tick_count < 0 or hdr.tick_count > self.capacity:
        raise RuntimeError(
            f"tick_count {hdr.tick_count} out of range for capacity {self.capacity}")

    slot_base = self.base_addr + self._slot_offset(slot)
    col_base = slot_base + HEADER_SIZE
    arrays = []
    for c in range(ARROW_TICK_COLUMNS):
        data_ptr = col_base + self.layout["validity_bytes"]
        arrays.append(self._import_column(data_ptr, hdr.tick_count,
                                            COLUMN_FORMATS[c], COLUMN_NAMES[c]))
        col_base += self.layout["column_block_bytes"]
    return pa.RecordBatch.from_arrays(arrays, names=COLUMN_NAMES)

It's worth being explicit about what the seqlock in _consistent_header() actually guarantees here: it protects the header read at the moment of import, nothing more. It says nothing about the payload's lifetime afterward. The RecordBatch this method returns holds pointers into memory the writer still owns, and once the writer republishes to this same slot, that memory can change under you - the imported batch does not own an immutable snapshot, no matter how cleanly its header was read. This is a central lifetime property of a zero-copy design, not an edge case; see The writer reusing a slot out from under you in Edge cases and pitfalls for what it means in practice.

Note what read_batch() now checks and what it still doesn't: column_count and tick_count are both validated against the header before anything is imported, which closes the risk of ArrowArray reporting a length longer than the buffer actually backing it. What's still unchecked is column order, types, and Arrow format codes - a column reordered or retyped on one side without the other would misinterpret bytes silently rather than raise. That gap is worth closing before trusting this code with anything other than a schema you control on both ends.

To import one column, _import_column() takes the mmap's base address computed in __init__ and adds the offset computed above: that sum is a pointer into this process's view of the shared segment, valid regardless of where the EA's view happened to land. It points buffers[1] at it directly - no allocation, no copy - and wires up a real release callback rather than a bare None:

#+-------------------------------------------------------------------+
#| _import_column                                                    |
#| Builds real ArrowSchema/ArrowArray C structs pointing at a        |
#| single column's data inside the mmap, and imports them into       |
#| pyarrow with zero copies.                                         |
#+-------------------------------------------------------------------+
def _import_column(self, data_ptr, n, fmt, name):
    # null_count=0 lets us hand Arrow a NULL validity buffer even
    # though the writer still physically wrote one - see the recap above.
    buffers = (ctypes.c_void_p * 2)(None, data_ptr)
    format_buf = ctypes.create_string_buffer(fmt.encode("ascii"))
    schema_release_cb = SCHEMA_RELEASE_FN(_release_schema)
    array_release_cb = ARRAY_RELEASE_FN(_release_array)

    schema = ArrowSchema()
    schema.format = ctypes.cast(format_buf, ctypes.c_char_p)
    schema.n_children = 0
    schema.release = ctypes.cast(schema_release_cb, ctypes.c_void_p)

    array = ArrowArray()
    array.length = n
    array.null_count = 0
    array.n_buffers = 2
    array.buffers = ctypes.cast(buffers, ctypes.POINTER(ctypes.c_void_p))
    array.release = ctypes.cast(array_release_cb, ctypes.c_void_p)

    # every object referenced above must outlive the returned Array,
    # or pyarrow later dereferences freed memory - see the gotcha below.
    self._pins.append(_Pinned(schema, array, buffers, format_buf,
                               schema_release_cb, array_release_cb))
    return pa.Array._import_from_c(ctypes.addressof(array), ctypes.addressof(schema))

The part that's easy to lose track of in the code above is which pieces get copied and which don't. ArrowArray, ArrowSchema, buffers, and format_buf are all small, ordinary heap allocations - maybe 200 bytes total per column. None of that is tick data. buffers[1] is simply set to data_ptr, a raw address that lands inside the mmap - pyarrow follows it directly into shared memory rather than reading through it into a new allocation.

This design leans on an invariant that's never actually checked in code: every tick value is valid, always. null_count=0 and a discarded validity buffer are only safe as long as that holds; if a future version of the writer ever needs to represent a missing value, this reader would need to start reading buffers[0] for real instead of silently dropping it.

ArrowArray/ArrowSchema Pointer Layout: Heap Structs vs. mmap'd Data

Fig. 2. The only heap allocations are the small structs on the left. buffers[1] and schema.format are raw addresses - one points into the mmap, the other into a tiny format string - so the multi-kilobyte tick data itself is never duplicated.

One cost of pinning everything this way: self._pins only ever grows, never shrinks. Each read_batch() call appends six more pinned structs that are designed to outlive their RecordBatch (their release callbacks are no-ops), so a reader polling in a long-running loop accumulates pinned objects for as long as it stays open. This is a deliberate trade-off of this article's demo implementation, not a production pattern: it trades unbounded growth for the simplicity of never worrying about a pin being freed too early. For production use, tie pinned objects to the imported batch's own lifetime rather than the reader's - for example by returning a small wrapper object whose __del__ releases its own pins - instead of retaining every column's pins for as long as the reader itself stays open. Periodically creating a fresh ArrowTickReader is the simplest way to bound that growth in the meantime.

The release callbacks referenced in the gotcha below aren't optional plumbing - they're what makes _import_from_c accept the struct at all. Each column gets its own no-op callback that marks itself released, wrapped in a real ctypes.CFUNCTYPE so it has an address pyarrow can call:

#+-------------------------------------------------------------------+
#| SCHEMA_RELEASE_FN / ARRAY_RELEASE_FN                              |
#| ctypes function-pointer types for the release callbacks below.    |
#+-------------------------------------------------------------------+
SCHEMA_RELEASE_FN = ctypes.CFUNCTYPE(None, ctypes.POINTER(ArrowSchema))

#+-------------------------------------------------------------------+
#| _release_schema                                                   |
#| Marks an ArrowSchema as released. No heap memory to free -        |
#| buffers live in the mmap, not on this struct - but a real         |
#| callback must exist or pyarrow refuses the import.                |
#+-------------------------------------------------------------------+
def _release_schema(schema_ptr):
    # nothing to free - buffers live in the mmap, not on this struct
    schema_ptr.contents.release = ctypes.cast(None, ctypes.c_void_p)

The full RecordBatch is assembled with pa.RecordBatch.from_arrays() across all six columns. From here, .to_pandas() or polars.from_arrow() both work, though they differ in cost: .to_pandas() materializes new NumPy-backed columns and copies the values out, while polars.from_arrow() can often keep referencing the same Arrow buffers without copying. Either way, nothing gets individually boxed into Python objects one tick at a time - the cost, if any, is a single bulk copy, not a per-tick loop. close() is the reader's only cleanup method, releasing the mmap once the caller is done:

#+-------------------------------------------------------------------+
#| close                                                             |
#| Releases the mmap. Any RecordBatch derived from this reader       |
#| must not be touched after this call - see Edge Cases.             |
#+-------------------------------------------------------------------+
def close(self):
    self.mm.close()

The one gotcha that will bite you first. Setting release = None on the schema or array struct does not mean "no cleanup needed." Per the Arrow spec, a NULL release pointer specifically signals "this structure has already been released, do not use it" - and pyarrow enforces this: importing a struct with release = None raises ArrowInvalid: Cannot import released ArrowSchema immediately.

You need a real callback, even if it does nothing but mark itself as released, and you must keep the ctypes.CFUNCTYPE object referencing it alive for as long as the imported array exists - if Python garbage-collects the callback closure, calling it later segfaults the interpreter. This is exactly the kind of bug that surfaces the moment release callbacks aren't wired up correctly, which is why arrow_tick_reader.py pins every schema, array, buffer array, format string, and release callback together in a single _Pinned dataclass instance per column, held for the reader's lifetime.

Benchmarking against the legacy struct bridge

What it implements. A standalone timing harness that never touches MetaTrader 5 - it builds synthetic tick columns in the exact physical layout Part 1's ArrowBufferWriter.mqh produces, then reads them back two different ways.

How it operates. build_arrow_column_buffer() writes synthetic values into a local bytearray using the same validity-bitmap-plus-data-buffer shape as the real bridge; zero_copy_read() and legacy_struct_read() then race against each other over 200 repeats per batch size, with time.perf_counter() around each.

Why this approach. A benchmark that depends on a live terminal is one more thing that can be unavailable or non-reproducible when someone tries to verify the numbers. Reproducing the byte layout locally keeps the comparison honest - both paths start from the identical underlying buffers, even though zero-copy only ever points at them while the legacy path actually reads every value out of them - while making the script runnable by anyone with just pip install pyarrow pandas.

Claims of "zero-copy" are cheap to make and easy to get subtly wrong, so this section backs the claim with a runnable, self-contained benchmark. It times two read paths at increasing batch sizes: the zero-copy Arrow import from “Reading zero-copy in Python” above, and a struct.unpack-per-field loop matching the approach used by the earlier flat shared-memory bridge article.

The synthetic data generator writes plausible XAUUSD values directly into the same validity-bitmap-plus-data-buffer shape Part 1's writer produces, so the benchmark never touches a live terminal but still exercises the real byte layout:

#+-------------------------------------------------------------------+
#| build_arrow_column_buffer                                         |
#| Writes one column's data buffer exactly as                        |
#| ArrowBufferWriter.mqh would (validity bitmap all-1s,              |
#| then packed 8-byte values).                                       |
#+-------------------------------------------------------------------+
def build_arrow_column_buffer(values, is_int64, layout):
    buf = bytearray(layout["column_block_bytes"])
    n = len(values)
    for i in range(n):
        byte_idx, bit_idx = divmod(i, 8)
        buf[byte_idx] |= (1 << bit_idx)
    fmt = "<q" if is_int64 else "<d"
    off = layout["validity_bytes"]
    for i, v in enumerate(values):
        struct.pack_into(fmt, buf, off + i * 8, int(v) if is_int64 else float(v))
    return buf

The zero-copy side of the comparison reuses the exact ctypes machinery from “Reading zero-copy in Python” above - ArrowSchema, ArrowArray, and the pinned release callbacks - but points its buffers at a plain Python bytearray instead of a shared memory mapping, since the benchmark's point is to isolate import cost, not IPC cost:

addr = ctypes.addressof((ctypes.c_char * len(buf)).from_buffer(buf))
...
arrays.append(pa.Array._import_from_c(ctypes.addressof(array), ctypes.addressof(schema)))
batch = pa.RecordBatch.from_arrays(arrays, names=COLUMN_NAMES)

The legacy comparison path is a field-by-field unpack loop matching the approach the earlier flat shared-memory bridge article used, now actually run and timed rather than just described:

#+-------------------------------------------------------------------+
#| legacy_struct_read                                                |
#| Mirrors the earlier CreateFileMapping bridge article: unpack      |
#| every field of every tick by hand into Python tuples, then        |
#| build a DataFrame. The O(n) path zero-copy import avoids.         |
#+-------------------------------------------------------------------+
def legacy_struct_read(column_buffers, n, layout):
    rows = []
    for i in range(n):
        row = []
        for c in range(ARROW_TICK_COLUMNS):
            off = layout["validity_bytes"] + i * 8
            if COLUMN_IS_INT64[c]:
                (v,) = struct.unpack_from("<q", column_buffers[c], off)
            else:
                (v,) = struct.unpack_from("<d", column_buffers[c], off)
            row.append(v)
        rows.append(tuple(row))
    return pd.DataFrame(rows, columns=COLUMN_NAMES)

Running it over 200 repeats per batch size produced the following medians. The loop itself warms up the import machinery once before timing starts, so the numbers reflect steady-state cost rather than one-time module setup:

for n in BATCH_SIZES:
    layout = layout_for_capacity(n)
    column_buffers = [build_arrow_column_buffer(cols[c], COLUMN_IS_INT64[c], layout)
                      for c in range(ARROW_TICK_COLUMNS)]

    _batch, _pins = zero_copy_read(column_buffers, n)   # warm-up, discarded

    t0 = time.perf_counter()
    for _ in range(REPEATS):
        batch, pins = zero_copy_read(column_buffers, n)
        _ = batch.num_rows                              # touch result, exclude to_pandas() cost
    t_arrow = (time.perf_counter() - t0) / REPEATS * 1000

    t0 = time.perf_counter()
    for _ in range(REPEATS):
        df = legacy_struct_read(column_buffers, n, layout)
    t_legacy = (time.perf_counter() - t0) / REPEATS * 1000

Batch size (ticks)	Zero-copy Arrow (ms)	Legacy struct.unpack (ms)	Speedup
64	0.1024	0.3857	3.8x
256	0.0992	0.6830	6.9x
1024	0.0985	1.8795	19.1x
4096	0.0979	7.0781	72.3x

One asymmetry worth being upfront about: the zero-copy path's timed result is a pa.RecordBatch, while the legacy path's timed result is a fully-built pd.DataFrame - to_pandas() conversion time is deliberately excluded from the zero-copy side (see the warm-up comment above), so this table measures Arrow-wrapper creation against unpack-plus-DataFrame-construction, not two paths converging on the same final object. It's a fair comparison of what each approach actually has to do to make tick data usable from Python, but it isn't a measurement of two identical pipelines - and it's a single-process, single-threaded benchmark besides: it doesn't exercise seqlock retries under contention, concurrent readers and writers, slot reuse mid-read, a capacity mismatch, or the null-bitmap path. It validates import cost, not pipeline correctness under load.

Zero-Copy Arrow Import vs. Legacy struct.unpack Benchmark

Fig. 3. Zero-copy import time is essentially flat as batch size grows, because it does the same fixed amount of work - build one struct, set some pointers - regardless of how many ticks are behind those pointers. The legacy path scales linearly with tick count because it touches every field of every tick individually.

The shape of this result is the whole point: zero-copy cost is dominated by struct setup, not data volume, so it stays essentially flat across the 64-to-4,096-tick range actually measured here. The legacy path is linear in tick count because it has to touch every field of every tick to build a Python object. The gap only widens with batch size inside that range - at 4,096 ticks zero-copy is already over 70x faster. Whether that trend keeps widening past 4,096 ticks wasn't measured here - page faults, cache and TLB effects, or GC pressure from accumulated _Pinned objects could all change the shape of the curve at larger sizes, so treat the flat line as demonstrated only up to the batch sizes actually benchmarked.

Put in throughput terms: at the 4,096-tick batch size, the zero-copy path creates Arrow wrappers at roughly 41.8 million ticks/sec, versus roughly 579,000 ticks/sec for the legacy path's unpack-and-build-DataFrame loop. That number measures how fast this process can wrap already-resident bytes in Arrow structs, not how fast a live MetaTrader 5 feed can be ingested or processed end-to-end - no real feed approaches either figure. What it does show is where the bottleneck moves: once feature computation sits on top of either path, the legacy side's per-tick object creation keeps eating into that budget, while the zero-copy side leaves nearly all of it free.

Edge cases and pitfalls

Release callback garbage collection. Covered in “Reading zero-copy in Python” above, but worth repeating - it's the failure mode you'll hit first: a ctypes.CFUNCTYPE callback that goes out of scope while the imported array is still alive leaves a dangling function pointer. The crash often doesn't happen immediately; it happens whenever pyarrow's internals later decide to call release, which is much harder to trace back. Pin everything.

Capacity mismatch between writer and reader. layout_for_capacity() must be called with the exact same capacity Part 1's EA was initialized with, or every offset downstream is wrong and you'll read garbage with no obvious error - the bytes are still there, just misinterpreted. This is the reader-side consequence of the capacity contract flagged in Part 1: treat capacity as fixed per mapping name, never as a free parameter the reader can guess.

Reader polling faster than the writer publishes. On a quiet symbol, a reader hammering read_batch() in a tight loop mostly re-reads the same stable slot - harmless, but wasteful. A light time.sleep() between polls avoids burning a CPU core for no new data.

Attaching before the mapping exists. mmap.mmap(-1, size, tagname=name) on Windows creates a new mapping if none exists under that name, rather than failing - a reader started before Part 1's EA can silently attach to an empty, zeroed segment. Always confirm tick_count > 0 and a sane batch_seq before trusting a first read.

Closing the mmap while a RecordBatch is still alive. close() unmaps the view, but every pa.Array imported from it still holds raw pointers into that now-invalid memory. If any DataFrame or RecordBatch derived from a reader is still in scope, calling close() and then touching that data is a use-after-free, not a clean Python exception. Convert to .to_pandas() - which copies - or a Polars DataFrame before closing the reader if the data needs to outlive it; either way, don't rely on the original RecordBatch remaining valid past close().

The writer reusing a slot out from under you. The seqlock only protects reading a header consistently - it says nothing about how long the data behind a slot stays valid. Once the writer republishes, it rotates back through the same two slots; a RecordBatch built from a slot earlier can have its backing memory overwritten by a new batch while a reference to it is still held. This is a more common real-world hazard than closing too early - if a batch needs to outlive the next publish, copy it (.to_pandas(), Polars, or a plain buffer copy) before polling again.

Validating the full pipeline

Part 1 verified the MetaTrader 5 side in isolation. This section closes the loop by running both halves together and confirming they agree, plus the throughput numbers that quantify the whole point of building this bridge in the first place.

Check	Method
End-to-end correctness	Start Part 1's ArrowTickStreamer EA first to keep the mapping open, run ArrowBridgeSelfTest.mq5 to print known values, then run arrow_tick_reader.py against the same mapping name - its output must match the logged values
Throughput	benchmark_compare.py, from “Benchmarking against the legacy struct bridge” above - median read latency across batch sizes, zero-copy vs legacy struct path
Symbol / Timeframe	XAUUSD/M5 (default; the bridge itself is symbol-agnostic)

To run the correctness check: open an XAUUSD chart and start Part 1's ArrowTickStreamer EA on it first - its mapping stays open for as long as the EA runs, which is what actually keeps the segment readable from Python. With the EA running, run ArrowBridgeSelfTest.mq5 from Part 1 to print a batch of known bid/ask/time_msc values to the Experts log, then run arrow_tick_reader.py against the same InpMappingName and compare its output against that log - the self-test's own handle closes the moment OnStart() returns, so it is only useful here as a source of known values, not as what keeps the mapping alive. The benchmark, by contrast, needs nothing from MetaTrader 5 at all - python benchmark_compare.py runs standalone and prints the table above directly.

If the two sides don't match, the mismatch itself usually points at the cause: a RuntimeError from the column_count check means the schema contract from “Quick recap: the layout and seqlock contract from Part 1” has drifted between the two languages; plausible-looking but wrong bid/ask values almost always trace back to a capacity mismatch (see “Edge cases and pitfalls” above) rather than a layout bug, since a wrong capacity still produces valid-looking floats, just read from the wrong byte offset; and a TimeoutError from _consistent_header() more often means a slot got stuck mid-publish or is being written faster than this reader can poll than it means "no writer" - a genuinely absent writer usually leaves generation sitting at an even value and resolves immediately rather than timing out, so a persistent timeout is worth treating as a write-side stall or a race, not purely a connectivity check.

Conclusion

Across both parts, the pattern generalizes past tick data. Anything MetaTrader 5 produces in a regular, fixed-width shape - order book snapshots, indicator buffer dumps, custom feature vectors - can ride the same columnar layout and seqlock discipline. Worth keeping if you adapt this: the three-way schema contract (never let writer and reader silently disagree about shape), the seqlock instead of a mutex (a torn read is caught and retried, never blocked), and never putting a pointer value inside shared memory itself.

What you get in exchange for getting the C struct layout exactly right is a read path whose cost barely moves as batch size grows - exactly what you want when a research pipeline is pulling thousands of ticks a second out of a live terminal and handing them to a feature calculation without ever wanting to see a Python for loop in between.

File	Type	Description
arrow_tick_reader.py	Python	Zero-copy reader - reconstructs ArrowSchema/ArrowArray via ctypes, imports into pyarrow
benchmark_compare.py	Python	Self-contained benchmark: zero-copy import vs legacy struct.unpack read, across batch sizes