Zero-Copy Tick Streaming (Part 1): Bridging MetaTrader 5 to Shared Memory with the Arrow C Data Interface

MetaTrader 5 — Integration | 8 September 2026, 07:46

265

Adedayo David Gbadebo

Introduction

If you have ever tried to get live tick data out of MetaTrader 5 and into a proper Python analytics stack, you already know the annoying part isn't the trading logic - it's the plumbing. You write a CSV exporter, or you open a socket, or you reach for a shared-memory hack, and then you spend the rest of your afternoon writing a parser on the Python side that turns raw bytes back into something pandas can use. Every one of those approaches pays a "translation tax" on every single tick: serialize on the way out, deserialize on the way in, and somewhere in the middle you're allocating a fresh Python object per field per tick.

This two-part series removes that tax entirely, using the Arrow C Data Interface - the same zero-copy mechanism that lets DuckDB, Polars, and pandas hand data to each other without a single byte being copied. MetaTrader 5 writes ticks natively in Arrow's columnar memory layout into a block of shared memory; Python reconstructs the real ArrowArray/ArrowSchema C structs pointing straight at that memory and hands them to pyarrow, which materializes a RecordBatch with no parsing loop at all. Part 1 - this article - builds the MetaTrader 5 side: the columnar memory layout, the lock-free publish mechanism, the Expert Advisor that streams live ticks, and a diagnostic script that proves the MetaTrader 5 half is correct on its own before Python ever gets involved. Part 2 covers the Python reader and a benchmark that quantifies the zero-copy claim with real numbers.

Where this sits relative to the series' other bridge articles. This is the fourth transport/IPC article in the series, and it is not a re-skin of the earlier ones. The Rust cdylib bridge crosses a function-call boundary. The Windows shared-memory bridge moves flat, row-based structs that Python has to unpack field-by-field with struct.unpack. The gRPC/Protobuf bridge serializes messages over a network socket. This article is the only one that lays data out in a columnar, schema-described format that a downstream library can import directly, with zero deserialization and zero per-tick Python object creation. If your goal is single-tick low-latency signaling, the shared-memory bridge is still the right tool. If your goal is handing a batch of ticks to pandas, Polars, or a feature pipeline as fast as physically possible, this is the one you want - and Part 2's benchmark quantifies exactly how much faster.

We'll build this in three files. ArrowBufferWriter.mqh builds Arrow's physical layout by hand (MetaTrader 5 has no native Arrow library, and per the series' rule against external dependencies, we're not reaching for one) and publishes batches into a double-buffered region using a lock-free seqlock. ArrowTickStreamer.mq5 collects live ticks on top of it and flushes on a batch-size or timeout trigger. ArrowBridgeSelfTest.mq5 sits alongside both, purpose-built for isolating MetaTrader 5-side bugs from Python-side ones before the two languages are ever tested together.

Why not just serialize?

Every IPC bridge in this series so far has moved data across a process boundary by agreeing on a wire format and then converting to and from it. That conversion step is where the cost lives. Take the earlier shared-memory bridge: MetaTrader 5 packs a tick into a fixed-size C struct, writes the raw bytes into a memory-mapped file, and Python reads those bytes back with struct.unpack - one function call per field, per tick. For a single tick that's invisible. For a batch of 4,096 ticks with six fields each, that's over 24,000 individual unpack calls, each one creating a Python int or float object and appending it to a growing list, before pandas even gets involved.

Arrow sidesteps this by describing memory instead of describing bytes-on-the-wire. An Arrow array is just: a validity bitmap, a contiguous buffer of values, and a tiny schema struct saying "this buffer holds N int64s" or "N float64s." If the consumer already knows how to read that physical layout - and pyarrow does, natively - there's nothing left to parse. You're not converting a wire format into an in-memory format; the wire format is the in-memory format. The Arrow C Data Interface formalizes this as a tiny, stable ABI: two C structs, ArrowSchema and ArrowArray, whose fields are just pointers into whatever memory the data actually lives in.

End-to-End Architecture: MetaTrader 5 to Python Zero-Copy Import

Fig. 1. End-to-end path across both parts of this series: the EA batches ticks and publishes them once via RtlMoveMemory; the Python side (Part 2) only ever moves a pointer, never the tick data itself.

The practical upshot is that "zero-copy" here is not a marketing phrase - it's a literal description of what happens on the read side. Part 2 shows the benchmark that backs this up with real numbers, not just the claim. This article builds everything up to and including the point where a correct, verifiable batch of ticks sits in shared memory, ready to be read.

Arrow's physical memory layout

Arrow's spec for a primitive array (the kind we need for int64 and float64 columns - no strings, no nested types) is intentionally simple. Every array is described by exactly two buffers:

Buffer 0 - validity bitmap. One bit per element, packed 8-to-a-byte, bit i set means element i is non-null. If every element is valid (our case - a tick either exists or it doesn't get written), the spec allows null_count = 0 and a NULL validity pointer, letting a consumer skip the bitmap check entirely.

Buffer 1 - data buffer. A flat, contiguous run of fixed-width values - 8 bytes per element for both our int64 and float64 columns, packed with no gaps.

Both buffers are padded to a 64-byte boundary. This isn't cosmetic - it's what lets SIMD-friendly consumers (and the CPU's own cache line fetches) operate on the buffer without unaligned-access penalties. Our layout function reproduces this padding rule exactly:

//+------------------------------------------------------------------+
//| Align64                                                          |
//| Rounds a byte count up to the next 64-byte boundary,             |
//| per the Arrow buffer alignment spec.                             |
//+------------------------------------------------------------------+
int Align64(const int nbytes) const
  {
   return((int)(((nbytes + ARROW_ALIGN - 1) / ARROW_ALIGN) * ARROW_ALIGN));
  }

A tick record in our schema has six columns: time_msc (int64), bid, ask, last, volume_real (all float64), and flags (int64, holding MetaTrader 5's tick flag bitmask). Every column is 8 bytes wide, which keeps the layout math uniform - one formula covers int64 and float64 alike, since we're just moving 8-byte words either way. Each slot in shared memory holds a fixed 64-byte header (generation counter, batch sequence number, tick count, column count, a write timestamp, and the slot's capacity) followed by six column blocks, each one a validity buffer and a data buffer back to back.

Arrow Columnar Slot Memory Layout with Six Tick Columns

Fig. 2. One slot's physical layout: a 64-byte header followed by six 64-byte-aligned column blocks, each a validity bitmap plus a flat data buffer.

One deliberate simplification worth calling out here rather than burying in the code: since every tick we write is valid by construction, the validity bitmap is always all-1s. ArrowBufferWriter.mqh still writes it faithfully - so the wire layout never has to change if a future version needs to represent gaps - but Part 2's Python importer takes the shortcut the Arrow spec explicitly allows and passes null_count = 0 with a NULL validity pointer, skipping the bitmap on import entirely.

The three-way schema contract

Every ML-facing article in this series enforces a compile-time constant, an actual buffer count, and a manifest-declared width, and refuses to run on mismatch. This article's version of that contract is schema-shaped rather than ONNX-shaped, but the discipline is identical:

//+------------------------------------------------------------------+
//| Arrow schema contract - column count, names, and format codes    |
//+------------------------------------------------------------------+
// Three-way contract anchor: this constant, the number of columns
// actually written by CArrowTickBridge::WriteBatch(), and the
// column_count field decoded on the Python side must all agree.
// A mismatch is treated as fatal at OnInit() and the EA refuses to run.
#define ARROW_TICK_COLUMNS   6

ContractOk() checks this at construction time, before a single WinAPI call is made:

//+------------------------------------------------------------------+
//| ContractOk                                                       |
//| Validates the compile-time / runtime schema contract             |
//| before touching WinAPI at all.                                   |
//+------------------------------------------------------------------+
bool ContractOk(string &reason)
  {
   if(ArraySize(ArrowColumnNames)!=ARROW_TICK_COLUMNS ||
      ArraySize(ArrowColumnFormats)!=ARROW_TICK_COLUMNS ||
      ArraySize(ArrowColumnIsInt64)!=ARROW_TICK_COLUMNS)
     {
      reason=StringFormat("Column metadata length mismatch vs ARROW_TICK_COLUMNS=%d",ARROW_TICK_COLUMNS);
      return(false);
     }
   return(true);
  }

OnInit() in the EA calls this before touching CreateFileMappingW at all, and returns INIT_PARAMETERS_INCORRECT on failure - loudly, at startup, rather than silently misreading a column six months from now because someone added a field to ArrowColumnNames without updating the count. Part 2's Python reader checks the decoded header's column_count against this same constant before importing anything, closing the loop on both ends of the pipe.

Double buffering and the seqlock generation counter

Shared memory with two independent processes touching it needs a consistency mechanism, and a full kernel mutex is overkill for a hot path that runs on every tick. We use the same pattern the Linux kernel uses for things like the timekeeping subsystem: a seqlock. Each slot's header carries a generation counter. Odd means "a writer is actively publishing into this slot right now, the data is inconsistent." Even means "this slot is stable, safe to read."

The write sequence for one batch is three steps: bump the counter to odd and write the header first (this is the "danger" flag going up), write all six column buffers, then bump the counter to the next even value and write the header again (this is the publish point - only now is the batch visible). A reader does the mirror image: read the generation, and if it's odd, back off and retry; if it's even, read the header fields, then re-read the generation and confirm it hasn't changed underneath you. If it has, someone started writing mid-read and you retry. Part 2 implements that reader half in Python; this article implements and shows the writer half completely, since it's the side that actually determines correctness.

// 3) flip generation to even (stable) - this is the publish point.
//    Only now does the batch become visible/consistent to readers.
hdr.generation=writeGen+1;         // even -> stable
ArrayInitialize(hb,0);
PackHeader(hb,0,hdr);
RtlMoveMemory(m_pBase+offset,hb,ARROW_HEADER_SIZE);

Because we double-buffer - two slots, and the writer always targets whichever slot it didn't just publish to - a slow reader never blocks the writer, and the writer never has to wait on a reader to finish. The writer always has a free slot to write into; a reader always has a stable slot to read from once it picks the one with the higher batch_seq.

Seqlock Generation Counter Timeline for Writer and Reader

Fig. 3. The writer's odd-generation window is the only period a reader must avoid; retrying costs a few microseconds, never a lock.

On the Python side (Part 2) this is implemented as a small retry loop with a bounded number of attempts, which is generous given that a publish-and-flip cycle at typical batch sizes completes in well under a millisecond - a reader would have to be catastrophically unlucky to exhaust the retry budget under normal load.

Inside ArrowBufferWriter.mqh

What it implements. A single class, CArrowTickBridge, that owns a WinAPI shared-memory mapping and knows how to lay Arrow-format tick batches into it. It has no knowledge of MetaTrader 5's tick stream, symbols, or timers - it only knows how to turn a MqlTick[] array and a capacity into bytes at the right offsets.

How it operates. Three phases per instance: Init() computes every layout constant from a single requested capacity and opens the mapping; WriteBatch() packs one batch of ticks into whichever slot isn't currently the "published" one, using the three-step seqlock sequence; Deinit() releases the WinAPI handles cleanly on EA shutdown.

Why this approach. Everything is built from MetaTrader 5's native kernel32.dll imports and plain structs - no ALGLIB, no external DLL, per the series' rule - and every offset is a pure function of capacity so nothing needs a pointer serialized into shared memory (Section 11 covers why that specific shortcut is a trap).

The header itself is a fixed-size struct with explicit packing, so its in-memory byte layout is completely predictable on both sides of the bridge:

//+------------------------------------------------------------------+
//| Fixed 64-byte slot header. Field order and sizes are frozen -    |
//| the Python reader decodes this with a matching struct.Struct     |
//| format string, so do not reorder fields without updating both    |
//| sides.                                                           |
//+------------------------------------------------------------------+
#pragma pack(push,1)
struct ArrowSlotHeader
  {
   long           generation;            // 8  odd = write in progress, even = stable
   long           batch_seq;             // 8  monotonically increasing batch id
   int            tick_count;            // 4  ticks actually populated this batch
   int            column_count;          // 4  must equal ARROW_TICK_COLUMNS
   long           write_timestamp_msc;   // 8  EA-side wall clock at publish time
   int            capacity;              // 4  max ticks this slot can hold
   int            reserved0;             // 4
   uchar          reserved1[24];         // 24 padding out to 64 bytes total
  };
#pragma pack(pop)

Two small private helpers turn a slot index into a byte offset and a struct into raw bytes at that offset - every other method in the class routes through these two rather than computing offsets ad hoc:

//+------------------------------------------------------------------+
//| SlotOffset                                                       |
//| Converts a slot index (0 or 1) into its byte offset              |
//| from the start of the mapping.                                   |
//+------------------------------------------------------------------+
long SlotOffset(const int slotIndex) const
  {
   return((long)slotIndex * (long)m_slotSize);
  }

//+------------------------------------------------------------------+
//| PackHeader                                                       |
//| Serializes a full ArrowSlotHeader struct into a byte             |
//| buffer at a given offset.                                        |
//+------------------------------------------------------------------+
void PackHeader(uchar &buf[],const int offset,const ArrowSlotHeader &hdr)
  {
   uchar tmp[];
   StructToCharArray(hdr,tmp);
   ArrayCopy(buf,tmp,offset,0,ArraySize(tmp));
  }

Every field width and order in ArrowSlotHeader is frozen deliberately - Part 2's Python side decodes this same 64 bytes with a matching ctypes.LittleEndianStructure, and the two must never drift apart. Init() turns a single requested capacity into every other layout constant the class needs, using the same Align64() helper from Section 4:

//+------------------------------------------------------------------+
//| Init                                                             |
//| Computes every layout constant from a requested capacity         |
//| and opens the shared memory mapping.                             |
//+------------------------------------------------------------------+
bool Init(const string mappingName,const int capacityTicks)
  {
   ...
   m_capacity        = capacityTicks;
   m_validityBytes   = Align64((m_capacity+7)/8);
   m_dataBytes       = Align64(m_capacity*ARROW_ELEM_SIZE);
   m_columnBlockBytes= m_validityBytes+m_dataBytes;
   m_slotSize        = ARROW_HEADER_SIZE + ARROW_TICK_COLUMNS*m_columnBlockBytes;
   m_totalSize       = (ulong)m_slotSize*2;

   m_hMapping=CreateFileMappingW(INVALID_HANDLE_VALUE_L,0,PAGE_READWRITE,
                                  (uint)(m_totalSize>>32),(uint)(m_totalSize & 0xFFFFFFFF),
                                  mappingName);
   ...
  }

Because MetaTrader 5 has no primitive for "write this double at this byte offset in unmanaged memory," we build each column into a local uchar array first, using MQL5's built-in StructToCharArray() to serialize each scalar value, then move the whole column block across with one RtlMoveMemory call:

//+------------------------------------------------------------------+
//| PackValue                                                        |
//| Serializes one scalar (int64 or double) into a byte              |
//| buffer at a given offset, for values whose type is               |
//| known only at the call site.                                     |
//+------------------------------------------------------------------+
template<typename T>
void PackValue(uchar &buf[],const int offset,T value)
  {
   uchar tmp[];
   StructToCharArray(value,tmp);
   ArrayCopy(buf,tmp,offset,0,ArraySize(tmp));
  }

The WinAPI surface is intentionally small - just enough to create a named, page-file-backed mapping and get a raw pointer to it:

//+------------------------------------------------------------------+
//| WinAPI imports - memory-mapped file plumbing.                    |
//+------------------------------------------------------------------+
#import "kernel32.dll"
long   CreateFileMappingW(long hFile,long lpAttr,uint flProtect,uint dwMaxSizeHigh,uint dwMaxSizeLow,string lpName);
long   MapViewOfFile(long hFileMappingObject,uint dwDesiredAccess,uint dwFileOffsetHigh,uint dwFileOffsetLow,ulong dwNumberOfBytesToMap);
void   RtlMoveMemory(long Destination,uchar &Source[],ulong Length);
#import

Notice what is deliberately absent: no pointers are ever written into shared memory. A pointer captured in the writer's address space is meaningless in the reader's - MapViewOfFile gives no guarantee that both processes see the mapping at the same base address. Instead, every offset in this layout is computed from a pure function of capacity alone (Align64, header size, column count), so Part 2's Python side reconstructs the identical offsets independently, then adds its own mapping's base address. This is the same trick that makes the layout portable across processes without ever serializing a pointer - only the shape of the memory is agreed on, never a location within it.

The actual per-column write, inside WriteBatch(), starts by picking which slot to write into - the concrete line behind the "always targets whichever slot it didn't just publish to" rule from Section 6 - then sets the validity bit for every populated tick and packs each value at its 8-byte slot in the data buffer:

int n = (count>m_capacity) ? m_capacity : count;
int targetSlot = 1-m_activeSlot;   // write into the slot NOT currently being read
long offset = SlotOffset(targetSlot);

Which MqlTick field feeds which of the six columns is a plain switch keyed on the column index - this is the one place in the file where the abstract "six columns" from Section 4 becomes concrete field names:

long   lv=0;
double dv=0.0;
switch(c)
  {
   case 0: lv=ticks[i].time_msc;    break;
   case 1: dv=ticks[i].bid;          break;
   case 2: dv=ticks[i].ask;          break;
   case 3: dv=ticks[i].last;         break;
   case 4: dv=ticks[i].volume_real; break;
   case 5: lv=(long)ticks[i].flags; break;
  }

// validity bitmap: all ticks are valid, so every bit is 1
// across the first ceil(n/8) bytes.
for(int i=0; i<n; i++)
  {
   int byteIdx=i/8, bitIdx=i%8;
   colBuf[byteIdx] = (uchar)(colBuf[byteIdx] | (1<<bitIdx));
  }

int dataOff = m_validityBytes;
for(int i=0; i<n; i++)
  {
   if(ArrowColumnIsInt64[c]==1)
      PackValue(colBuf,dataOff+i*ARROW_ELEM_SIZE,lv);
   else
      PackValue(colBuf,dataOff+i*ARROW_ELEM_SIZE,dv);
  }

RtlMoveMemory(m_pBase+offset+colBase,colBuf,m_columnBlockBytes);

This loop runs once per column per batch, not once per tick per WinAPI call - the six RtlMoveMemory calls per batch (one per column, plus two for the header's write-in-progress and stable flips) are the entire cost of publishing, regardless of whether the batch holds 64 ticks or 4,096. The header's opening "danger" flag - the step that happens before any column data is touched - is what makes the seqlock from Section 6 actually work:

// 1) mark slot as "being written" immediately so a reader that is
//    mid-poll on this slot backs off.
ArrowSlotHeader hdr;
ZeroMemory(hdr);
hdr.generation=writeGen;              // odd -> write in progress
hdr.batch_seq=m_batchSeq;
hdr.tick_count=n;
hdr.column_count=ARROW_TICK_COLUMNS;
hdr.write_timestamp_msc=(long)GetTickCount64();
hdr.capacity=m_capacity;

uchar hb[];
ArrayResize(hb,ARROW_HEADER_SIZE);
PackHeader(hb,0,hdr);
RtlMoveMemory(m_pBase+offset,hb,ARROW_HEADER_SIZE);

Only after this odd-generation write lands does the column loop from above run, followed by the even-generation publish shown in Section 6. Deinit() is the class's last substantial method - it unmaps the view and closes the mapping handle, called from the EA's OnDeinit() so the OS-level mapping refcount drops cleanly on shutdown rather than lingering until the terminal process exits. (Rounding out the class's public surface: the constructor just zero-initializes member variables, the destructor calls Deinit() as a safety net, and SlotSize()/Capacity()/BatchSeq() are one-line getters used by the diagnostic script in Section 9 - none change behavior, so they're omitted here for space.)

//+------------------------------------------------------------------+
//| Deinit                                                           |
//| Releases the mapped view and the mapping handle.                 |
//| Safe to call more than once.                                     |
//+------------------------------------------------------------------+
void Deinit(void)
  {
   if(m_pBase!=0)   { UnmapViewOfFile(m_pBase); m_pBase=0; }
   if(m_hMapping!=0){ CloseHandle(m_hMapping);  m_hMapping=0; }
  }

The EA: ArrowTickStreamer.mq5

What it implements. A thin Expert Advisor that sits on top of CArrowTickBridge - it owns none of the Arrow layout logic itself, only the decision of when to call WriteBatch().

How it operates. OnTick() appends every incoming tick to a small RAM array; a size trigger (InpBatchSize) or a millisecond timer (InpFlushMs) calls FlushPending(), which hands the accumulated ticks to the bridge in one call.

Why this approach. Publishing on every single tick would mean a full seqlock write-and-flip cycle per tick, defeating the point of batching for the Python side. Accumulating first amortizes that fixed publish cost across many ticks, which is exactly what Part 2's benchmark measures the payoff of.

OnInit() runs the schema contract check from Section 5 before it does anything else, including before it ever calls into WinAPI - a bad column count fails fast with a log message instead of silently mis-writing shared memory:

//+------------------------------------------------------------------+
//| Expert initialization function                                   |
//+------------------------------------------------------------------+
int OnInit()
  {
   string reason;
   CArrowTickBridge probe;
   if(!probe.ContractOk(reason))
     {
      PrintFormat("FATAL: Arrow schema contract mismatch - %s. Refusing to start.",reason);
      return(INIT_PARAMETERS_INCORRECT);
     }

   if(!g_bridge.Init(InpMappingName,InpBatchSize))
     {
      Print("FATAL: could not initialise ArrowTickBridge shared memory mapping.");
      return(INIT_FAILED);
     }
   ...
   EventSetMillisecondTimer(50);
   return(INIT_SUCCEEDED);
  }

Its runtime job is to accumulate ticks in a small RAM buffer and flush them into the bridge either when the buffer fills or when a timeout elapses, so that a quiet symbol doesn't leave stale data sitting unpublished for minutes:

//+------------------------------------------------------------------+
//| Expert tick function                                             |
//+------------------------------------------------------------------+
void OnTick()
  {
   MqlTick t;
   if(!SymbolInfoTick(_Symbol,t))
      return;

   g_pending[g_pendingCount] = t;
   g_pendingCount++;

   if(g_pendingCount>=ArraySize(g_pending))
      FlushPending();
  }

//+------------------------------------------------------------------+
//| Timer - guarantees low-traffic symbols still flush promptly      |
//+------------------------------------------------------------------+
void OnTimer()
  {
   if(g_pendingCount>0 && (GetTickCount64()-g_lastFlushMs)>=(ulong)InpFlushMs)
      FlushPending();
  }

The InpBatchSize input controls both the shared memory slot capacity and how many ticks accumulate before a flush - 256 is a reasonable default for XAUUSD M5, where tick arrival is bursty around news but otherwise moderate. InpFlushMs (default 250) is the safety net: even on a symbol with sparse ticks, Python never waits more than a quarter second to see fresh data. Both are tunable per-symbol without recompiling - though here they're plain EA inputs rather than a FILE_COMMON manifest, since there's no cross-language parameter that needs syncing beyond the schema contract already covered in Section 5. The actual publish call is a single line, wrapped with bookkeeping so a failed write doesn't silently drop ticks:

//+------------------------------------------------------------------+
//| Publishes whatever ticks are currently buffered                  |
//+------------------------------------------------------------------+
void FlushPending()
  {
   if(g_pendingCount<=0)
      return;

   if(g_bridge.WriteBatch(g_pending,g_pendingCount))
     {
      g_totalTicksSent += g_pendingCount;
      g_totalBatches++;
     }
   else
      Print("WARNING: WriteBatch failed - shared memory mapping may have been closed.");

   g_pendingCount = 0;
   g_lastFlushMs  = GetTickCount64();
  }

On shutdown, OnDeinit() flushes anything still buffered rather than dropping the tail of the session, then releases the bridge - this is the call that reaches Deinit() from Section 7:

//+------------------------------------------------------------------+
//| Expert deinitialization function                                 |
//+------------------------------------------------------------------+
void OnDeinit(const int reason)
  {
   EventKillTimer();
   if(g_pendingCount>0)
      FlushPending();
   g_bridge.Deinit();
   PrintFormat("ArrowTickStreamer stopped. total_ticks=%I64d total_batches=%I64d",
               g_totalTicksSent,g_totalBatches);
  }

The diagnostic script: ArrowBridgeSelfTest.mq5

What it implements. A standalone script (not an EA - it runs once and exits) that exercises CArrowTickBridge directly against real terminal tick history, completely independent of whether ArrowTickStreamer.mq5 is even running.

How it operates. It pulls a fixed, indexable snapshot of ticks with CopyTicks(), runs it through the identical ContractOk()/Init()/WriteBatch() calls the live EA uses, and prints the first and last rows it wrote so they can be diffed by eye against Part 2's Python output.

Why this approach. A live EA's tick stream never stops moving, which makes it hard to answer "did tick 47 arrive with the right bid value" after the fact. A frozen snapshot plus explicit print statements turns a fuzzy live-debugging problem into a simple side-by-side value comparison - and lets this article's half of the bridge be verified as correct entirely on its own, before Part 2's Python code enters the picture at all.

//+------------------------------------------------------------------+
//| Script program start function                                    |
//+------------------------------------------------------------------+
void OnStart()
  {
   string reason;
   CArrowTickBridge bridge;
   if(!bridge.ContractOk(reason))
     {
      PrintFormat("FAIL: contract check - %s",reason);
      return;
     }
   Print("PASS: schema contract (ARROW_TICK_COLUMNS matches metadata arrays)");

   if(!bridge.Init(InpMappingName,InpBatchSize))
     {
      Print("FAIL: could not create/map shared memory segment");
      return;
     }

   MqlTick history[];
   int copied = CopyTicks(_Symbol,history,COPY_TICKS_ALL,0,InpBatchSize);
   if(copied<=0)
     {
      Print("FAIL: CopyTicks returned no data - open a chart for this symbol first");
      bridge.Deinit();
      return;
     }

   if(!bridge.WriteBatch(history,copied))
     {
      Print("FAIL: WriteBatch rejected the batch");
      bridge.Deinit();
      return;
     }
   PrintFormat("PASS: wrote %d real ticks from CopyTicks() into shared memory, batch_seq=%I64d",
               copied,bridge.BatchSeq());

   PrintFormat("First tick: time_msc=%I64d bid=%.5f ask=%.5f",
               history[0].time_msc,history[0].bid,history[0].ask);
   PrintFormat("Last  tick: time_msc=%I64d bid=%.5f ask=%.5f",
               history[copied-1].time_msc,history[copied-1].bid,history[copied-1].ask);

   bridge.Deinit();
  }

The printed first/last rows are meant to be read side by side with whatever Part 2's arrow_tick_reader.py reports for the same mapping: if the bid/ask/time_msc values match on both sides, the columnar layout, the seqlock publish, and the Python-side import are all confirmed correct end to end in one pass, rather than being debugged separately.

Validating the MetaTrader 5 side

Because this half of the project has no trading logic, "testing" here means byte-level correctness, not strategy performance - there's no equity curve to show, and forcing one in as a placeholder would misrepresent what this article does.

Check	Method
Correctness	ArrowBridgeSelfTest.mq5, Section 9 - pulls a real batch via CopyTicks(), pushes it through WriteBatch(), and prints first/last row values ready to diff against Part 2's Python output on the same mapping
Symbol/Timeframe	XAUUSD/M5 (default; the bridge itself is symbol-agnostic)

To run it: open an XAUUSD chart so the terminal has recent tick history, then run ArrowBridgeSelfTest.mq5 as a script - it will print PASS/FAIL for the contract check, the mapping creation, and the write itself, followed by the first and last tick values it wrote. Part 2 picks up from exactly this point, reading the same mapping name and comparing against these same printed values. Throughput and the full zero-copy-versus-legacy-struct benchmark are covered in Part 2, once there's a Python side to measure against.

Edge cases and pitfalls

Pointer serialization is a trap, not a shortcut. It's tempting to write the writer's raw pointer values into the shared header so a reader can "just use them." Don't - MapViewOfFile gives no guarantee both processes see the mapping at the same virtual address. Compute every offset from capacity alone, as this article does, and let each process add its own base address locally. Part 2 shows the reader side of this same discipline.

Capacity is part of the contract, not a free parameter. Every consumer of a given mapping - the EA, the self-test script, and eventually Part 2's Python reader - must be initialized with the exact same capacity value. A mismatch doesn't crash; it silently misreads every offset downstream, since the bytes are still there, just misinterpreted at the wrong positions. Treat capacity as fixed per mapping name, the same way the six-column schema is fixed.

Symbol or account changes mid-session. The mapping name is fixed at OnInit() (InpMappingName). Running multiple EAs across multiple symbols requires a distinct mapping name each - two EAs racing to write the same named segment with different schemas or capacities will corrupt each other's slots silently, since neither side is aware the other exists.

Conclusion and what's next

At this point the MetaTrader 5 half of the bridge is complete and independently verifiable: a schema contract that fails loudly rather than silently, a columnar memory layout that needs no translation step, a seqlock that keeps a fast writer and a slow reader from ever corrupting each other's view, and a diagnostic script that proves all of it works using nothing but MetaTrader 5's own tick history. Nothing here depends on Python existing yet - which is exactly the point of splitting the verification this way.

Coming in Part 2. The companion article picks up exactly where this one ends: reconstructing real ArrowSchema/ArrowArray C structs in Python with ctypes, importing them into pyarrow with zero copies, and a runnable benchmark that measures the payoff against a legacy struct.unpack read path - with real numbers, not just the claim.

File	Type	Description
ArrowBufferWriter.mqh	Include	CArrowTickBridge class - Arrow columnar layout, WinAPI shared memory, seqlock publish
ArrowTickStreamer.mq5	Expert Advisor	Collects live ticks and flushes batches into the bridge on size or timeout
ArrowBridgeSelfTest.mq5	Script	Correctness check: writes a real CopyTicks() batch and prints values ready to diff against Part 2's Python output

Attached files |

Download ZIP

MQL5.zip (8.18 KB)

Warning: All rights to these materials are reserved by MetaQuotes Ltd. Copying or reprinting of these materials in whole or in part is prohibited.

This article was written by a user of the site and reflects their personal views. MetaQuotes Ltd is not responsible for the accuracy of the information presented, nor for any consequences resulting from the use of the solutions, strategies or recommendations described.