Building a PDF Creation Library in MQL5 (Part 2): Why PDFs Break

Contents

• Introduction
• Where we are
• A viewer reads a PDF backwards
• The table is a set of promises
• Twenty bytes, one of them invisible
• Breaking it on purpose
• What real viewers do about it
• The four checks
• The code
• What the report says
• When it goes wrong
• The cheat card
• Project structure
• Conclusion

Introduction

A PDF file can open perfectly, display every line of text correctly, and still contain structural errors that go completely unnoticed. This is one of the reasons PDF has a reputation for being difficult to work with. The format itself is relatively simple. The bookkeeping is not, and the bookkeeping can fail silently. In Part 1, we built a PDF by hand. It was 592 bytes long, contained five objects, and opened in every viewer we tested. But that small file was fragile: seven of its numbers represented byte positions, and changing a single character of text could make several of them incorrect. The document might still look exactly as expected, even though its internal references no longer matched its actual contents.

This article explores that fragility through a practical experiment. We will deliberately break the same PDF in three different ways and open each modified file in two independent PDF implementations. All three files will be accepted and all three will still yield the correct text. Neither implementation refuses a file, and the only trace of the damage is a set of warnings that one of them prints only when warnings are explicitly switched on. The central lesson is simple: a PDF opening successfully is not evidence that the PDF is structurally correct. Understanding that distinction is essential when building tools that generate or manipulate PDF files.

By the end of the article, we will have developed PdfValidate.mqh, a validation class that reads a PDF, checks four structural properties that PDF viewers rely on, and reports discrepancies with the affected object number, the expected byte position, and the actual byte position. The validator can also be pointed at PDF files created by other tools, as long as they are written with a classic cross-reference table; the constructs it declines to read are listed alongside the code. This class will become an important foundation for the rest of the series. Starting with Part 3, our library will generate these byte positions automatically, and we will need an independent way to verify that it generates them correctly.

Where we are

Part 1 built hello.pdf by hand and wrote Demo01_HelloPdf.mq5 to reproduce it from MQL5. This part reuses that file unchanged, breaks copies of it, and adds the first file that will stay in the library for the rest of the series.

A viewer reads a PDF backwards

Before breaking anything it is worth knowing what a viewer actually does when you double-click a PDF, because the order surprises people and it explains which failures are fatal and which are survivable.

Figure 1. A viewer starts at the end of the file and works backwards.

Figure 1 lists the six things it looks for, in the order it looks for them. Follow them from the top. The viewer seeks to the end of the file and reads backwards looking for startxref. It takes the number that follows and jumps to that byte, expecting the word xref. It reads the table of byte offsets it finds there, then the trailer just below, which tells it that /Root 1 0 R is the catalog. Only now, on step 6, does it look at an object — and it finds object 1 by looking up entry 1 in the table it just read.

Steps 1 to 4 are pure bookkeeping. No content, no drawing, no fonts. Four of the six things a viewer does before it can render a single character are byte arithmetic, and all four are performed on numbers that a generator had to calculate correctly.

This is also why PDF is fast on large documents. A viewer opening page 400 of a 600-page report does not read the first 399 pages; it reads the index, looks up the object it needs, and seeks straight to it. The price of that speed is that the index has to be right.

The table is a set of promises

Here is the table from hello.pdf next to the file it describes.

Figure 2. Each entry claims a byte position the file may or may not agree with.

Look at the arrows in Figure 2. Each one is a plain integer, and each one is a claim: "object 5 begins at byte 339". Now look at what is missing. Object 5 does not contain its own address. Nothing inside 5 0 obj says "I am at 339". The claim and the thing it describes are stored in two different places, hundreds of bytes apart, with no checksum, no length field and no marker tying them together.

That is the structural weakness, and it is worth stating as a rule: in a PDF, position is metadata. Most of the rest of the format carries its own delimiters — a dictionary is delimited by << and >>, a string by brackets, an object by obj and endobj. Stream data is an exception, since its extent comes from /Length rather than from a delimiter. But the one thing that is never self-describing is where anything is, and that is precisely the thing the format depends on most.

An analogy that holds up well: the objects are books on a shelf and the xref is the library catalog, listing each book by the number of centimeters from the left-hand wall. Slide one extra book onto the shelf and every measurement to its right is wrong, even though no catalog card was touched and every book is still there.

Twenty bytes, one of them invisible

The second trap is in the shape of an entry rather than its value. Everything in this section describes the classic cross-reference table, which is the form this series writes; from PDF 1.5 the index may instead be stored as a compressed cross-reference stream, and Part 35 covers that form.

Figure 3. The fixed layout of an xref entry, byte by byte.

Count the cells in Figure 3. Ten digits of byte offset, a space, five digits of generation number, a space, one flag letter, and then a two-byte end-of-line. The specification allows that ending to be written in three ways — a space followed by a carriage return, a space followed by a line feed, or a carriage return followed by a line feed — so whether the entry carries a trailing space depends on which one the generator chose. What never changes is the total: twenty bytes, every time, regardless of how small the numbers are. Object 1 at byte 9 is written 0000000009, padded with eight zeros that carry no information at all.

The padding is not decoration. It is what makes the table seekable. A viewer that wants object 340 does not read 339 entries to reach it: in a single-run table that starts at object 0 — the form this series writes — it multiplies 340 by 20, adds that to the position of the first entry, and jumps. A table split into several numbered runs needs one addition more, but the arithmetic is of the same kind. That is the same trick as an array index, and it is why the format is willing to waste eight zeros per entry.

The consequence is that the twentieth byte is load-bearing. Figure 3 shows the arrangement our own files use: byte 18 is a plain space and byte 19 is a line feed, so the entry is nineteen visible characters plus one line feed. Delete the space — which no text editor displays, and which looks like tidying up — and the entries become nineteen bytes. Every jump the viewer computes now lands one byte further into the wrong place with each entry, and by entry five it is reading digits from the middle of a neighboring line. A file written with \r\n endings has no trailing space to delete, but its exposure is the same: remove either half of the ending and the same drift follows.

The f in the first entry is the last piece of vocabulary. n means the object is in use; f means it is free, that is, deleted. The free entries form a linked list, and 0000000000 65535 f is its head: offset zero, and the generation number 65535, which is the largest value the field can hold. It exists so that a file edited in place can reuse an object number, and for a file we write from scratch it is a constant. Copy it and move on.

Breaking it on purpose

Now the experiment. We take hello.pdf and produce three damaged copies, each broken in one specific way.

The first is the one that happens in real life. Nobody edits an xref table by accident; people edit the text and forget that text has a size.

Figure 4. Two extra characters falsify three numbers nobody edited.

Read Figure 4 from the top. We change (Hello, PDF!) Tj to (Hello, world!) Tj. That is two characters more. Now trace the consequences down the left column, none of which we touched:

• /Length 43 in object 4 described a stream that is now 45 bytes long.
• 0000000339 claimed object 5 was at byte 339; everything below the stream has slid down, so it is at 341.
• startxref 409 claimed the table began at byte 409; it now begins at 411.

Three numbers, all false, from one edit that had nothing to do with any of them. This is the whole problem of the series in one picture, and it is why Part 3 exists.

The second break removes the trailing space from each xref entry, as discussed above. The third points startxref at byte 400 instead of 409, which lands in the middle of object 5 — the failure mode of any generator that edits a file after computing its offsets.

What real viewers do about it

Here is where the article stops being theoretical. We opened all four files in two independent implementations: PDFium, the rendering engine Chrome and Edge use to display PDFs, and pypdf, a Python library that tooling uses to read and rewrite PDF structure rather than to draw it. pypdf parses that structure explicitly, and in its default mode it rebuilds what it cannot trust instead of refusing the file; its strict mode, which we are not using here, turns the same findings into errors. Two implementations and four small files are a demonstration, not a survey — what follows is what these two did, not a statement about every PDF consumer.

Figure 5. Every broken file opens, in both engines, with no visible complaint.

The two engine columns of Figure 5 read the same all the way down, and the last column is the number of problems our own validator finds in each file. Every file is accepted. Every file still yields the correct text — drawn by PDFium, extracted by pypdf. hello_text.pdf displays "Hello, world!" exactly as intended, despite a stream whose declared length is wrong and a startxref that points into the middle of an object.

They manage it by giving up on the index. When a viewer cannot make sense of the table, it falls back to repair mode: it scans the entire file from byte zero looking for anything of the form N 0 obj, builds its own table from what it finds, and carries on. Our validator uses the same scan, for the same reason.

pypdf does the same thing, and if you turn its warnings on it will tell you so:

incorrect startxref pointer(1)
Ignoring wrong pointing object 5 0 (offset 339)
Object 5 0 found

Note the second line. pypdf independently identified the same defect our validator reports — object 5, offset 339 — and then worked around it and carried on. Those warnings are the only sign of it, and with warnings left at their default the caller sees nothing at all.

This behavior is a courtesy, not a guarantee. Nothing in the specification requires a consumer to repair a broken file, and the cases where the courtesy runs out are exactly the cases that matter for a report you send to somebody:

• Scanning is linear. Repairing a 400-byte file is instant. Repairing a 40 MB report with 60,000 objects is not, and some tools impose a time limit.
• Repair guesses. If an object number appears twice — which happens when a generator appends rather than rewrites — the scan has to pick one, and it may not pick the one your table meant.
• Not every consumer renders. Pre-flight tools, PDF/A validators, archival systems and many server-side libraries check the structure rather than draw it, and they reject what a viewer forgives.
• A compressed cross-reference stream cannot be rebuilt by this kind of scan. From PDF 1.5 the index can itself live inside a compressed object, and Part 35 of this series uses that format. A scan may still find the top-level objects that carry an ordinary N 0 obj header, but the objects packed inside an object stream have no such header, and the mapping from object number to location is gone with the index.

So the conclusion is not "viewers are forgiving, relax". It is that the viewer is not a test. A file that opens on your machine may be quietly relying on repair, and you will find out when a client's archiving system rejects it.

The four checks

We need our own test. A useful validator does not have to understand the whole format. The four properties below are the ones steps 1 to 4 of Figure 1 depend on, and they are also the ones our own generator computes and can therefore get wrong.

1. The header is at byte 0. %PDF- must be the first five bytes. A byte-order mark in front of it is invisible in every editor that produces it.
2. startxref points at the word xref. If it does not, the table cannot be found, and the only way left to locate an object is to scan the whole file for its header.
3. Entry i points at i 0 obj. And the entries are twenty bytes each, so the table can be indexed rather than read. Generation 0 is an assumption here: an object of a higher generation is legal, and this check will flag it.
4. /Length reaches endstream. Step forward by the declared length and the keyword should be there.

These four are not the whole of a PDF, and passing them is necessary rather than sufficient. A viewer needs a great deal more before it can draw a page — a trailer with a usable /Root, a catalog, a page tree, resources, fonts and content streams — and a full structural validation would also examine the trailer dictionary, dictionary syntax, object streams, cross-reference streams, incremental updates, the %%EOF marker, object generations and the references between objects. None of that is in scope here. What these four do cover is the byte arithmetic a generator works out for itself, which is where our own bugs will be, and a failure in any of them prevents normal indexed opening and may force repair mode.

The code

PdfValidate.mqh is the first file that stays in the library. It is a debugging tool rather than part of the writer, and it earns its place because every later part invents a new way to produce a broken file.

The file opens with its guard and two limits:

//+------------------------------------------------------------------+
//|                                                  PdfValidate.mqh |
//+------------------------------------------------------------------+
#ifndef PDF5_VALIDATE_MQH
#define PDF5_VALIDATE_MQH

//--- A file with more objects than this is refused rather than risking a
//--- runaway loop on a file that is not a PDF at all.
#define PDF_MAX_OBJECTS 4096

//--- How far back from the end of the file we look for "startxref". The
//--- specification allows the trailer to be long, but in practice it is a
//--- few hundred bytes, and scanning the whole file backwards is wasteful.
#define PDF_TAIL_SCAN   2048

Both limits exist because this class will be pointed at files it did not write, including files that are not PDFs at all. A validator that hangs on malformed input is worse than no validator, so every loop in the file has a bound. The numbers are chosen for the files this series produces, and they are worth knowing before the class is pointed at something larger: a real report can hold far more than 4096 objects, and a long trailer can push startxref further than 2048 bytes from the end. In both cases the class declines or reports a problem rather than guessing, and the answer is to raise the constant rather than to trust the verdict. Sizes and offsets are held in int for the same reason of simplicity, which puts a two-gigabyte ceiling on the files it can describe.

Each finding is a small structure rather than a string, so a caller can act on it and not only print it:

//+------------------------------------------------------------------+
//| One problem found in the file.                                   |
//+------------------------------------------------------------------+
struct SPdfProblem
  {
   string            message;    // what is wrong, in one sentence
   long              position;   // byte position it concerns, -1 if none
   int               object_no;  // object number it concerns, -1 if none
  };

Keeping the byte position as a separate field rather than only inside the message is what will let Part 3's unit tests assert "the failure was at byte 258" without matching text. A message is for a human; the two integers are for code.

Now the class itself:

//+------------------------------------------------------------------+
//| CPdfValidator                                                    |
//+------------------------------------------------------------------+
class CPdfValidator
  {
private:
   uchar             m_data[];             // the whole file in memory
   int               m_size;               // its length in bytes

   //--- What the xref table claims.
   int               m_claimed[PDF_MAX_OBJECTS];
   int               m_claimed_count;

   //--- What a scan of the file actually finds.
   int               m_actual[PDF_MAX_OBJECTS];
   int               m_actual_count;

   SPdfProblem       m_problems[];
   int               m_problem_count;

Two arrays, side by side: what the table promises and what the file contains. Almost every useful message this class produces comes from comparing them. A validator that only checked "does entry 5 point at object 5" could report a failure; one that holds both numbers can report the fix, which is the difference between a tool you use once and a tool you keep.

   //--- byte level helpers
   bool              Load(const string filename);
   int               Find(const string needle, const int from) const;
   int               FindLast(const string needle, const int from) const;
   bool              Match(const string needle, const int at) const;
   string            Text(const int at, const int length) const;
   int               ReadInt(const int at, int &end) const;
   int               SkipSpace(const int at) const;

   //--- the four checks
   void              CheckHeader(void);
   int               CheckStartXref(void);
   void              CheckXrefTable(const int xref_pos);
   void              CheckOffsets(void);
   void              CheckStreams(void);

   //--- a full scan, used to say what the offsets should have been
   void              ScanObjects(void);

   void              Problem(const string message, const long position = -1,
                             const int object_no = -1);

public:
                     CPdfValidator(void) { Reset(); }

   void              Reset(void);
   bool              Check(const string filename);

   int               ProblemCount(void) const { return(m_problem_count); }
   SPdfProblem       ProblemAt(const int i) const { return(m_problems[i]); }
   int               ObjectCount(void) const { return(m_actual_count); }

   //--- Named ByteCount and not FileSize: a method called FileSize would
   //--- hide the terminal's FileSize() function inside every method of
   //--- this class, and Load() needs to call it.
   int               ByteCount(void) const { return(m_size); }

   void              PrintReport(const string filename);
  };

That comment on the last method is worth reading twice. It describes real MQL5 behavior, not a style preference. A member function named FileSize would shadow the global FileSize() in all class methods. Load() needs the global function. The compiler error you get is about argument counts and does not mention shadowing at all.

Reading the file

//+------------------------------------------------------------------+
//| Reads the whole file into memory.                                |
//+------------------------------------------------------------------+
bool CPdfValidator::Load(const string filename)
  {
   int handle = FileOpen(filename, FILE_READ | FILE_BIN);
   if(handle == INVALID_HANDLE)
     {
      Problem(StringFormat("cannot open %s, error %d", filename,
                           GetLastError()));
      return(false);
     }

   m_size = (int)FileSize(handle);
   if(m_size <= 0)
     {
      FileClose(handle);
      Problem("the file is empty");
      return(false);
     }

   ArrayResize(m_data, m_size);
   int read = (int)FileReadArray(handle, m_data, 0, m_size);
   FileClose(handle);

   if(read != m_size)
     {
      Problem(StringFormat("read %d bytes of %d", read, m_size));
      return(false);
     }
   return(true);
  }

The whole file goes into memory. That is not laziness: Figure 1 showed that a PDF is read from its end before it is read from its start, so a streaming reader would have to seek backwards immediately. Reading into a uchar array through FileReadArray also means every position we report is a real byte position, which is what the reader will be counting when they go looking in a hex editor.

Note that Load calls Problem() before m_size is set. That is safe because Reset() cleared the list first, and it means a file that does not exist produces a report rather than a silent false.

Finding things in a byte array

MQL5 has good string functions and we cannot use them, because converting 40 MB of binary to a string to call StringFind would be both slow and wrong — a zero byte inside an image would truncate it. So the class carries four small primitives that work on bytes.

//+------------------------------------------------------------------+
//| True when the bytes at `at` equal `needle`.                      |
//+------------------------------------------------------------------+
bool CPdfValidator::Match(const string needle, const int at) const
  {
   int n = StringLen(needle);
   if(at < 0 || at + n > m_size)
      return(false);

   for(int i = 0; i < n; i++)
      if(m_data[at + i] != (uchar)StringGetCharacter(needle, i))
         return(false);
   return(true);
  }

Match compares a literal against the bytes at a position. Every keyword we look for — xref, obj, endstream — is plain ASCII, so comparing character codes with StringGetCharacter is exact. The bounds test on the first line is what makes it safe to call with a position read out of a damaged file, which is the normal case here.

//+------------------------------------------------------------------+
//| First position of `needle` at or after `from`, or -1.            |
//+------------------------------------------------------------------+
int CPdfValidator::Find(const string needle, const int from) const
  {
   int n = StringLen(needle);
   for(int i = MathMax(0, from); i + n <= m_size; i++)
      if(Match(needle, i))
         return(i);
   return(-1);
  }

//+------------------------------------------------------------------+
//| Last position of `needle` at or before `from`, or -1.            |
//+------------------------------------------------------------------+
int CPdfValidator::FindLast(const string needle, const int from) const
  {
   int n = StringLen(needle);
   for(int i = MathMin(from, m_size - n); i >= 0; i--)
      if(Match(needle, i))
         return(i);
   return(-1);
  }

Forwards and backwards search, both built on Match. This is the naive algorithm, comparing at every position, and for a validator that is the right choice: the files are small, the patterns are short, and a Boyer-Moore implementation would add fifty lines that could themselves be wrong. FindLast exists for exactly one caller — finding startxref from the end — which is step 2 of Figure 1.

Text is only ever used to quote a fragment back to the reader in a message. Escaping matters here. Without it, a quoted 12-byte fragment can split into multiple lines in the Experts log, and the byte position may appear on a separate line.

//+------------------------------------------------------------------+
//| Skips spaces, tabs and line endings.                             |
//+------------------------------------------------------------------+
int CPdfValidator::SkipSpace(const int at) const
  {
   int i = at;
   while(i < m_size)
     {
      uchar c = m_data[i];
      if(c != ' ' && c != '\t' && c != '\r' && c != '\n')
         break;
      i++;
     }
   return(i);
  }

//+------------------------------------------------------------------+
//| Reads a decimal integer at `at`; `end` receives the position     |
//| just past the last digit. Returns -1 when there is no digit      |
//| there.                                                           |
//+------------------------------------------------------------------+
int CPdfValidator::ReadInt(const int at, int &end) const
  {
   int i = SkipSpace(at);
   int value = 0;
   bool any = false;

   while(i < m_size && m_data[i] >= '0' && m_data[i] <= '9')
     {
      value = value * 10 + (int)(m_data[i] - '0');
      any = true;
      i++;
     }

   end = i;
   return(any ? value : -1);
  }

ReadInt() returns the value and, through end, the position after the last digit, because every place we read a number we immediately need to read another one after it — 0 6 in the xref header, then the offset and generation of each entry. Returning -1 rather than 0 for "no digits here" is what lets the caller tell startxref 0 apart from startxref followed by garbage.

Check 1: the header

//+------------------------------------------------------------------+
//| Check 1: the file must begin with the version marker.            |
//+------------------------------------------------------------------+
void CPdfValidator::CheckHeader(void)
  {
   if(Match("%PDF-", 0))
      return;

   int found = Find("%PDF-", 0);
   if(found > 0)
      Problem(StringFormat("%%PDF- is at byte %d, not byte 0 - there are %d "
                           "stray bytes in front of it", found, found), 0);
   else
      Problem("this file does not contain %PDF- at all", 0);
  }

The interesting branch is the middle one. If %PDF- is present but not at byte 0, we say how many bytes are in front of it, and that number is the diagnosis: 3 means a UTF-8 byte-order mark, 2 means UTF-16, anything else means the file was concatenated with something. Reporting "invalid header" would have thrown that information away.

Check 2: startxref

//+------------------------------------------------------------------+
//| Check 2: startxref, and where it points.                         |
//+------------------------------------------------------------------+
int CPdfValidator::CheckStartXref(void)
  {
   int tail = MathMax(0, m_size - PDF_TAIL_SCAN);
   int at = FindLast("startxref", m_size - 1);

   if(at < 0 || at < tail)
     {
      Problem("no startxref keyword in the last " +
              IntegerToString(PDF_TAIL_SCAN) + " bytes of the file");
      return(-1);
     }

The keyword has to be near the end. Searching the whole file would find the startxref of a previous revision in an incrementally-updated document and report the wrong one, so a file whose only startxref is 30 KB from the end is treated as having none.

   int end = 0;
   int xref_pos = ReadInt(at + 9, end);      // 9 == StringLen("startxref")
   if(xref_pos < 0)
     {
      Problem("startxref is not followed by a number", at);
      return(-1);
     }

   if(xref_pos >= m_size)
     {
      Problem(StringFormat("startxref says %d but the file is only %d bytes",
                           xref_pos, m_size), at);
      return(-1);
     }

   if(!Match("xref", xref_pos))
     {
      Problem(StringFormat("startxref points at byte %d, which holds \"%s\" "
                           "and not \"xref\"", xref_pos,
                           Text(xref_pos, 12)), xref_pos);
      return(-1);
     }

   return(xref_pos);
  }

Three failures, three different messages. The last one quotes the twelve bytes that are actually at the named position, and that quotation is the most useful line this class prints: it tells you immediately whether you are a few bytes off or in a completely different region of the file.

Check 3: the table

//+------------------------------------------------------------------+
//| Check 3a: read the table itself.                                 |
//+------------------------------------------------------------------+
void CPdfValidator::CheckXrefTable(const int xref_pos)
  {
   int end = 0;
   int first = ReadInt(xref_pos + 4, end);   // 4 == StringLen("xref")
   int count = ReadInt(end, end);

   if(first < 0 || count < 0)
     {
      Problem("the xref table has no \"first count\" line", xref_pos);
      return;
     }

   if(first != 0)
      Problem(StringFormat("the table starts at object %d; this series only "
                           "writes tables that start at 0", first), xref_pos);

   if(count > PDF_MAX_OBJECTS)
     {
      Problem(StringFormat("the table claims %d objects, which is more than "
                           "this validator will read", count), xref_pos);
      return;
     }

A real xref may be split into several numbered runs — 0 6 then 12 3 — and this validator handles only the single-run form that the series writes. Saying so in the message is better than pretending: a reader pointing this at a file from another generator deserves to know the tool declined rather than passed.

//--- The entries begin after the line ending that follows "first count".
   int at = end;
   while(at < m_size && (m_data[at] == '\r' || m_data[at] == '\n'))
      at++;

   m_claimed_count = 0;
   for(int i = 0; i < count; i++)
     {
      int entry = at + i * 20;
      if(entry + 20 > m_size)
        {
         Problem(StringFormat("the table is cut short after %d entries", i),
                 entry);
         return;
        }

at + i * 20 is the whole point of Figure 3, written as code. We do not read the entries one after another; we index them, because that is what a viewer does. A validator that parsed the entries line by line would happily accept nineteen-byte entries and miss the second of our three breakages entirely.

      //--- Bytes 17 and 18 of every entry are the type flag and a space.
      //--- If they are not where they belong, the entries are not 20 bytes
      //--- long, and every entry after this one is being misread.
      uchar flag = m_data[entry + 17];
      if(flag != 'n' && flag != 'f')
        {
         Problem(StringFormat("entry %d does not have its type flag at byte "
                              "17 - the 20-byte rule is broken here, and "
                              "every entry below it is misaligned", i),
                 entry, i);
         return;
        }

      int unused = 0;
      int offset = ReadInt(entry, unused);
      m_claimed[m_claimed_count++] = (flag == 'f') ? -1 : offset;
     }
  }

Checking byte 17 is a cheap way to detect a broken twenty-byte rule. If the entries are the right size, that byte is always n or f; if they are not, it is a digit or a line ending within a couple of entries. The check stops at the first failure and says every entry below is misaligned, because reporting five consecutive garbage offsets would bury the one fact that matters. It is an alignment test and not a syntax test: the ten-digit offset, the five-digit generation number, the separators and the exact end-of-line are not validated individually, so an entry that is twenty bytes long but malformed inside still passes here.

Free entries are stored as -1 rather than 0. A free object legitimately has offset 0, and without the distinction the check below would report "object 0 is not at byte 0" on every valid file.

Scanning for the truth

//+------------------------------------------------------------------+
//|Finds every "N 0 obj" in the file by reading it straight through. |
//+------------------------------------------------------------------+
void CPdfValidator::ScanObjects(void)
  {
   m_actual_count = 0;
   ArrayInitialize(m_actual, -1);

   int at = 0;
   while(at < m_size && m_actual_count < PDF_MAX_OBJECTS)
     {
      int found = Find(" obj", at);
      if(found < 0)
         break;

This is repair mode, the same fallback PDFium and pypdf used in Figure 5. We search for " obj" with its leading space so the obj inside endobj is skipped for free. The search is a byte scan and not a parse, so the same sequence occurring inside stream data, a string or a comment is taken for an object header. Every tool that rebuilds a table this way carries that risk; it is the price of a fallback that works when the index does not.

      //--- Walk backwards over " 0" and the object number to find the
      //--- first byte of "N 0 obj", which is what the offset must name.
      int i = found;
      while(i > 0 && m_data[i - 1] == ' ')
         i--;
      while(i > 0 && m_data[i - 1] >= '0' && m_data[i - 1] <= '9')
         i--;                                    // generation number
      while(i > 0 && m_data[i - 1] == ' ')
         i--;

      int start = i;
      while(start > 0 && m_data[start - 1] >= '0' && m_data[start - 1] <= '9')
         start--;                                // object number

      int unused = 0;
      int number = ReadInt(start, unused);
      if(number > 0 && number < PDF_MAX_OBJECTS)
        {
         if(number >= m_actual_count)
            m_actual_count = number + 1;
         m_actual[number] = start;
        }

      at = found + 4;
     }
  }

The backwards walk is the fiddly part: from the space before obj we step over the generation number, the space before it, and then the object number, to land on the first digit of N. That first digit is the byte an xref entry has to name, so getting this walk wrong would produce offsets that are consistently a few bytes early and a report full of false alarms.

Indexing m_actual by object number rather than appending means the array has gaps when a file skips numbers, which is why it is initialized to -1.

Comparing the two

//+------------------------------------------------------------------+
//| Check 3b: does every claimed offset point at the right object?   |
//+------------------------------------------------------------------+
void CPdfValidator::CheckOffsets(void)
  {
   for(int i = 1; i < m_claimed_count; i++)
     {
      int offset = m_claimed[i];
      if(offset < 0)
         continue;                               // a free entry, skip it

      string expected = IntegerToString(i) + " 0 obj";

      if(offset >= m_size)
        {
         Problem(StringFormat("object %d is said to be at byte %d, past the "
                              "end of a %d byte file", i, offset, m_size),
                 offset, i);
         continue;
        }

      if(Match(expected, offset))
         continue;   

The test is one line: are the bytes at the claimed position the text "5 0 obj"? Everything else in this method exists to produce a good message when they are not.

      //--- Say what is there, and where the object really is.
      string found_here = Text(offset, 16);
      int real = (i < m_actual_count) ? m_actual[i] : -1;
      if(real >= 0)
         Problem(StringFormat("object %d: the table says byte %d, but \"%s\" "
                              "is there; the object is really at byte %d, a "
                              "difference of %+d", i, offset, found_here,
                              real, real - offset), offset, i);
      else
         Problem(StringFormat("object %d: the table says byte %d, but \"%s\" "
                              "is there, and the object is nowhere in the "
                              "file", i, offset, found_here), offset, i);
     }
  }

%+d prints the sign, so the message ends with +2 or -14 rather than a bare number. When a generator is off by a constant amount — which is the usual case, because one mistake early in the file shifts everything after it equally — seeing the same +2 on every line identifies the cause immediately.

Check 4: stream lengths

//+------------------------------------------------------------------+
//| Check 4: every /Length must match the bytes in its stream.       |
//+------------------------------------------------------------------+
void CPdfValidator::CheckStreams(void)
  {
   int at = 0;
   while(at < m_size)
     {
      int keyword = Find("stream", at);
      if(keyword < 0)
         break;

      //--- Skip the "stream" inside the word "endstream".
      if(keyword >= 3 && Match("endstream", keyword - 3))
        {
         at = keyword + 6;
         continue;
        }

endstream contains stream, so a naive search finds every stream twice. The three-byte look-behind is cheaper than any cleverer approach and reads plainly. Like the object scan, this is a byte search and not a tokenizer: a stream sequence inside binary data or a string can be picked up as a keyword, and requiring an endstream to follow reduces that risk without removing it.

      //--- The dictionary with /Length is just above the keyword. Search
      //--- backwards for it rather than parsing the dictionary properly,
      //--- which Part 3 makes unnecessary by writing the value itself.
      int len_key = FindLast("/Length", keyword);
      if(len_key < 0)
        {
         Problem("a stream has no /Length above it", keyword);
         at = keyword + 6;
         continue;
        }

      int unused = 0;
      int claimed = ReadInt(len_key + 7, unused);   // 7 == "/Length"

Searching backwards for /Length instead of parsing the dictionary is a shortcut, and the comment says so. It fails on a stream whose /Length is an indirect reference — /Length 9 0 R, which some real files use — and on a dictionary that mentions /Length in a nested sub-dictionary; and if the current dictionary carries no /Length at all, the backwards search can reach the one belonging to an earlier object. All of these are outside what this series writes. A validator that quietly did the wrong thing here would be worse than one that admits its limits in a comment.

      //--- The stream data starts after the end-of-line that follows the
      //--- keyword. CR LF counts as one ending, not two.
      int data = keyword + 6;
      if(data < m_size && m_data[data] == '\r')
         data++;
      if(data < m_size && m_data[data] == '\n')
         data++;

      int stop = Find("endstream", data);
      if(stop < 0)
        {
         Problem("a stream is never closed by endstream", keyword);
         return;
        }

The two separate if statements, rather than an else if, are what makes CRLF count as a single ending. Getting this wrong shifts the start of the data by one byte and makes every stream in a Windows-authored file report an off-by-one error. Note that the specification requires an end-of-line after the stream keyword, while this code skips one if it is there rather than insisting on it: a file that omits the ending is still measured, and still passes when the length matches.

      //--- The test a viewer applies: step /Length bytes forward and see
      //--- whether "endstream" is sitting there. The specification allows
      //--- one optional end-of-line in front of the keyword that is not
      //--- counted in /Length, so both positions are accepted. Measuring
      //--- the distance to "endstream" and comparing it instead would call
      //--- a correct file wrong whenever that optional byte is present.
      int probe = data + claimed;
      int skip = probe;
      if(skip < m_size && m_data[skip] == '\r')
         skip++;
      if(skip < m_size && m_data[skip] == '\n')
         skip++;

      bool ok = (claimed >= 0 && probe <= m_size &&
                 (Match("endstream", probe) || Match("endstream", skip)));

      if(!ok)
        {
         int actual = stop - data;
         Problem(StringFormat("a stream declares /Length %d but endstream is "
                              "%d bytes further on, a difference of %+d",
                              claimed, actual, actual - claimed), len_key);
        }

      at = stop + 9;
     }
  }

This block was written the obvious way first, and the obvious way was wrong. The first version measured the distance from the start of the data to endstream, stripped a trailing line ending, and compared. It reported hello.pdf — a file we know is correct, and which every viewer accepts — as having a stream one byte too long.

The reason is simple: the specification allows an optional end-of-line before endstream , and it is not counted in /Length. Our file does not have that optional byte; the line feed before endstream is the last byte of the content itself, and /Length 43 counts it. Both arrangements are legal, and measuring cannot tell them apart.

Stepping forward by /Length and asking whether endstream is there can. It is also what a viewer does, which is the better reason: the purpose of this class is to predict how a viewer will behave, so where there is a choice it should perform the viewer's test rather than an equivalent-looking one.

Running the checks

//+------------------------------------------------------------------+
//| Runs every check. Returns true when the file has no problems.    |
//+------------------------------------------------------------------+
bool CPdfValidator::Check(const string filename)
  {
   Reset();

   if(!Load(filename))
      return(false);

   ScanObjects();
   CheckHeader();

   int xref_pos = CheckStartXref();
   if(xref_pos >= 0)
     {
      CheckXrefTable(xref_pos);
      CheckOffsets();
     }

   CheckStreams();
   return(m_problem_count == 0);
  }

ScanObjects() runs first so that every later message can name the real position of an object. The offset checks are skipped when startxref failed, because without a table there is nothing to compare against — and that produces the layered behavior the demo shows below. CheckStreams() runs regardless, since it needs no index at all.

//+------------------------------------------------------------------+
//| Prints the findings to the Experts log.                          |
//+------------------------------------------------------------------+
void CPdfValidator::PrintReport(const string filename)
  {
   PrintFormat("--- %s: %d bytes, %d objects found by scanning",
               filename, m_size, MathMax(0, m_actual_count - 1));

   if(m_problem_count == 0)
     {
      Print("    no problems");
      return;
     }

   for(int i = 0; i < m_problem_count; i++)
     {
      string where = "";
      if(m_problems[i].position >= 0)
         where = StringFormat(" [byte %d]", m_problems[i].position);
      PrintFormat("    %d. %s%s", i + 1, m_problems[i].message, where);
     }
  }

#endif // PDF5_VALIDATE_MQH

The header line reports the object count from the scan rather than from the table, because when the table is broken the scan is the only number that means anything.

Reproducing the breakages

Demo02_BreakIt.mq5 writes the good file and four damaged copies. It includes HelloPdf.mqh, which is Part 1's builder moved into a header so both demos can use it; the file is unchanged from Part 1 apart from being wrapped in a guard, so it is listed in the project structure rather than reprinted here.

//+-------------------------------------------------------------------+
//| Break 1: make a string two characters longer.                     |
//+-------------------------------------------------------------------+
string BreakLongerText(void)
  {
   string s = BuildHelloPdf();
   StringReplace(s, "(Hello, PDF!) Tj", "(Hello, world!) Tj");
   return(s);
  }

One StringReplace, two extra characters, three false numbers. Nothing else in the function is needed, which is the point: producing a broken PDF takes one line and no mistakes.

//+-------------------------------------------------------------------+
//| Break 2: remove the trailing space from every xref entry.         |
//+-------------------------------------------------------------------+
string BreakLostSpace(void)
  {
   string s = BuildHelloPdf();
   StringReplace(s, "65535 f \n", "65535 f\n");
   StringReplace(s, "00000 n \n", "00000 n\n");
   return(s);
  }

//+------------------------------------------------------------------+
//| Break 3: point startxref at the wrong byte.                      |
//+------------------------------------------------------------------+
string BreakStartXref(void)
  {
   string s = BuildHelloPdf();
   StringReplace(s, "startxref\n409\n", "startxref\n400\n");
   return(s);
  }

The space removal is written as two replacements because the free entry ends in f and the others in n. Both patterns include the line feed, so a f that happened to appear inside a string elsewhere in the file would not be touched.

//+------------------------------------------------------------------+
//| Break 1, half repaired: the same two extra characters, with      |
//| startxref corrected to 411 and the table left alone.             |
//+------------------------------------------------------------------+
string BreakLongerTextRepaired(void)
  {
   string s = BreakLongerText();
   StringReplace(s, "startxref\n409\n", "startxref\n411\n");
   return(s);
  }

This fourth variant is the interesting one. It applies the text change and then corrects startxref to the position the table has actually moved to — a plausible half-repair. As the report below shows, fixing the first failure only uncovers the next one.

//+------------------------------------------------------------------+
//| Script entry point.                                              |
//+------------------------------------------------------------------+
void OnStart(void)
  {
   Print("Writing one good file and four broken ones:");

   WriteRawFile(path_folder + "hello_good.pdf",      BuildHelloPdf());
   WriteRawFile(path_folder + "hello_text.pdf",      BreakLongerText());
   WriteRawFile(path_folder + "hello_text2.pdf",     BreakLongerTextRepaired());
   WriteRawFile(path_folder + "hello_space.pdf",     BreakLostSpace());
   WriteRawFile(path_folder + "hello_startxref.pdf", BreakStartXref());

   Print("");
   Print("Open them all in a PDF viewer, then run Demo02_Validate.");
  }

The second demo is shorter still:

#include <MQLPDF/PdfValidate.mqh>

//--- Every article in the series writes under one shared root folder, with
//--- its own subfolder
#define PDF_ROOT_FOLDER  "MQLPDF"
#define PDF_SUB_FOLDER   "Part 02"
string path_folder = PDF_ROOT_FOLDER + "//" + PDF_SUB_FOLDER + "//";



//+------------------------------------------------------------------+
//| Checks one file and prints what it found.                        |
//+------------------------------------------------------------------+
void CheckOne(CPdfValidator &validator, const string name)
  {
   validator.Check(name);
   validator.PrintReport(name);
   Print("");
  }

//+------------------------------------------------------------------+
//| Script entry point.                                              |
//+------------------------------------------------------------------+
void OnStart(void)
  {
   CPdfValidator validator;


   CheckOne(validator, path_folder + "hello_good.pdf");
   CheckOne(validator, path_folder + "hello_text.pdf");
   CheckOne(validator, path_folder + "hello_text2.pdf");
   CheckOne(validator, path_folder + "hello_space.pdf");
   CheckOne(validator, path_folder + "hello_startxref.pdf");
  }

One validator instance checks all five files, because Check() calls Reset(). That single CheckOne() line is also what makes this a tool rather than a demonstration: name any other PDF under MQL5\Files, including one produced by another program, and the report will tell you what these four checks make of it — keeping in mind the constructs listed above that the class declines to read.

What the report says

Figure 6. The validator's verdict where the viewers had none.

Running Demo02_Validate over all five files produces this:

--- hello_good.pdf: 592 bytes, 5 objects found by scanning
    no problems

--- hello_text.pdf: 594 bytes, 5 objects found by scanning
    1. startxref points at byte 409, which holds "j\nxref\n0 6\n0" and not
       "xref" [byte 409]
    2. a stream declares /Length 43 but endstream is 45 bytes further on, a
       difference of +2 [byte 258]

--- hello_text2.pdf: 594 bytes, 5 objects found by scanning
    1. object 5: the table says byte 339, but "j\n5 0 obj\n<< /Ty" is there;
       the object is really at byte 341, a difference of +2 [byte 339]
    2. a stream declares /Length 43 but endstream is 45 bytes further on, a
       difference of +2 [byte 258]

--- hello_space.pdf: 586 bytes, 5 objects found by scanning
    1. entry 1 does not have its type flag at byte 17 - the 20-byte rule is
       broken here, and every entry below it is misaligned [byte 438]

--- hello_startxref.pdf: 592 bytes, 5 objects found by scanning
    1. startxref points at byte 400, which holds ">\nendobj\nxre" and not
       "xref" [byte 400]

Read the first two reports together. hello_text.pdf has the wrong startxref, so the table was never read and the offset check never ran. hello_text2.pdf is the same file with startxref corrected, and now the object 5 error appears — the table says byte 339 ... really at byte 341, a difference of +2, which is the two characters we added, propagated exactly as Figure 4 predicted.

That layering is how a damaged PDF is repaired in practice. Each fix reveals the next problem, because a viewer, and this validator, stop at the first thing they cannot resolve. It is also a warning about testing: a generator that fixes only the failure it can see will produce files that fail one layer deeper.

The third report shows the twenty-byte rule doing its job. hello_space.pdf contains five perfectly correct byte offsets — we changed none of them — and it is still unreadable, because the entries are the wrong size. There is no message about a wrong offset here, and there should not be: the offsets are right, the geometry is wrong.

The last one is the mildest failure of the four. startxref points into object 5, so the index cannot be found, but nothing else in the file is untrue. Correct that single number and the file is valid again.

When it goes wrong

Three things surprise people the first time they use this class.

It reports "no problems" on a file that looks wrong in a viewer. That is correct behavior. The validator checks structure, not content. A page whose text is invisible because the content stream names a font that is not in /Resources is structurally perfect, and Part 5's demos hit this. Structure and drawing are separate failures, and this tool only sees the first kind.

It reports a problem on a PDF produced by a well-known program. Usually the file uses a feature we declined to implement: a cross-reference stream instead of a table, several numbered runs in one table, /Length as an indirect reference, or an object whose generation number is not 0. The messages name most of these, and none of them means the file is broken. The reverse holds as well: the class never looks at the trailer dictionary, the /Root entry, the page tree or the %%EOF marker, so a file missing any of those can still be reported clean. This class is a check on our output, and a reading aid on everyone else's.

It finds nothing on a file that will not open. Check the four bytes at the start with a hex viewer. If the header check passed, the file begins with %PDF-, and the remaining likely cause is a byte-order mark or transfer corruption further in. The scan count in the report header is a quick sanity check: a file that reports zero objects found is not a PDF.

The cheat card

Figure 7. The four checks, and the reason the tool exists.

Those are the four checks, and each carries a qualification worth keeping:

• %PDF- must be at byte 0 — no byte-order mark and no blank line in front of it.
• startxref must name the byte at which the word xref begins, and it is read from the end of the file, not the start.
• Entry i must point at i 0 obj, and every entry must be exactly twenty bytes long.
• /Length must reach endstream, allowing for one optional end-of-line between the two.

And the reason the card ends the way it does: opening a file says very little about its structure, and a check at the byte level says a good deal more. Where a document has to satisfy more than these four properties, preflight tools and PDF/A validators go further than this class ever will.

Project structure

MQL5/
├── Scripts/
│    └── MQLPDF/
│        └── Part 02/
│            ├── Demo02_BreakIt.mq5     // Create multiple PDF files (valid and invalid files)
│            └── Demo02_Validate.mq5    // Check the validation of the files
│
└── Include/
│    └── MQLPDF/
│        ├── HelloPdf.mqh                // To create simple PDF file (not in final library)
│        └── PdfValidate.mqh             // To check the validation of PDF file and list the problems
└── Files/
     └── MQLPDF/
         └── Part 02/
             ├── hello_good.pdf           // good file (output file)
             ├── hello_space.pdf          // invalid file (output file)
             ├── hello_startxref.pdf      // invalid file (output file)
             ├── hello_text.pdf           // invalid file (output file)
             └── hello_text2.pdf          // invalid file (output file)

Unzip into MQL5\Scripts\ and MQL5\Include\, compile both scripts, run Demo02_BreakIt.mq5 first and Demo02_Validate.mq5 second. The files appear in MQL5\Files\MQLPDF\Part 02\.

HelloPdf.mqh is demo support and will not survive into the library; from Part 3 the PDF is generated rather than typed. PdfValidate.mqh is the first permanent file, and it will be included by the test scripts of every part that follows.

#	File	Directory	Holds
1	PdfValidate.mqh	MQL5/Include/MQLPDF/	To check the validation of PDF file and list the problems
2	HelloPdf.mqh	MQL5/Include/MQLPDF/	To create simple PDF file (not in final library)
3	Demo02_BreakIt.mq5	MQL5/Scripts/MQLPDF/Part 02	Create multiple PDF files (valid and invalid files)
4	Demo02_Validate.mq5	MQL5/Scripts/MQLPDF/Part 02	Check the validation of the files
5	MQLPDF Library - Part 02.zip		archive containing all the attached files and their paths relative to the terminal's root folder.

Conclusion

A PDF is held together by integers that describe positions, and nothing in the format checks them. One edit to a string falsifies a stream length, an xref entry and startxref at once, and no part of the file records enough information to notice. Entries in the table are twenty bytes each because a viewer indexes them rather than reading them, which makes a single deleted space as damaging as a wrong number.

Real consumers hide all of this. Both implementations we tested accepted every broken file and still produced the correct text, because when the index fails they scan the file and rebuild it. Two of them on four small files is a demonstration and not a rule: the same damage in a larger file, or a different kind of damage, may not be repaired at all — and preflight tools, archival systems and the compressed index format that large reports use do not extend the courtesy in the first place.

You now have PdfValidate.mqh, which performs four of the checks a viewer depends on and reports the object number, the byte position and the difference. Point it at any PDF written with a classic cross-reference table, including ones you did not write. It will not pronounce a document valid, but it will tell you whether its byte arithmetic adds up. From here on, every part of this series is tested with it.

What we still do not have is a way to produce those numbers. Every offset in Demo01 and Demo02 is a literal that a human counted, and we have now spent an entire article on why that does not scale past one file.

Part 3 (“Your first PDF from MQL5: the byte-exact writer”) hands the byte counting to the computer. We build CPdfBuffer, the only class in the library that knows what a byte is, and CPdfWriter, which records where each object starts as it writes it and emits the xref table from what it recorded. The test is strict: the generated file must be byte-for-byte identical to the one we typed in Part 1, and Part 2's validator must report no problems on it. The take-away is a tool that turns any text — a log, a settings dump, a list of open positions — into a paginated PDF.

The project is supported on MQL5 Algo Forge. Each part of this series has its own release, frozen at exactly the code that part explains, so whichever article you are reading, its release is the one to download.

Part 2 is here: https://forge.mql5.io/SandroBegashvil/MQLPDF/releases/tag/part-02