Investor's Last Exam: Why AI Forecast Accuracy Collapses to 52.5% Without Hindsight—and Why Brier Calibration Matters
What happens when you lock frontier AI agents inside a "frozen" market database where they cannot peek at future outcomes?
A new benchmark study released at the end of October 2026 (Investor's Last Exam: Can AI Agents Do Equity Investment Without Seeing the Future?, arXiv: 2610.38342) tested leading AI models across 400 temporal cutoff scenes inside frozen point-in-time market databases.
📉 The "Investor's Last Exam" Benchmark Findings
- Analytical Fluency vs. Forward Reality: Across 12 frontier models, AI agents achieved a solid 68.1 score on historical data retrieval, financial logic, and context summarization.
- The 52.5 Collapse: When forced to make forward investment decisions without future memory leakage, actual forecast accuracy dropped to 52.5—statistically barely above a coin toss.
- Where the Chain Breaks: Models consistently failed at translating past evidence into calibrated forward probabilities when market regimes shifted.
Summarizing the Past Is Not Calibrating the Future
Why do AI models—and over-optimized indicators—look so convincing on historical data, only to fall apart at the right edge of the chart?
Because describing what already happened and calibrating the probability of what happens on the next bar are two completely different mathematical tasks. Most indicators give you a binary label ("bullish" or "bearish") without ever stating the empirical base rate of that regime surviving, let alone tracking whether its probability estimates are actually calibrated over time.
Why Every Probabilistic Tool Needs a Brier Score
In quantitative finance and meteorology, you do not evaluate a probabilistic model by asking whether it "felt confident." You evaluate it using a Brier Forecast Score—measuring the mean squared error between predicted probabilities and actual closed-bar outcomes:
Brier Score = (1 / N) * Σ (f_t - o_t)²
With modern MetaTrader 5 updates (including native vector/matrix methods and fast array operations), we can compute empirical state transition matrices and rolling Brier calibration scores directly on closed bars in real time—without relying on external black-box opinions.
Every tool I publish follows a falsification-first discipline: closed-bar values are final, nothing repaints, and every measurement states its own statistical limits:
📐 TREND PERSISTENCE — How often the current state has survived to the next bar
- ▸ Trend Survival States (MT5) · 🆓 · Classifies the market into five objective structural states on closed bars
- ▸ Trend Survival Rate (MT5) · $99 · Adds the empirical bar-to-bar survival rate and a live Brier forecast score to verify calibration
🔍 REGIME AND STRUCTURE — Documenting agreement and divergence
- ▸ Oscillator Regime Suite (MT5) · $89 · MT4 Version ($89)
- ▸ Structure Flow and Trailing Stop (MT5) · $99 · MT4 Version ($99)
🔬 MEASUREMENT — Point it at any indicator and read what it actually does
👉 Explore all 18 measurement tools on my MQL5 profile: mql5.com/en/users/naamo (@naamo)
💬 Do you track calibration metrics like the Brier score on your regime filters, or do you treat trend indicators as binary on/off switches?
📚 References & Further Reading:
- Investor's Last Exam: Can AI Agents Do Equity Investment Without Seeing the Future? (arXiv: 2610.38342, Oct 2026): https://arxiv.org/abs/2610.38342
