In an audit of 91 LLM trading papers, 51 % of the 75 with a trading test deduct no trading costs

20 November 2026, 09:00
Jevon Naldo Yoshield
0
5

arXiv 2611.18934 is Henry Zhang's audit of 91 LLM trading papers. Of the 75 with a trading test, 51 % deduct no trading costs and 9 % model costs from data. The median test window is 12 months, and a quarter are six months or shorter.

The audit was pre-registered with a ten-field rubric, and a blind second coder recoded a random 20 %. Two of its five hypotheses pass, short windows and uncontrolled look-ahead.

Of 51 papers whose test window is not known to lie after the training cutoff, 98 % neither test after the cutoff, evaluate live, nor use a chronologically consistent model. Separately, only 9 % of the papers report a multiplicity-adjusted result.

Only five papers in the audit combine deducted costs, a leakage control and a reported trial count. Before I trust a backtest or paper that claims an LLM edge, I ask whether costs were deducted, whether the test window lies after the training cutoff, and whether a trial count is reported.

I read the abstract, not the paper. It is a preprint by one author on a curated corpus of LLM research, and it does not measure Expert Advisors.