Artificial intelligence can turn an invoice, filing, or scanned form into structured data in seconds. The harder question is whether a business can trust the result when a document runs for hundreds of pages, repeats thousands of records, mixes handwriting with tables, or requires every extracted value to point back to its source.
A new technical report called ExtractBench argues that ordinary accuracy tests do not answer that question. Submitted to arXiv on July 31 by researchers at LlamaIndex, the benchmark evaluates 14 extraction systems on 370 documents containing 4,869 pages. Its corpus spans eight business domains and 67 document types, including finance, energy, government, supply chain, healthcare, legal, automotive, and real estate material.
One score can hide several failures
Schema-guided extraction asks a system to read a document and fill a user-defined structure, usually JSON. A plausible value is not enough. The system must put it in the correct field, return null when information is absent, preserve every row in repeated records, and ideally provide evidence showing where the answer came from.
ExtractBench separates those requirements. It measures order-insensitive value accuracy, record completeness, page- and word-level grounding, and the observed cost per page. Documents are also tagged for specific challenges such as long lists, facts buried in lengthy files, dense forms, scans, handwriting, cross-page tables, and tables with more than a thousand rows.
The practical lesson is that extraction reliability is a four-part constraint: value accuracy, record completeness, source traceability, and operating cost must all clear a workflow’s threshold at the same time. A system that gets most individual fields right but silently drops half a schedule is not reliable. Neither is a system that returns correct values without enough evidence for a reviewer to audit them.
Long documents change the ranking
The report’s clearest result is how sharply some systems degrade as documents grow. Gemini 3.5 Flash scored 87.9 percent value F1 on short documents but 27.9 percent on files longer than 50 pages. LlamaExtract Agentic Plus moved from 96.6 percent on short documents to 94.4 percent on long ones. The paper attributes much of the long-file weakness to recall: systems may correctly read the rows they return while stopping before the full list is complete.
Cost complicates the picture. The researchers report overall value F1 of 95.6 percent for LlamaExtract Agentic Plus at an average 8.1 cents per page. Codex GPT-5.5 reached 93.6 percent at 27.8 cents per page. Those figures are benchmark results under the authors’ configurations, not universal prices or guarantees, but they illustrate why model quality and workflow economics cannot be separated.
Grounding remains a substantial weakness even at the top of the table. The best reported word-level grounding F1 was 46.4 percent, compared with 84.9 percent at page level. Commercial vision-language models and coding agents in the evaluation did not return source evidence by default, so they received no grounding credit. For high-stakes back-office work, locating the correct page is helpful, but locating the exact value is what makes fast human review possible.
What a second benchmark adds
The CVPR 2026 Workshops paper VAREX provides a useful comparison. VAREX uses 1,798 programmatically filled government forms, each paired with a varied schema and shown through controlled text, layout-preserving text, image, and combined inputs. Its design is well suited to isolating whether models follow schemas and how different input representations change performance.
ExtractBench focuses on another layer of the problem: varied enterprise files, long repeated records, scan and handwriting noise, grounding, and measured cost. It combines real documents, synthetic long lists built from real layouts, and human-verified forms. That broader realism comes with a tradeoff: the new report is a vendor-authored preprint, its authors’ product leads the reported results, and the findings have not yet been independently replicated.
Read together, VAREX and ExtractBench reveal a useful division of labor in evaluation: controlled synthetic forms isolate whether a model can follow varied schemas, while mixed real and synthetic enterprise files expose whether a system can keep going through noise, length, and repeated records. Neither design alone represents every production environment.
A better procurement test
Organizations evaluating document AI should begin with their own failure costs. A missing invoice line, an invented identifier, a value copied from the wrong section, and an untraceable answer are different errors. Each deserves its own acceptance threshold and review process.
That means buyers should ask for a failure map, not a single leaderboard number: scores split by document length, table structure, scan quality, missing-value handling, evidence quality, and cost at the expected production volume. They should also test whether a system signals uncertainty or truncation instead of returning polished but incomplete output.
ExtractBench does not settle which product is best for every company. It offers something more durable: a stronger definition of what “working” should mean. The future of document automation will depend less on extracting a few correct fields in a demo and more on proving that every required record is complete, auditable, and economical at scale.
Sources: ExtractBench technical report by Boyang Zhang, Adrian Lyjak, Eli Stewart, Zhaoqi Li, and Simon Suo; VAREX paper by Udi Barzelay and colleagues in the CVPR 2026 Workshops proceedings.
Photo: Jacob W. Frank, U.S. National Park Service via Wikimedia Commons. Public domain (U.S. National Park Service). Center-cropped from 3:2 to 16:9 and resized.
