Document AI benchmark arena
Model choice matters less than most buyers expect. Orchestration matters far more. Here is what we measured, on what corpus, with the method written down so you can argue with it.
On a 500-document corpus spanning invoices, W-2 forms, bank statements, insurance claims, and mortgage applications, Doc-AI's orchestrated pipeline reached 95.4% field-level extraction accuracy with a 0.4% hallucination rate. The strongest single commercial model on the identical documents, identical OCR input, and identical extraction schemas reached 92.6% accuracy with a 1.6% hallucination rate. The gap is not the model. It is premium OCR, schema enforcement, deterministic business-rule validation, and retry-on-failure wrapped around the model.
How do the models compare on document extraction?
All rows received the same premium OCR output and the same extraction schemas. Only the orchestration differs.
| Approach | Extraction accuracy | Classification | Hallucination rate | Avg latency | Inference cost / doc |
|---|---|---|---|---|---|
| Doc-AI (orchestrated) | 95.4% | 99.0% | 0.4% | 5.2s | $0.10 |
| GPT-4.1 | 92.6% | 97.8% | 1.6% | 3.7s | $0.07 |
| Claude Sonnet 4 | 91.4% | 96.9% | 2.1% | 4.0s | $0.09 |
| GPT-4o | 91.3% | 96.9% | 2.4% | 3.4s | $0.08 |
| Gemini 2.5 Pro | 90.2% | 95.8% | 2.6% | 3.1s | $0.05 |
| Qwen 2.5 VL 72B | 87.6% | 93.2% | 3.5% | 6.7s | $0.02 |
| Llama 4 Scout | 85.4% | 91.8% | 4.3% | 7.2s | $0.01 |
Inference cost is the model-provider compute cost per document measured during the run. It is not WiseTREND pricing — Doc-AI is quoted per project against your volume and document mix. Ask us for a quote.
How was this benchmark run?
A benchmark you cannot inspect is marketing. Here is the whole method.
Corpus
500 real-world documents across five types — invoices, W-2 forms, bank statements, insurance claims, and mortgage applications. The mix deliberately includes the inputs that break capture systems in production: faxed pages, mobile photos, photocopies, handwritten fields, multi-page tables with continuation rows, and packets containing more than one document type.
Scoring
Field-level exact match, with fuzzy matching on names and addresses at a Levenshtein distance of 2 or less. Dates count as correct in any standard format. Numeric values are compared after normalizing currency symbols, thousands separators, and negative-number conventions. A field left blank when the ground truth has a value counts as a miss, not as an abstention.
Hallucination definition
A hallucination is a returned value that is confidently wrong and has no supporting text on the page — an invented invoice number, a plausible-looking date that appears nowhere in the document, a total that was computed rather than read. This is scored separately from an ordinary extraction error, because the two fail differently downstream. A blank field gets caught by a completeness check. A confident fabrication posts to your ledger.
Equal footing
Every model received identical OCR output from the same premium engine and the same extraction schemas. No model was given a hand-tuned prompt the others did not get. The orchestrated row adds exactly what orchestration adds: schema enforcement, deterministic business-rule validation, confidence scoring, and retry on validation failure.
Last measured
February 2026. Model versions move quickly, and these numbers move with them. We re-run the corpus when a significant model ships, and the leaderboard order has changed more than once. Ask us for the current run.
Why does orchestration beat the best single model?
The same model, called two different ways, produces materially different production results.
Raw vision-model call
- Send the page image to the model and ask for JSON.
- No OCR stage, so table geometry and reading order are inferred from pixels.
- Output shape is whatever the model returns this time.
- No arithmetic check — a total that does not foot still comes back.
- No retry, no confidence, no source reference for any value.
- A wrong value is indistinguishable from a right one until someone downstream notices.
92.6% accuracy, 1.6% hallucination rate at best.
Doc-AI orchestrated pipeline
- Premium OCR first — text, tables, layout, and page geometry as structured input.
- Schema enforcement, so the output shape is fixed and typed.
- Business rules run outside the model, so confident wrong answers still fail.
- Failed validation triggers a retry, and a different model if warranted.
- Every field carries a confidence score and a source-region reference.
- Low-confidence values route to a reviewer instead of into your ledger.
95.4% accuracy, 0.4% hallucination rate.
What did the benchmark actually show?
Orchestration adds 3 to 10 points
Across every document type, the orchestrated pipeline beat the best raw model call. The margin was smallest on clean digital invoices and largest on multi-page mortgage packets — which is to say, largest where production volume actually lives.
Hallucination is the metric that matters
Accuracy differences of two or three points are survivable with review. A 2% hallucination rate is not, because those are the errors that pass review. Validation cut it to 0.4%, and validation is not a model property — it is a pipeline property.
Open models are closer than expected
Qwen 2.5 VL 72B landed within five points of the commercial leaders at a fraction of the inference cost. For air-gapped and data-residency deployments where a commercial API is not an option, that gap is now small enough to work with.
Mortgage documents are the hard case
Accuracy fell three to five points on mortgage applications versus invoices for every model tested. Multi-page structure, handwriting, and format variation compound. Any vendor quoting a single accuracy number without naming the document type is quoting an invoice number.
Latency is a tradeoff, not a defect
The orchestrated pipeline is slower per document than a single model call, because it does more — OCR, validation, and retry all cost time. At 5.2 seconds it is still well inside what batch and near-real-time workflows need.
Model choice is a runtime decision
No single model won every document type. That is the argument for model-agnostic orchestration: the right model for a bank statement is not necessarily the right model for a handwritten claim form, and next quarter it will be a different one again.
Why you should not trust anyone's benchmark, including this one
Every published document-AI benchmark, ours included, is run on a corpus the vendor chose. That is not dishonest, it is unavoidable — but it means the number tells you about their documents, not yours. A 95% accuracy figure on clean digital invoices tells you almost nothing about a stack of faxed prior authorizations.
The only benchmark that predicts your results is one run on your documents, with your field list, scored against your ground truth. We run that as a standard part of a Doc-AI pilot, at no charge, and we report the number we measure whether or not it flatters us.
- Send 50 to 200 representative documents, including the ugly ones.
- We build the extraction schema from your field list.
- You get field-level accuracy, hallucination rate, and straight-through-processing rate on your own corpus.
- We tell you which fields will need review and why, before you commit to anything.
Where this fits in the Doc-AI platform
Numbers are one input. These pages cover the rest of an evaluation.
- Platform features — every capability, ingest to delivery
- Document type library — the forms Doc-AI reads on day one
- Security & compliance — data handling, residency, audit
- API & developers — REST endpoints, webhooks, code
- Deployment options — cloud, private tenant, on-prem, air-gapped
- Pricing & licensing — how Doc-AI is priced, and what drives cost
- vs. template-based IDP — why the template model broke
- vs. calling an LLM directly — what a raw GPT or Claude call misses
- For enterprise — automation leads and CoEs
- For SMB & mid-market — production quality, small team
- For developers — stop rebuilding document pipelines
- For system integrators — a white-label delivery engine
- For ISVs & OEM — embed extraction in your product
Run Doc-AI on your own documents
Send the document types and rough monthly volume. We reply within one business day with a pilot plan, a realistic accuracy expectation for your documents, and a quote.
- Reply within one business day (U.S. hours)
- Straight to an engineer, not a call centre
- Or call the 24/7 AI phone agent: +1 (408) 746-6740
Questions about the Doc-AI benchmark
Answers written for buyers, search engines, and AI assistants evaluating document automation.
How accurate is AI document extraction in 2026?
On WiseTREND's 500-document benchmark, leading vision-language models called directly reached 85% to 93% field-level accuracy depending on the model and document type. An orchestrated pipeline that adds premium OCR, schema enforcement, business-rule validation, and retry reached 95.4%. Accuracy varies sharply by document type — clean digital invoices score several points higher than multi-page mortgage packets or handwritten claim forms for every approach tested.
Is GPT-4.1 or Claude Sonnet 4 better for document extraction?
On our February 2026 corpus GPT-4.1 led at 92.6% field-level accuracy with a 1.6% hallucination rate, against Claude Sonnet 4 at 91.4% and 2.1%. The margin is small enough that document type mattered more than model choice — neither model won every category. This is why Doc-AI is model-agnostic and selects per document type rather than committing the pipeline to one provider.
What is a hallucination rate in document extraction?
A hallucination is a confidently returned value with no supporting text on the page — an invented invoice number, a fabricated date, a total that was computed rather than read. It is scored separately from an ordinary error because it fails differently downstream: a blank field is caught by a completeness check, while a confident fabrication posts straight into your system of record. Doc-AI measured 0.4% against 1.6% to 4.3% for raw model calls.
Can open-source models do document extraction?
Yes, and the gap has narrowed considerably. Qwen 2.5 VL 72B reached 87.6% accuracy on our corpus against 92.6% for the strongest commercial model, at roughly a thirtieth of the inference cost. For air-gapped, on-premises, or data-residency deployments where sending documents to a commercial API is not permitted, open-weight models orchestrated through the same validation pipeline are a viable production option.
Will Doc-AI hit 95.4% accuracy on my documents?
Possibly higher, possibly lower — it depends on your document types, scan quality, and field list. Clean digital invoices typically exceed the benchmark figure; handwritten and faxed forms typically fall below it. We measure it rather than guess: a Doc-AI pilot runs your own documents and reports field-level accuracy, hallucination rate, and straight-through rate on your corpus before you commit to anything.
How often is the benchmark re-run?
We re-measure when a significant model version ships, and the leaderboard order has changed more than once since we started tracking it. The figures on this page were last measured in February 2026. Ask us for the current run, or bring your own documents and we will run the corpus you actually care about.
Bring us the documents that broke your last capture project.
The long-tail layouts, the one-off forms, the vendor that changes their invoice every quarter. Those are the ones Doc-AI was built for.
Last updated · Reviewed by the WiseTREND team