Template-based extraction is genuinely better than a language model on the vendors it has
been configured for. It is faster, cheaper, deterministic, and it never invents anything.
If you process documents from eleven vendors and that list never changes, do not buy this
— buy a template engine and be happier.
The problem is the long tail. Real accounts-payable document mixes have hundreds of
issuers, a steady trickle of new ones, and layouts that change without notice. Each new
vendor is a configuration task, and — worse — a layout change on an existing vendor is a
silent failure: the template still matches something, it just matches the wrong region.
The failure mode of template OCR is confidently wrong data with no signal attached.
Language models generalise across layouts they have never seen, which is precisely the
capability the long tail needs. Their failure mode, though, is also confidently wrong data
— so the engineering problem is not “use a model”, it is “make the model’s uncertainty
legible and make its arithmetic checkable”.
That is what the rest of the pipeline is for, and it is where most of the code is.
Making hallucination risk structural rather than hopeful
Prompting a model to “be accurate” is not a control. Three mechanisms do the actual work:
Schema constraints. Extraction requests a strict output schema — typed fields,
required keys, enumerated currencies. Anything that does not conform is rejected and
retried rather than parsed leniently. This eliminates a whole class of downstream failure
where a malformed date or a stray currency symbol propagates into the ledger.
Grounding to source regions. Every extracted value must be locatable in the document’s
text-and-layout representation. A field the model returns that cannot be tied back to a
region on the page is not treated as an extraction — it is treated as a guess, scored low,
and routed to review. This is the single most useful signal in the system, because it
catches the specific failure where a model fills in a plausible value for a field that is
simply absent from the document.
Independent arithmetic validation. Sums, tax rates and date ordering are checked by
ordinary code that has no knowledge of what the model thought. A document where every
field scored high confidence but the line items do not sum to the stated total is more
suspicious than one with a couple of low-confidence fields — and the queue ranks it
accordingly.
Confidence, in other words, is not the model’s self-reported certainty. It is a composite
of how well the value is grounded in the page, whether the field is one this vendor
usually has, and whether the document’s internal arithmetic closes.
Vendor learning through retrieval, not fine-tuning
The obvious way to improve accuracy on a client’s specific vendors is to fine-tune. We
deliberately do not, for three reasons: it couples accuracy improvements to a training
cycle measured in days, it makes per-tenant data separation much harder to guarantee, and
it makes any given extraction impossible to explain after the fact.
Instead, every accepted extraction becomes a retrievable exemplar. A new document is
fingerprinted by layout and issuer, the nearest accepted exemplars from that tenant are
retrieved, and they are supplied as few-shot context. The effect on the second and
subsequent documents from a vendor is substantial, it takes effect immediately rather than
after a training run, and the tenant boundary is a WHERE clause rather than a promise
about a model’s weights.
It also degrades gracefully. A vendor seen for the first time gets zero-shot extraction —
which works, just with lower confidence and more review — rather than failing outright the
way an unconfigured template would.
PII, residency, and what we do not keep
Invoices carry bank details, addresses, and sometimes personal names. The handling rules
are boring on purpose:
- Originals are encrypted at rest and hash-linked to every record derived from them.
- Documents can be configured to purge after a retention window while the extracted data
and the audit trail remain, so a firm can meet a “delete the source after 90 days”
obligation without losing its books.
- The public demo on this site stores nothing at all. The uploaded file exists in memory
for the duration of the request and is never written to disk or to a log.
- Where a client has a residency obligation, the whole stack including the model endpoint
can be pinned to one jurisdiction — this constrains which models are available, and we
say so before it becomes a surprise.
The number that actually matters
Not accuracy. Straight-through processing rate — the proportion of documents that clear
every threshold and every validation rule without a human touching them.
Accuracy is easy to quote and hard to interpret, because it depends entirely on the
document mix it was measured against. Straight-through rate is the number that converts
directly into cost per document and headcount, and it is the number that goes up over the
first few weeks of a deployment as the vendor corpus fills in. It is the primary metric on
the analytics dashboard for that reason, and it is what we measure on a sample of your real
documents during discovery, before either side has committed to anything.