How We Measure Receipt OCR Accuracy: Our Benchmark Methodology

2026-10-05 Sheece Gardezi benchmarksocraccuracy

If a receipt total comes back wrong, the error doesn't stay in the app. It lands in your books, and you find it at reconciliation time, after you've already trusted it. That's why "accuracy" claims matter, and why so few of them hold up. Almost no receipt capture tool publishes how it measured: which dataset, which metric, which pipeline version, on which date. This post is our answer to that gap: the exact datasets we benchmark against, the exact metrics we compute, the real numbers we get (including the ones we're not proud of yet), and a methodology you can check for yourself. If you take one thing away, take this: we would rather show you a 70% with a methodology attached than a 99% with none.

Why methodology matters more than the headline number

"99% accurate" is the most common sentence in receipt-OCR marketing, and it is close to meaningless. Accuracy against what? A vendor demo set of clean, flat, perfectly lit scans is a different test from a crumpled thermal receipt photographed in a car park at night. Accuracy measured how? Reading most of the characters right is not the same as reading the total right, and reading the total right is not the same as the line items actually adding up to it.

When we built SnapReceipt we needed to know, for our own engineering decisions rather than for marketing, whether the product was genuinely good enough to trust with someone's books. So we built the measurement first, and only afterwards put numbers on the landing page. Every quantitative claim in our proof section traces back to a dated measurement run and the methodology described here. This post is the full version of that footnote.

The datasets

We benchmark on two public datasets, plus an internal corpus. Public matters: anyone can download the same scans and audit what we did.

CORD v2: the harder test set

CORD (A Consolidated Receipt Dataset for Post-OCR Parsing) is a public dataset of receipt photos with structured annotations: fields like subtotal, tax, and total, plus line items, labelled by humans. It is deliberately not easy: the receipts are photographed in the wild, with skew, glare, folds, and faded thermal print. We use the CORD v2 test split as our harder benchmark. This is the set where honest numbers get uncomfortable, and we think that is exactly why it is the more important one to publish.

Zenodo-SRD: 193 labeled real-world scans

Our "everyday receipts" benchmark is a set of 193 labeled real-world receipt scans from the public Zenodo-SRD dataset, kept in our local corpus with human-written transcripts as ground truth. These look like what a bookkeeper actually receives: supermarket dockets, fuel receipts, café receipts, supplier invoices. The scans average 856 pixels on the long edge, which is modest resolution, close to what a phone upload looks like after client-side compression.

The local corpus: 242 scans

On top of the labeled sets we keep a 242-scan local corpus that also includes production upload photos (shared with consent, stored on our own systems). Production photos have no ground-truth transcript, so they can't be scored for transcription accuracy. But they can be scored by any metric that needs no labels, like the arithmetic self-checks below. That lets us measure real-world behaviour on the receipts that matter most: the ones users actually photograph.

The metrics, defined precisely

We report three families of metrics because no single number captures "is this extraction good enough to bookkeep from." Here is exactly what each one means.

Field exact-match rate. For each target field (subtotal, total, tax), the fraction of receipts where the extracted value equals the human-labeled value exactly, as a number. Not "close," not "right after manual cleanup": exactly right. This is the metric a bookkeeper cares about, because a total that is one character off is a wrong total.
Character accuracy / character error rate (CER). The model transcribes the entire receipt; we compare that transcript to the human ground-truth transcript using Levenshtein edit distance over characters, computed as whitespace-normalized CER (runs of whitespace collapse to one space on both sides, so line-wrapping differences don't count as errors). Character accuracy is 1 − CER. One honest caveat baked into the measurement: thermal receipts print separator rules (====, ----) that the corpus ground truth omits, so we strip separator-only lines from both sides. Otherwise we'd be penalizing the model for correctly reading characters that are really printed on the receipt.
Self-check pass rate. Unique to SnapReceipt: after extraction, we verify the arithmetic. Do the extracted line items sum to the extracted subtotal (within a ±$0.10 tolerance)? Does subtotal plus tax equal the total? The pass rate is the fraction of receipts where the extracted numbers are internally consistent. Crucially, this metric needs no ground truth at all. It checks the extraction against itself, which is why it also runs on unlabeled production photos, and why the same check runs on every extraction in the live product.

Why all three? Because they fail in different ways. A model can have beautiful character accuracy and still swap the subtotal and total. It can get every field exactly right on easy scans and fall apart on hard ones; you only see that if you separate the datasets. And it can produce fields that look plausible but are arithmetically impossible, which only the self-check catches. A single headline number would hide whichever of these was going wrong.

Our results, honestly framed

Here are the numbers as measured, with their provenance. Everything below was produced by our in-house extraction pipeline, the same proprietary, purpose-built stack that processes production receipts, not a demo build. The same figures appear on the landing page proof section; this post is where they are explained.

Whole-receipt transcription

Across the 193 Zenodo-SRD scans, mean character accuracy of our transcription stage is 94.2%, versus 61.1% for Tesseract run on the very same scans (whitespace-normalized CER, separator lines stripped from both sides; re-measured 2026-10-03 on our production pipeline at the time). An earlier measurement on a previous version of the pipeline read CER 0.105, so 89.5%, which tells you how much the pipeline version matters.

What this proves: on everyday real-world scans, our transcription is dramatically more faithful than a classical OCR engine. What it does not prove: that 94.2% of fields are right (character-level fidelity and field-level correctness are different things), or anything at all about receipts unlike the ones in the dataset.

Field exact-match

On the everyday Zenodo-SRD set, the subtotal is read exactly right on 95% of receipts and the total on 92%. On the harder CORD v2 test set, exact-match drops to 70-75% across subtotal, total, and tax. The legacy baseline (Tesseract plus rule-based parsing, graded by the same scoring methodology on the same receipts) scores 30-33% on the everyday set and 5-22% on the harder set.

We publish the harder-set number prominently, not buried, for two reasons. First, it's the number most likely to match your experience if your receipts are crumpled, skewed, or faded. Second, a benchmark you only report when it flatters you isn't a benchmark.

The arithmetic self-check

Across the 242-scan local corpus (labeled scans plus production photos), the extracted line items sum to the extracted subtotal on 86.4% of receipts: 209 out of 242, measured with our published scoring tools, no ground truth required.

The product behaviour matters more than the number here: the 13.6% that fail the check are flagged for review, never silently passed through. The self-check turns an unknown error rate into a known one. You always find out when the math doesn't hold, because the system tells you. We wrote about how this works in detail in Self-checking math on every extraction.

What these numbers do not prove

Being precise about the limits is part of the methodology, so:

The numbers describe the datasets measured. Your receipts may be easier or harder; a benchmark on public scans is evidence, not a guarantee.
Character accuracy is not field accuracy, and field accuracy on 193 everyday scans is not field accuracy on your worst crumpled docket. That is why we report the harder set separately.
The self-check pass rate measures internal consistency, not correctness against the paper. An extraction can in principle be consistently wrong (though the check catches the far more common case of fields that contradict each other).
Results are pipeline-version-dependent. The 94.2% was measured on our production pipeline on 2026-10-03; earlier and later versions will score differently. We date every measurement for this reason.

How this differs from OCR-engine benchmarks

Most published "receipt OCR" numbers benchmark a raw OCR engine: character or word error rates on document images, or layout-detection scores. That is a real and useful layer of measurement (ImageToTable.ai's public receipt OCR benchmark is a good example of engine-layer comparison done openly), but it answers a different question from the one a bookkeeper asks.

SnapReceipt is not an OCR engine; it is an end-to-end capture product. So we benchmark at the product layer: did the total field come out exactly right, do the line items sum to the subtotal, does subtotal plus tax equal the total? The product also includes layers no engine benchmark touches: arithmetic validation on every result, and ABN verification against the Australian Business Register for Australian business receipts. Character error rate is an ingredient in those outcomes, not the outcome itself. If you are choosing a tool to do bookkeeping with, field exact-match and self-check pass rate are the numbers that map to your day; engine-layer CER mostly tells you why a system fails when it fails.

Reproducibility: check our work

The discipline is the same one we hold ourselves to: our scoring tools are published alongside the methodology, the baseline is graded by the same tools on the same receipts, and the quality measurements are deliberately dated. They run against our live in-house pipeline, on demand, rather than as part of an automated test gate, so every published figure carries the date it was measured.

Our published scoring tools grade our proprietary pipeline with the exact rigor they apply to the baseline. The transcription-accuracy comparison against Tesseract scores both systems with the identical whitespace-normalized CER metric on the identical 193 scans: apples to apples, not a literature cherry-pick.

If you get a different number, we want to hear about it. That is the entire point of publishing the method.

What we're still working to improve

The honest gap is the harder test set: 70-75% field exact-match on CORD v2 is far better than the classical baseline's 5-22%, and it is also not good enough to stop working. Hard scans fail in predictable ways: severe skew, motion blur, thermal fade where the ink has physically vanished. Each failure mode has its own remediation, from better client-side capture guidance to extraction improvements to leaning harder on the self-check to route borderline results to human review.

We would rather you choose SnapReceipt knowing exactly where it stands than discover the gap on your own receipts. When these numbers move, the landing page moves with them. The proof section and this methodology are maintained together.

FAQ

Why don't you just say "99% accurate"?

Because we can't define it honestly. Accuracy against which dataset, measured how, on which pipeline version? On our everyday benchmark the subtotal is exactly right 95% of the time; on the harder public test set it's 75%. Reporting one flattering number would mean picking the easiest dataset and the loosest metric, which is precisely the marketing habit this post exists to avoid.

Can I verify your numbers myself?

Yes. The datasets are public (CORD and the Zenodo-SRD scans), and our scoring tools are published alongside the methodology. They run against the live in-house pipeline on demand, which is why measurements are dated, on-demand runs rather than part of an automated test gate, but the methodology above is exactly what produced the published figures.

What does the 86.4% self-check number actually mean for me?

It means that on our 242-scan corpus, the extracted line items summed to the extracted subtotal 209 times out of 242, and, more importantly, that the other 33 were flagged for review instead of passed through. The check runs on every extraction in the live product, not just in the benchmark, so its benefit doesn't depend on your receipts resembling the corpus.

Do the numbers apply to my receipts?

They're evidence, not a promise. If your receipts look like typical everyday scans, the everyday numbers are the better guide; if they're crumpled, faded, or photographed at odd angles, expect something closer to the harder test-set figures. Either way, the arithmetic self-check means you'll be told when a result doesn't hold together. The failure mode is visible, not silent.

Why do your numbers differ from OCR-engine benchmarks I've seen elsewhere?

Different layer. Engine benchmarks measure raw character or word error on document images; we measure end-to-end capture outcomes (field exact-match and arithmetic consistency) because that's what a bookkeeper consumes. Our transcription CER lets us compare against engines directly where the layers overlap, and on the same 193 scans that's 94.2% character accuracy versus 61.1% for Tesseract.

If you want the short version, the proof section on the landing page carries the same numbers with their one-line provenance. And if you'd rather generate your own evidence than read ours, the fairest test is your own shoebox: try SnapReceipt on your real receipts at app.snapreceipt.dev and watch what the self-check catches.