How OCR Actually Works: Detection, Recognition, and Where It Fails

By the onecue.ai team · August 11, 2026 · ~9 min read

Most explanations of OCR treat it as one thing: an image goes in, text comes out. That framing is why OCR results are so often confusing in practice — a tool reports "98% confidence" while an entire line of the photo is missing from the output, and nothing in the interface explains the contradiction. The explanation is that OCR is not one model. It is two, running in sequence, and they fail in completely different ways.

This article walks through both stages as they run in onecue.ai's pipeline, with numbers measured on real user photos rather than benchmark datasets. Where a decision was made by measurement, the measurement is included.

Two models, two failure modes

The first model is the detector. It looks at the whole image and answers a single question: where is there text? Its output is a set of quadrilaterals — corner coordinates around each line it believes it found. It does not read anything. A detector can be perfectly confident that a region contains text while having no idea what the text says.

The second model is the recognizer. It receives one cropped region at a time, already isolated by the detector, and turns those pixels into characters. It never sees the full image. It cannot know that it was handed a picture of a leaf instead of a word, and it cannot recover a line the detector never handed over.

That division produces two distinct failures, and confusing them is the source of most disappointment with OCR tools:

A recognition failure is a misread. The text is found but decoded wrongly — クリスタル (crystal) comes back as a string that is nearly, but not quite, the right characters. These arrive with a confidence score attached, so they can be filtered.
A detection failure is silence. No box was drawn, so no crop was sent, so no score exists. The text simply is not in the output, and nothing marks its absence. This is the failure users notice as "it missed a whole line," and it is invisible to every confidence metric.

Keep that distinction in mind, because it explains the metric problem in the next section — and it is the reason our result screen has a feature that lets you point at text the model skipped.

Why "accuracy" is the wrong number to show

Suppose an image contains 38 lines of text. The detector finds all 38. The recognizer reads 19 of them above the confidence threshold; the other 19 score too low and get discarded. What accuracy should the tool report?

The tempting answer is the average confidence of the surviving lines. It is also the worst one, because it is anti-correlated with how well the tool actually did. Discarded text appears in neither the numerator nor the denominator, so the more text a tool throws away, the higher its reported accuracy climbs. A pipeline that keeps only its three most certain lines can advertise 99% while silently dropping most of the page.

We hit this directly. Our results carry a field called ocrStats with three counts recorded before any merging or filtering:

detected — boxes the detector produced, before the threshold
accepted — of those, the ones that survived it
rejected — the difference, which reaches you as unread text

accepted ÷ detected is coverage: of the text we found, how much did we actually read? It is the number worth watching, and it moves in the opposite direction from average confidence when a pipeline gets stricter. Raising a threshold always improves the confidence figure and always lowers coverage. Only one of the two tells you whether the output is useful.

Coverage still has a blind spot, and it is worth being explicit about it: it counts only what the detector found. Text that was never boxed is missing from the denominator too. There is no automatic measurement for "text the detector never saw" — the only reliable signal is a human looking at the image and noticing. That is exactly what the correction feature on our result screen collects, and the data from it is what the last section of this article is built on.

Resolution: the setting that mattered most

Before detection runs, the image is scaled onto a fixed square canvas. This is normal — detectors are trained at a particular input size — but the size you choose turns out to dominate everything else about the pipeline's behavior on real photos.

Our detection canvas was originally 800×800, inherited from a pipeline built for scraped product images: clean, cropped, screen-resolution graphics where 800px is plenty. Our actual input is a phone photo of a menu, a shop sign, or a printed card. An iPhone photo is 3024×4032. Fitted into an 800px square it becomes roughly 585×780 — about 19% of the original. Body text that was 24 pixels tall in the original arrives at the detector as 4.6 pixels tall.

Nothing downstream can recover from that. The glyphs are gone before either model sees them. This single setting was the largest cause of whole passages coming back unread from photos, and no amount of threshold tuning would have touched it — the low scores were a symptom, not the disease. Raising the canvas to 1600 doubles the linear resolution and quadruples the pixel area available for every glyph.

There is a second, subtler detail in the same code path. The original resize had a branch that squashed nearly-square images into a perfect square, ignoring aspect ratio — and it omitted the resampling argument, so those images were downscaled with nearest-neighbour instead of Lanczos. Both are the kind of thing that never shows up in a demo on clean inputs and quietly costs accuracy on real ones. Aspect ratio is now always preserved, and the resample filter is always specified.

What thresholds actually trade away

The recognizer returns a confidence score with every line, and a threshold decides what survives. It sounds like a quality knob. It is really a knob that chooses which kind of wrong answer you prefer.

Set it high and you get silence: correct readings get discarded because the model was not certain enough, and the user sees a photo with text missing. Set it low and you get fabrication: unreadable input becomes plausible-looking output. On a vertical-writing Japanese card in our test set, a low floor let クリスタル through as a nonsense romanization and 恋愛の石 as an unrelated pair of Korean words. To the user, that is worse than nothing — untranslated text is obviously untranslated, but a confident mistranslation reads as a fact.

Our thresholds are 0.7 for Chinese and 0.65 for Japanese. The Japanese figure used to be 0.80 — stricter than Chinese — which is backwards for our inputs. That table came from a pipeline tuned on clean scraped images; photographed signage scores structurally lower than a screenshot does, so the stricter bar was discarding a great deal of correctly-read Japanese. It was lowered together with the resolution fix, not before it: on its own, a lower threshold would only have admitted more noise, because the low scores were coming from 4-pixel glyphs.

That ordering is the general lesson. A threshold change is only meaningful once the input to the model is as good as you can make it. Tuning the filter before fixing the signal just moves the failure around.

What we learned from users pointing at missed text

Because detection failures are invisible to metrics, our result screen lets you tap text the model skipped; the region is read again and the line is added. Every one of those taps is a labelled example of "here is text the detector missed," which is the data no automatic measurement produces.

The first 17 corrections tell a clear story. Fourteen produced text (82%); three came back empty. Sorting them by shape explains the split completely:

Eleven were single glyphs, roughly 60×80 pixels — one character the detector had dismissed. These are the classic detection failure: a simple, low-stroke character like 一 reads as a decorative rule rather than text.
All three empty results were regions that aren't shaped like a line. One was squarer than a line of text, one was an 11:1 sentence, one was a vertical column. The recognizer resizes any crop into a 32×320 box, so past roughly 10:1 the glyphs are squeezed into nothing; and a vertical column laid on its side has every upright character tipped over. Regions of those shapes now run detection first, and come back as one line per row.
Scores ranged from 0.22 to 0.997, median 0.81 — a far wider spread than normal extraction produces, because a user-picked region is read without a threshold. That is deliberate: you pointed at it, so a low-confidence answer beats none. But it means the result list now marks anything under 0.65 as unsure, since a 0.22 guess and a 0.99 reading should never look alike.

None of these conclusions came from a benchmark. They came from counting what happened on real images, which is the only method that surfaces the failures a benchmark has already excluded.

The models, and why these ones

For Chinese and Japanese we run PP-OCRv5 for detection and, for Chinese, PP-OCRv4 for recognition — the newer v5 recognizer measured worse than v4 on our inputs, so the older one stayed. Japanese uses v5. Everything else routes to a separate multilingual PP-OCRv6 pair covering roughly 47 Latin-script languages, kept in completely separate code so that changes there cannot regress Chinese and Japanese quality.

All of these are open pretrained weights running as ONNX models on ordinary hardware. There is no cloud OCR API behind the tool. That is a deliberate trade: it means your images are processed on machines we control rather than being forwarded to a third party, and it means the failure modes in this article are ones we can actually fix rather than file a support ticket about.

Reading the output on your own images

If you want to see the distinction between the two failure modes on your own photo, the fastest route is the coverage figure on the result screen:

Coverage is high but text is missing → detection failure. The missing text was never boxed. Tap it on the image and it will be read.
Coverage is low → recognition failure. Lines were found but scored too low. Usually resolution or focus: reshoot closer, with the text filling more of the frame.
Text appears but is wrong → check the percentage next to it. Anything marked unsure was read without a threshold because you asked for that region specifically.

You can try both paths on Image To Text, which returns every line with its position, or Image Translator, which runs the same pipeline and then puts the translation back where the original text was.