Letting Users Fix What OCR Missed — and What Their Corrections Taught Us

By the onecue.ai team · August 11, 2026 · ~11 min read

There is one OCR failure no metric can see. When the recognizer misreads a line, the mistake arrives with a confidence score attached and can be filtered. When the detector never boxes the text at all, nothing is produced: no crop, no score, no entry in any count. The text is simply absent from the output, and every accuracy figure the tool reports remains perfect.

We have written about why that distinction matters. This article is about what we built because of it: a way for users to point at text the model skipped, and what the first seventeen of those corrections changed in the pipeline. It includes the version that shipped broken, because the reason it broke is the most useful part.

The insight the feature rests on

The obvious approach to a missed line is to re-run OCR on the region. That is exactly wrong, and seeing why is what makes the feature work.

If the detector missed a character once, it will miss it again — more reliably, in fact, because a cropped region has less context than the full image did. The very reason a lone 一 gets skipped is that it looks like a decorative rule rather than text, and cropping tightly around it removes the surrounding characters that might have suggested otherwise.

So the rule became: the user's rectangle IS the detection. When someone marks a region, the crop goes to the recognizer directly, with detection skipped entirely. There is also no score threshold on that path. They pointed at it; an uncertain answer with its score attached beats a confident silence.

Pricing a correction at zero

Our free tier is 500 operations a month, and every OCR job counts against it. A correction obviously should not: the user is fixing our miss, and charging them for it reads as punitive in a way that would stop people using the feature — which would also stop the data arriving.

Free paths need guards, though, or they become an unlimited free tier with extra steps. Two caps do that job without ever being visible to a normal user:

Ten corrections per source job. A page has only so much missed text. This is enforced inside the same transaction that creates the job, reading and writing the parent document so concurrent calls serialize and cannot race past the limit.
500 corrections per month. The per-parent cap alone does not bound the total — 500 parent jobs × 10 is 5,000 extra worker jobs. The monthly counter closes that.

The server also verifies that the parent job belongs to the caller, and refuses to chain a correction onto another correction. Neither restriction is something a real user would ever encounter; both are things a script would find in minutes.

Four attempts at the interface

The backend was the easy half. The interface took four passes, and the failures are more instructive than the final design.

v1 — a button that opened a selection mode

"Add missed text" in the results header; click it, then drag a box. It worked, and nobody would have found it. The button was styled as a secondary action and sat among other buttons, so at the exact moment a user notices a missing word, nothing on screen suggests that anything can be done about it.

v2 — no button, just drag on the image

We removed the button entirely, put a prominent instruction above the image, and made the image itself draggable. On desktop this was a clear improvement: the crosshair cursor advertises the interaction, and dragging is immediate.

On mobile it did nothing at all, and the reason is a genuinely instructive CSS mistake. Touch gestures on a scrollable page are arbitrated by touch-action. We set pan-y, reasoning that vertical swipes should still scroll the page while horizontal drags — the shape of a text line — would reach us as a selection.

But a selection box needs height. Every drag has a vertical component, and a vertical component is precisely the signal the browser uses to claim the gesture as a scroll. It took the gesture and cancelled our pointer mid-drag, every time. The setting we chose to protect scrolling made the feature impossible on the devices most likely to need it.

Two lessons, one general and one specific. The specific one: touch-action is not a preference, it is a declaration of which gestures you are giving to the browser, and giving away vertical movement means giving away every rectangle. The general one: we verified v2 with synthetic pointer events at a desktop viewport and shipped it. A device profile with a coarse pointer would have caught it in a minute, and now does.

v3 — tap, and we find the word

The mobile fix could have been a tap-to-arm step before dragging, and briefly was. But the report that came back — "the box is hard to make" — pointed at something the arming step did not address: drawing a precise rectangle with a fingertip, over the glyphs your finger is covering, in one attempt, is simply a bad interaction.

So tapping became the whole gesture. A tap lands on or near the missed text, and the client finds the word's bounding box itself. That also removed the arming step, since a tap never conflicts with scrolling in the first place.

The detection runs on canvas pixels, and it reuses an idea already in our worker for recovering clipped line ends:

Take a window around the tap and split ink from background with a local Otsu threshold. Local, not global, because photographs have uneven lighting — a threshold computed across the whole page mislabels a shadowed corner.
Treat the minority class as ink. That handles dark text on paper and white lettering on a dark sign with the same code.
Seed from the nearest ink pixel to the tap, not the tapped pixel. A finger rarely lands exactly on a stroke.
Grow vertically while rows still contain ink — an empty row is the gap between lines, which stops the box from swallowing the line above or below.
Grow horizontally across gaps up to about one line height, stopping at anything wider. That is the difference between the space inside a word and the space between words.

It is deliberately not a model. It is a hundred lines of pixel arithmetic with unit tests covering word spans, taps in letter gaps, adjacent lines, distant words, light-on-dark text and blank areas. When the tap lands on nothing, it returns null and the interface says so, rather than sending an empty crop to the worker.

v4 — propose, don't submit

Automatic detection gets the word right most of the time. "Most" is a problem when a miss costs one of the ten corrections and forces a retry.

So the tap no longer submits anything. It proposes a box, with corner handles and a Read / Cancel bar. Drag the body to move it, a corner to resize, then confirm. A drag-drawn box gets the same treatment, so a slipped drag costs an adjustment rather than a read.

The mobile constraint reappears here in a smaller form, and this time the scope is right: touch-action: none is set on the box and its handles only, not the image. Adjusting never turns into a page scroll, and the rest of the image still scrolls normally. The handles render at 14 pixels with a 40-pixel hit area, because a fingertip needs the target even when the design does not.

What we store, and what we deliberately don't

Every correction is a labelled example of "here is text the detector missed" — the data no automatic measurement produces. It is also generated from images we promise not to retain.

The resolution is to store the geometry and nothing else: the region as fractions of the image (so it stays comparable across a phone photo and a scan), the image dimensions, the language, and afterwards the length and score of whatever was read. No image bytes. No recognized text. The dataset can tell us that regions of a certain size and shape fail in a certain language, which is what tuning needs, and it cannot reconstruct anything a user uploaded.

One design detail is worth calling out because the first draft got it wrong. Corrections were initially written by the client. A review caught that this would make it the only client-writable collection in the project, with rules that constrained field names but not their values — so a script could have inflated the dataset, backdated rows, or stuffed a megabyte of base64 into a field meant for a language code. The record is now written server-side inside the same transaction that validates the correction, and clients can neither write nor read the collection.

What the first seventeen corrections changed

Then the interesting part: reading the data back. Seventeen corrections, fourteen producing text. Sorting them by shape explained the split completely, and produced two code changes the same day.

All three empty results were regions that are not shaped like a line. One was squarer than a line of text, one was an 11:1 sentence, one was a vertical column. The reason is mechanical: the recognizer resizes any crop into a 32×320 box, so past roughly 10:1 the glyphs are squeezed into nothing, and a vertical column laid on its side has every upright character tipped over. Regions of those shapes now run detection first and come back as one line per row — the one case where re-running detection is correct, because the failure there is shape, not visibility. Small crops keep the direct read, since eleven of the seventeen were single glyphs around 60×80 pixels.

Scores ranged from 0.217 to 0.997, median 0.81 — a much wider spread than normal extraction produces, because this path has no threshold by design. That is correct behaviour and a display problem: a 0.22 guess and a 0.99 reading looked identical in the results list, with only a percentage to tell them apart, and a bare number does not read as a warning. Lines under 0.65 — the lower of our two engine thresholds, so anything a normal extraction would have dropped — now carry a red rule and read "unsure 54%" instead of "54%". The tool also says so at the moment the line lands, next to the image you would check it against.

There is a third finding we deliberately have not acted on. Tap detection almost always returns a single character: eleven of fourteen successful reads were one glyph. Widening the word-gap threshold would grab more per tap — but now that the proposed box is adjustable, an over-eager box is cheap to shrink and an under-eager one is cheap to grow. The honest answer is that we do not yet know which error costs more, so the parameter stays where it is until the share of confirmations that needed no adjustment says otherwise. That share is now measured.

Honest notes

Two things worth stating plainly, since a post like this drifts toward sounding like everything worked.

Seventeen corrections is not a dataset. They came from a single account testing the feature the day it shipped. The shape finding is mechanical enough to act on — the 32×320 constraint is a property of the model, not of the sample — but the frequency numbers mean nothing yet.

Re-running the three failures against the original images was humbling. Two were not model failures at all: one region contained a hat and hair with no text, and the other was clock-face digits the full-image OCR had already read correctly. The empty answer was the right answer. The fix stands on the mechanism rather than on those examples, and finding that out required going back to the actual pixels instead of trusting the summary table.

The general shape of this

The pattern generalizes past OCR. Any system with a silent failure mode — a search that returns nothing, a classifier that abstains, an importer that skips a row — has the same blind spot: the user can see the failure and the system cannot. A cheap way for them to point at it converts an invisible failure into a labelled example, and the interface for pointing has to be cheap enough that people actually use it. Ours took four attempts to get there, and the version that shipped broken on mobile taught us more than the two that worked.

You can try it on Image To Text: run any photo, and if a word is missing from the list, tap it on the result image.