Translating Text Inside an Image Without Wrecking the Layout

By the onecue.ai team · August 11, 2026 · ~9 min read

Translating the text in a photograph sounds like three steps: read it, translate it, write it back. The first two are largely solved by models you can call. The third is where the work is, and it is the reason most image translation output looks obviously edited — grey rectangles over the original words, text spilling past its box, characters replaced by empty squares.

This article covers what happens after the translation comes back, using decisions from onecue.ai's renderer. Every problem below was found on real photographed menus, signage, and product cards rather than in a specification.

The first problem: the original text has to disappear

You cannot draw a translation on top of the source text; you have to remove it first, and remove it in a way that leaves a plausible background behind. The general tool for this is generative inpainting — a model that fills a masked region with content consistent with its surroundings. That is the right approach for text on a photograph, a texture, or a gradient.

It is the wrong approach for the most common case in signage. Inpainting repaints a small masked region with whatever surrounds it, and a label is usually a flat block of colour inside a differently-coloured panel. Feed it a red band containing white lettering inside a white panel and it does exactly what it was designed to do: fills the band with the white of its surroundings. The band disappears, and then white translated text gets drawn onto white background. The output is a blank panel where a sign used to be.

So the renderer classifies each box before touching it. It samples the region's colours and asks whether the non-glyph pixels are nearly uniform — a per-channel standard deviation below a small threshold means the box sits on a flat fill it can reproduce exactly. Flat boxes skip inpainting entirely and are repainted with the sampled colour. Only boxes on photographic or textured backgrounds go to the inpainting model.

Erasing a label that isn't axis-aligned

Repainting a flat box with a rectangle fill introduces its own failure, and it appears the moment a sign is photographed at any angle. A label's edge then runs diagonally through the detection box, and no axis-aligned rectangle matches it: fill the whole box and the label's colour spills out onto the surroundings — a red band's fill visibly poking into the white panel around it.

The fix is to stop filling rectangles and paint only the glyphs. Inside the box, pixels far from the background colour are grouped into connected components. Components touching the box border are the surroundings leaking in and are left alone; components fully enclosed by the flat fill are the characters, and only those get painted over with the background colour. Everything already matching the background keeps its original pixels, so the label's grain and edges survive untouched.

There is a second, related trim. A detection box often overhangs its label — a rotated polygon's bounding box reaches past the edge — so the renderer walks each edge inward while the row or column has almost no background-coloured pixels. That gives the label's true extent, which is what the translation gets centred on. Centring on the raw detection box would push the text off the label whenever the box overhangs.

The colour problem

Translated text should look like the text it replaced, which means recovering two colours from the original: the background and the ink.

Both come from the box's pixels, sampled before anything is erased. The background is the per-channel median — glyphs are the minority of pixels in any text box, so the median lands on the fill. The ink is the median of the pixels that differ most from that background. Two guards sit on top:

If foreground and background come out too close to distinguish, the renderer substitutes black or white based on the background's luminance. Without this, a low-contrast sample can produce text that is invisible against its own background.
For boxes that went through inpainting, the model chose the background, not us. So after inpainting, the sampled ink colour is checked against what was actually painted and swapped for a contrasting one if it no longer reads.

The length problem

Translation changes length, and it changes it in both directions. Japanese and Chinese are compact: 恋愛の石 is four characters that become "Stone of Love" in English and 연애의 돌 in Korean. German and Finnish routinely expand past English. A layout that was designed around the source text has no reason to accommodate the target.

Naively scaling the font to fill the box height fails immediately. A short translation in a tall box renders as two enormous characters; a long one in a wide box runs off the end. The renderer instead searches for the largest font size at which the wrapped text fits both dimensions, starting from roughly the box height and stepping down.

Wrapping is script-aware, because line breaking is not universal. Space-delimited scripts wrap at word boundaries. Chinese, Japanese and Korean have no spaces to break at, so they wrap per character — which is correct for those scripts, and the only way to fill a tall box with several lines rather than stretching one line across it.

The font problem, and the tofu it produces

This one is easy to get wrong in a way that only shows up in production, on other people's images.

When translating into Korean, the obvious choice is a Korean font — Apple SD Gothic Neo on macOS, for instance. It covers Hangul and Latin beautifully. But the renderer does not only draw Korean: anything the translator leaves in the source language still has to be drawn. A dropped OCR box, a partially translated string, a place name passed through untouched — all of it arrives as Chinese or Japanese characters that a Hangul-focused font may not contain.

The result is tofu: the empty box glyph a font renders when it has no character. We had 現在地 ("current location") on a mall floor guide come out as ⊠在地, with one character replaced by a box, because the font lacked that particular Han character. Common signage and menu characters — 麦, 噌, 鶏, 税込, 恋 — were missing too.

Worse, the obvious fallback did not fix it. Google Fonts' Noto Sans KR variable font is a subset and misses several of the same characters. Only the full pan-CJK cut covers them. So the Korean rendering path downloads that font once and uses it, and a script in the repository verifies coverage by reading the font's cmap table directly rather than by eyeballing output. If the download cannot happen, the worker falls back to a system font and renders anyway — tofu is bad, but failing the whole job is worse.

Vertical text needs its own rendering path

A box that is still much taller than it is wide after merging is a standing vertical column, and horizontal text does not belong in it. Fitting a short translation there produces the giant-glyph failure; fitting a long one shrinks it to nothing.

For Korean, Japanese and Chinese targets the renderer stacks the translation one character per line, which is how those scripts are set vertically anyway. For Latin-script targets it rotates the text 90° clockwise, the way vertical Latin signage is set. It decides between orientations by fitting both and taking whichever achieves the larger font, rather than trusting the aspect ratio — that keeps a narrow detection like "5" or "I" from being turned sideways for no reason. The detail of how those columns get assembled in the first place is covered in the vertical Japanese article.

Order of operations matters

One structural decision is worth stating because it is easy to get wrong and hard to debug: the renderer works in three passes over the whole image rather than handling one box at a time.

First every box is classified and its colours sampled — from the original image, before anything changes. Then all erasing happens. Only then is any text drawn. Interleaving these would let a later box's erase paint over an earlier box's freshly drawn translation, or skew a colour sample by reading pixels that were already modified. Both happen only when boxes overlap or sit close together, which is exactly the case on dense menus — the images where the tool is most useful.

What this means for your images

Flat-background text reproduces best. Menus, signs, labels and packaging come out cleanest, because the background can be reproduced exactly rather than guessed.
Text over photographs is harder. Inpainting has to invent the background, and invented backgrounds are the most visible artefact in the output.
Untranslated fragments are a coverage signal. If some text stayed in the source language, OCR did not read it — that is the detection or recognition stage, not the translation. See how OCR actually works for what to do about it.

Try it on Image Translator — the source image and the rendered translation are shown side by side, so the erasing and refitting are visible on your own photos.