Text is everywhere in the real world. Street signs, shop boards, product labels, shipping boxes, number plates, and menus all carry words that people read without thinking. For a machine, reading them is much harder. The letters sit on curved bottles, tilt at odd angles, and fade in strong sunlight.
Teaching a model to read this kind of text takes carefully labeled photos. Good scene text annotation tells the model where the text is, what it says, and how it should be read. This post explains why text in photos is hard to label and what computer vision training data needs to get right.
Scene text is writing that appears inside a natural photo or video frame. It is different from a scanned page, where text sits in neat lines on a flat background. A photo of a street, a shelf, or a warehouse has text placed anywhere, in any size, on any surface.
This post stays on text in photos. Scanned forms and records raise different problems, such as page layout and filing, and they need their own rules.
Annotation for scene text usually has three parts.
Models trained on these labels power OCR data annotation projects such as reading product labels, sorting parcels, finding shop names on maps, and helping visually impaired users hear what a sign says.
Real scenes create problems that clean documents never do.
The shape a labeler draws shapes what the model learns. A plain box is quick but works poorly for tilted or curved words. A rotated box handles angles. A polygon with several points fits curved text well, but it takes more time and needs clear rules.
A project should pick the shape by the kind of text in its images and by what the model will do. A model that reads flat labels may need only boxes. One that reads text on bottles needs polygons. The key point is to decide early and apply the choice evenly, since mixed shapes in one dataset confuse training.
Detection and transcription are two jobs that look similar but fail in different ways.
In detection, the question is “is there text here, and where does it start and end?” Mistakes include boxes that cut off letters, boxes that merge two separate lines, and missed text in cluttered areas. Reviewers check shape and coverage.
In transcription, the question is “what exactly does it say?” Mistakes include swapped letters, wrong case, dropped spaces, and guessed words when the image is unclear. Reviewers check the typed text against the picture, character by character.
A model can detect text perfectly and still read it badly, so both jobs need their own checks.
Typing what you see sounds easy, but small choices add up across thousands of images. Teams need written rules for questions like these.
Without these rules, two annotators will type the same sign in two ways, and the model sees conflicting answers for the same input.
In many countries a single street scene holds several languages. A shop board may show a regional script on top, English below, and a phone number in numerals of a third style. Some packaging mixes scripts inside one line.
Annotators need to know every script that appears, and the label should record the language of each text region. A reviewer who reads only one script will mislabel the rest, or mark real text as noise. Reading direction matters too, since some scripts run right to left and others stack vertically. For this work, reviewers who read the local languages are not optional.
Some text cannot be read, and the label should say so. If a sign is blurred beyond recognition, a transcription guess teaches the model to hallucinate. A better workflow lets annotators mark text as present but unreadable. Such regions can then be kept for detection training, excluded from transcription training, or sent to an expert.
This also protects accuracy testing. A model should not be marked wrong for failing to read something no human could read either.
Photos of streets and shelves can include private text: house numbers, license plates, names on badges, and messages on phone screens. Teams should decide early which text is kept, which is masked, and who sees the raw images. Our guide to visual privacy in computer vision datasets explains how to mask these details without damaging the data.
People who know the setting read scene text more accurately. A reviewer who understands retail packaging knows which codes are batch numbers and which are prices. Someone familiar with local roads knows how signs are formatted and which abbreviations are standard. Native readers of each script catch errors that a general annotator would miss.
Specialists also write the rules for shapes, case, spacing, and unreadable items. This reflects Globik AI’s approach of using verified domain experts for work that depends on judgment, where a generalist would be guessing.
Before starting a scene text project, settle these points:
Reading text in the real world means handling angles, curves, glare, and many scripts at once. Strong scene text annotation uses the right shape for each piece of text, clear transcription rules, an honest label for unreadable items, and reviewers who read the languages in the images. With that foundation, models can read the signs, labels, and packaging that people rely on every day.
It is the labeling of text found in natural photos and video, such as signs, labels, and packaging. Annotators mark where the text is, type what it says, and note its language so models can learn to read it.
Scene text appears anywhere in a photo, on curved or tilted surfaces, with glare and busy backgrounds. Document OCR works on flat, ordered pages. Scene text needs flexible shapes and rules for harder, messier conditions.
Mark it as present but unreadable instead of guessing. The region can still help detection training, while staying out of transcription training. This stops the model from learning to invent words it cannot actually see.
A plain box cuts off or includes extra background around tilted or curved text. Rotated boxes and polygons follow the real outline, giving the model cleaner examples of where text begins and ends.
Specialists understand local scripts, packaging codes, and sign formats. They can tell real text from decoration and apply consistent transcription rules, which a generalist reading unfamiliar scripts would struggle to do.
