ToolSink
Back to blog

Image to Text OCR: How It Actually Works (And When It Fails)

ToolSink Team

Optical Character Recognition (OCR) turns an image of text — a scanned document, a photo of a whiteboard, a screenshot — into actual selectable, searchable, editable text. It's genuinely useful, but it's also not magic: specific, predictable things make it fail, and knowing them upfront saves a lot of "why didn't this work" troubleshooting after the fact.

The actual pipeline

1. Pre-processing. Before recognition starts, the image gets cleaned up:

  • Binarization — converting to pure black and white to maximize contrast between text and background
  • Deskewing — auto-rotating an image captured at an angle so text lines run horizontally
  • Noise reduction — smoothing out scan artifacts, dust, or compression noise that could be mistaken for punctuation

2. Character recognition. Modern engines primarily use feature extraction — breaking each character into structural features (line segments, closed loops, intersections) rather than just template-matching against a font database. This is what lets OCR handle unfamiliar fonts and, to a degree, handwriting — it's recognizing structural patterns, not memorized shapes.

3. Post-processing. The raw character output gets checked against a dictionary and language model. If the engine reads "l-p-p-l-e," a language model flags this as not a real word and corrects it to "apple" based on context — this step is responsible for most of the accuracy improvement in the last decade of OCR, more than raw character recognition itself.

Try it yourself

Tesseract (open-source, the engine behind many free OCR tools) via Python:

import pytesseract
from PIL import Image

image = Image.open("document.png")
text = pytesseract.image_to_string(image)
print(text)

# Get confidence scores per recognized word, not just the text
data = pytesseract.image_to_data(image, output_type=pytesseract.Output.DICT)
for word, conf in zip(data['text'], data['conf']):
    if word.strip():
        print(f"{word}: {conf}% confidence")

The confidence score matters in practice: a batch OCR job processing hundreds of scanned forms can flag any word below, say, 70% confidence for manual review, rather than trusting every extracted character equally.

What actually breaks OCR accuracy

  • Low resolution — text under roughly 150 DPI-equivalent detail starts producing frequent character misreads (e.g., confusing "rn" for "m", or "1" for "l")
  • Skewed or rotated images — most engines auto-deskew reasonably well up to a point, but a steep angle or a curved page (like a photo of an open book) degrades accuracy noticeably
  • Cursive or highly stylized handwriting — feature extraction handles print and clear cursive reasonably, but fast, inconsistent handwriting still has meaningfully higher error rates than typed text
  • Low contrast — light gray text on white, or text over a busy background image, undermines the binarization step before recognition even starts
  • Complex layouts — multi-column text, tables, or text overlapping images can get extracted out of reading order, since the engine is working out layout structure algorithmically, not understanding the document's actual logical flow

If OCR output looks wrong, checking which of these applies to your source image is usually faster than assuming the tool itself is broken.

Where this is genuinely useful

  • Digitizing paper archives — contracts, forms, historical records become searchable rather than requiring manual reading
  • Accessibility — extracted text can feed screen readers or text-to-speech, making printed material accessible to visually impaired users
  • Data entry automation — extracting structured data from receipts, forms, or business cards instead of manual transcription
  • Making scanned documents searchable — turning a 50-page scanned PDF into something you can Ctrl+F instead of reading manually

Choosing a tool

  • Language support — confirm the tool handles the specific language/script you need; accuracy varies significantly by language, and support for non-Latin scripts is uneven across tools
  • Batch processing — matters if you're digitizing volume rather than one-off images
  • Export format — plain text is universal; some tools also produce a searchable PDF (original image + an invisible text layer) which preserves the visual document while adding searchability

For a quick one-off extraction, ToolSink's Image to Text tool runs this process directly in your browser without needing any setup.

Back to blog