We describe PhotoOCR, a system for reading text in natural images captured by smartphone cameras. Our approach combines advances in text detection, character classification using deep neural networks, and distributed language modeling at datacenter scale. The system significantly outperforms commercial OCR engines on challenging benchmarks with text affected by blur, low resolution, unusual fonts, and non-uniform illumination. We achieve a mean processing time of 600 ms per image while substantially reducing error rates compared to prior methods. The system has been deployed in Google Translate for Android and several other Google products, demonstrating its practical viability at scale.

Reading text in natural scenes is a problem with tremendous practical importance. Smartphones have become ubiquitous, and users increasingly rely on camera-based input for translation, navigation, and information retrieval. Unlike traditional document OCR, which operates on clean, well-formatted scanned documents, scene text recognition must cope with extreme variations in appearance, perspective distortion, partial occlusion, complex backgrounds, and variable illumination. Existing commercial OCR systems, designed primarily for document scanning, perform poorly on such challenging inputs.

PhotoOCR addresses these challenges through a tightly integrated pipeline consisting of three main components: (1) a text detection module that localizes text regions in the image using a combination of connected component analysis and trained classifiers; (2) a character recognition module that employs deep neural networks (DNNs) with HOG-like features for robust character classification; and (3) a language model that leverages Google's datacenter-scale n-gram models to decode character sequences into words. Each component is optimized for the specific challenges of unconstrained text in real-world photos.

Scene text detection has been addressed through connected component methods like Maximally Stable Extremal Regions (MSER) and Stroke Width Transform (SWT), as well as sliding window approaches. For text recognition, Mishra et al. used CRF-based word recognition, while Wang et al. applied convolutional neural networks for character classification. Netzer et al. introduced the Street View House Numbers (SVHN) dataset, benchmarking deep learning approaches on digit recognition. Our work differs in providing a complete, production-quality system that integrates state-of-the-art methods for each stage with datacenter-scale language modeling, validated not only on benchmarks but through real-world deployment.

The text detection module combines connected component analysis with learned classifiers. We first extract candidate text regions using enhanced MSER on multiple color channels, followed by stroke width verification to filter non-text components. Candidate components are then grouped into text lines using geometric constraints (alignment, spacing, size consistency). A trained binary classifier scores each candidate text line based on HOG features, edge density, and stroke width statistics. This multi-stage pipeline achieves high recall (>90%) while maintaining reasonable precision, as subsequent stages can filter false positives.

For character classification, we train a deep neural network (DNN) with five hidden layers and several thousand hidden units per layer. The input features are HOG descriptors computed at multiple scales from character image patches. Training data is generated through extensive data augmentation including affine transformations, blur, noise, and contrast variations applied to millions of font-rendered character images supplemented with manually labeled real-world examples. The DNN achieves character-level accuracy exceeding 98% on held-out test sets. Word-level decoding combines the DNN's character probabilities with a distributed n-gram language model trained on web-scale text corpora, using beam search decoding to find the most likely word sequence.

We evaluate PhotoOCR on three public benchmarks: ICDAR 2003 (258 images), ICDAR 2011 (255 images), and Street View Text (SVT, 249 images). On ICDAR 2003 word recognition, we achieve 93.9% accuracy, compared to 76.1% for the best prior method. On the more challenging SVT dataset with street-level imagery, we achieve 90.4% accuracy, a relative error reduction of 34% over the state of the art. End-to-end processing (detection + recognition) on full images achieves an F-measure of 0.86 on ICDAR 2011. The mean processing time is 600 ms per image on a single machine, demonstrating practical efficiency. We also compare against commercial OCR engines (ABBYY, Tesseract), showing dramatically lower error rates especially on degraded inputs.

PhotoOCR demonstrates that a carefully engineered system combining deep neural networks for character recognition with large-scale language modeling can dramatically advance the state of the art in scene text reading. By leveraging massive training data, extensive data augmentation, and datacenter-scale language models, our system achieves substantial error reductions on all evaluated benchmarks. The successful deployment in Google Translate and other products validates the approach in real-world conditions far more diverse than any benchmark. We believe that the combination of deep learning and large-scale language priors is a promising direction for text understanding in the wild.