Automatically describing the content of an image is a fundamental problem in artificial intelligence that connects computer vision and natural language processing. In this paper, we present a generative model based on a deep recurrent architecture that combines recent advances in computer vision and machine translation. Our model is trained to maximize the likelihood of the target description sentence given the training image. Experiments on several datasets show the accuracy of the model and the fluency of the language it learns solely from image descriptions. Our model is often quite accurate, which we verify both with automatic metrics (BLEU) and with human evaluation. The model is based on a convolutional neural network (CNN) encoder followed by a long short-term memory (LSTM) recurrent neural network decoder.

Being able to automatically describe the content of an image using properly formed English sentences is a very challenging task, but it could have great impact, for instance in helping visually impaired people understand images on the web. The task requires not only understanding the visual content (objects, actions, spatial relationships) but also being able to express this understanding in natural language. This is in contrast to most computer vision tasks that produce structured but non-linguistic outputs such as bounding boxes or class labels. Our inspiration is the recent success of sequence-to-sequence models for machine translation. We propose to treat image captioning as translating an image into a sentence, using a similar encoder-decoder framework.

Previous work on image captioning can be broadly grouped into template-based methods that fill in blanks in predefined sentence templates, retrieval-based methods that find the most similar caption from a database, and generation-based methods that produce novel sentences. Early generation approaches typically involve complex pipelines with separate visual detection, language modeling, and sentence planning stages. Concurrently with our work, Karpathy and Fei-Fei and Mao et al. have proposed similar neural approaches. Our key architectural choice — using a pre-trained CNN as encoder and LSTM as decoder, with the image fed only at the first time step — is inspired by the encoder-decoder paradigm in neural machine translation by Cho et al. and Sutskever et al.

Our model, inspired by machine translation, takes an image I as input and is trained to maximize the probability p(S|I) of producing a target sequence of words S = {S_1, S_2, ...}. Using the chain rule, this probability is decomposed as: log p(S|I) = sum_t log p(S_t | I, S_0, ..., S_{t-1}). The image is encoded using a deep CNN (GoogLeNet), producing a fixed-length vector representation. This vector is fed into an LSTM as the initial input, which then generates the caption one word at a time. The LSTM maintains a hidden state that acts as a "memory" of the sentence generated so far and of the visual content. Each word is represented as a one-hot vector mapped through a word embedding matrix.

The model is trained end-to-end using stochastic gradient descent to minimize the negative log-likelihood of the correct word at each step. The CNN encoder is initialized with weights pre-trained on ImageNet and fine-tuned jointly with the LSTM. Training is performed on pairs of (image, caption) from datasets such as Flickr30k and MS COCO. We use a vocabulary of the most frequent words and replace rare words with an <UNK> token. Dropout is applied for regularization. The training process is straightforward and does not require any task-specific engineering beyond the end-to-end neural architecture.

At test time, we use beam search to approximately find the most likely sentence given the image. Beam search maintains the top-k candidate partial sentences at each time step, expanding each by one word and keeping only the k best according to the model's log-probability. We found that a beam size of 20 provides a good trade-off between quality and computation time. Compared to greedy search (beam size 1), beam search significantly improves the quality of generated captions as measured by BLEU score.

We evaluate on Flickr30k, Flickr8k, MS COCO, and the SBU dataset. On MS COCO, our model achieves a BLEU-4 score of 27.7, the best at the time of submission. Human evaluations show that our descriptions are often indistinguishable from human-written captions — in a forced-choice setting, raters preferred or rated as equal our model's captions over 20% of the time when compared to ground-truth human descriptions. Qualitative analysis shows the model generates fluent and relevant descriptions for diverse image types. Common failure modes include misidentifying objects, generating overly generic descriptions, and failing to capture unusual spatial configurations.

We have presented a simple yet effective neural image caption generator based on a CNN encoder and LSTM decoder, inspired by neural machine translation. The model is trained end-to-end and generates novel, fluent English descriptions of images. Despite its simplicity, the model achieves state-of-the-art results on multiple benchmarks. We believe the key factors behind its success are the powerful visual features from a pre-trained CNN, the expressive language model from the LSTM, and the end-to-end training that allows these components to jointly adapt. Future work includes incorporating attention mechanisms to allow the model to focus on different image regions when generating different words.