This paper presents stacked attention networks (SANs) that learn to answer natural language questions from images. The key innovation is using semantic question representations to search for regions in an image that are related to the answer. The authors argue that image QA requires multiple reasoning steps, so they develop layered SANs that query an image multiple times to infer the answer progressively. Experiments on DAQUAR, COCO-QA, and VQA datasets demonstrate that the proposed model significantly outperforms previous methods.

Image question answering (image QA) has emerged as a challenging task at the intersection of computer vision and natural language processing. Given an image and a free-form natural language question about the image, the task is to automatically produce a concise and accurate answer. Most existing approaches extract a global image feature vector using a convolutional neural network (CNN) and combine it with an LSTM-encoded question representation. However, these methods fail when answers relate to fine-grained regions in an image rather than the global scene.

The proposed Stacked Attention Network (SAN) architecture has three key components: (1) an image model based on a CNN that extracts spatial feature maps preserving region-level information; (2) a question model using either CNN or LSTM to encode the question semantics into a vector; and (3) a stacked attention model that uses the question representation to progressively search for relevant image regions through multiple attention layers. Each attention layer refines the query vector by aggregating attended image features, enabling a multi-step reasoning process.

Prior work on image question answering can be grouped into two categories. The first uses semantic parsing to map questions to logical representations and query structured knowledge bases, which limits flexibility. The second treats VQA as classification by combining CNN image features with LSTM question encodings. Notable works include Malinowski et al.'s Neural-Image-QA and Ren et al.'s image-question encoders. However, all these methods use global image features without spatial attention, making them unable to focus on specific image regions relevant to the question.

The image model uses VGGNet to extract features from the last pooling layer, producing a 512 x 14 x 14 dimensional feature map. Each 512-dimensional vector corresponds to a 32 x 32 pixel region of the input image, preserving spatial layout. For the question model, two variants are explored: an LSTM-based model that takes the final hidden state as the question vector, and a CNN-based model that applies unigram, bigram, and trigram convolutions with max-pooling to capture phrase-level semantics.

The core innovation is the stacked attention mechanism. For each attention layer k, the model computes an attention distribution over the 196 image regions (14 x 14 grid) using the previous layer's refined query vector. The attention weights are calculated by projecting both the image features and the query vector into a common space, then applying a softmax. The attended image feature vector is computed as the weighted sum of all region features, which is then added to the previous query vector to form the refined query for the next layer. This iterative process allows the model to progressively narrow its focus from broad object categories toward answer-relevant regions.

The number of attention layers is a hyperparameter. The authors experiment with one-layer and two-layer SANs, finding that two attention layers consistently outperform a single layer across all datasets, while further stacking yields diminishing returns. The first attention layer tends to attend to all instances of the relevant object category, while the second layer focuses on the specific instance most relevant to the question.

Experiments are conducted on four benchmarks: DAQUAR-ALL (6,795 training questions on 795 images), DAQUAR-REDUCED (3,876 training samples on 25 test images), COCO-QA (78,736 training samples across four question types), and VQA (248,349 training questions), the largest image QA dataset at the time. The two-layer SAN achieves significant improvements over all baselines: 29.3% accuracy on DAQUAR-ALL (vs. 23.4% best baseline), 61.6% on COCO-QA (vs. 55.9%), and 58.9% on the VQA test set (vs. 54.1%). Performance gains are largest for "Color" and "Object" question types, consistent with the hypothesis that spatial attention helps identify specific visual attributes.

The paper demonstrates that multiple attention layers enable superior performance through progressive visual focus refinement. The Stacked Attention Network queries the image multiple times, with each layer progressively narrowing attention from broad object categories toward answer-relevant regions. Visualizations clearly show the multi-step refinement process. The method achieves state-of-the-art results across all evaluated benchmarks, confirming that spatial attention is a critical component for image question answering.