Attention mechanisms have become a key component in Visual Question Answering (VQA). However, existing attention models do not take into account the spatial relations between regions when predicting the attention distribution. This paper proposes to model visual attention as a multivariate distribution over a grid-structured Conditional Random Field (CRF) on image regions. The authors demonstrate that iterative inference algorithms, including Mean Field and Loopy Belief Propagation, can be unrolled as recurrent neural network layers, enabling end-to-end training. The approach achieves improvements of 9.5% on CLEVR and 1.625% on the VQA dataset compared to prior baselines.

Visual Question Answering requires a system to understand both an image and a natural language question, then produce a correct answer. Attention mechanisms have been widely adopted to focus on relevant image regions given the question context. Most existing approaches compute attention as independent softmax scores over image regions, treating each region in isolation. This ignores the spatial arrangement and relational structure among regions, which is critical for questions involving spatial reasoning such as "What is to the left of the red object?"

The limited receptive field of convolutional neural networks further exacerbates this problem — regions that are spatially distant may lack overlapping feature maps, making it difficult for the network to capture their relationships. The authors propose structured attentions that explicitly model the spatial dependencies between image regions using a graphical model, specifically a grid-structured CRF. This allows the attention distribution to encode spatial coherence and relational information that flat attention mechanisms miss.

Attention in VQA has evolved from simple soft attention over image grids to stacked attention networks that refine focus iteratively. Co-attention models jointly attend to image regions and question words. Despite their effectiveness, these mechanisms treat attention as factored distributions where each region's weight is computed independently. Conditional Random Fields have been extensively used in semantic segmentation to enforce spatial consistency, and recent work has shown that CRF inference can be implemented as differentiable neural network layers.

The core innovation is formulating attention as the marginal distribution of a grid-structured CRF defined over image regions. The unary potentials are derived from the question-conditioned feature similarity between image regions and the question representation. The pairwise potentials encode spatial relationships between neighbouring regions, encouraging nearby regions to have correlated attention values. The energy function is: E(a) = sum_i phi_i(a_i) + sum_{i,j} psi_{ij}(a_i, a_j), where the pairwise term captures spatial structure.

Computing the exact marginals of the CRF is intractable. The authors employ approximate inference algorithms — Mean Field (MF) and Loopy Belief Propagation (LBP) — and show that their iterative updates can be unrolled as recurrent neural network layers. Each iteration of MF or LBP corresponds to one time step of the RNN. This construction is fully differentiable, enabling end-to-end training with backpropagation. The number of inference iterations is treated as a hyperparameter, with 3-5 iterations typically sufficient.

The method is evaluated on three datasets: VQA, Visual7W, and CLEVR. On CLEVR, which specifically tests spatial reasoning, the structured attention achieves 9.5% improvement over flat attention baselines, demonstrating the strong benefit of spatial modelling for relational questions. On VQA, the improvement is 1.625% — smaller but consistent across question types. Ablation studies show that the pairwise potential contributes most to spatial reasoning questions, while the unary term dominates for object recognition questions. LBP slightly outperforms MF across all datasets.

This paper demonstrates the importance of encoding spatial relations in visual attention for VQA. By formulating attention as marginal inference in a grid-structured CRF and implementing inference as differentiable RNN layers, the approach seamlessly integrates structured probabilistic modelling into deep networks. Empirical validation across three representative datasets shows substantial improvements, particularly for spatial reasoning tasks. The framework is general and could be extended to other vision-language tasks requiring structured spatial understanding.