Models with convolutional layers have dominated recent approaches to visual recognition, while models with recurrent layers are used extensively for sequential tasks. In this work, we propose Long-term Recurrent Convolutional Networks (LRCNs), a class of architectures that combines the strengths of both convolutional and recurrent networks to jointly process visual and sequential data. We demonstrate the value of these models on three visual tasks that benefit from sequential processing: activity recognition, image description, and video description. Our models are "doubly deep" in that they are both spatially deep through the CNN and temporally deep through the LSTM. We achieve state-of-the-art or competitive results on all three tasks.

Visual recognition tasks have been traditionally addressed as "one-shot" predictions: given an image, produce a label. However, many visual tasks are inherently sequential: video understanding requires processing frames over time, and image captioning requires generating words sequentially. While CNNs excel at extracting spatial features from individual frames, they lack the ability to model temporal dynamics or sequential output. Conversely, RNNs (in particular LSTMs) are designed for sequential data but lack strong visual feature extractors. We propose to combine the two in a unified architecture that can be applied to visual tasks with sequential inputs, outputs, or both.

Activity recognition methods have evolved from hand-crafted features (HOG, HOF, dense trajectories) to deep learning approaches. Two-stream networks process spatial and temporal information separately, while 3D convolutional networks apply convolutions in both space and time. However, these approaches typically use fixed temporal windows and cannot model long-range temporal dependencies. For image and video description, template-based and retrieval-based methods are being superseded by neural generation approaches. Concurrent work by Vinyals et al. (Show and Tell) uses a similar CNN+LSTM architecture for image captioning. Our contribution is to demonstrate that a single CNN+LSTM framework handles not just caption generation but also recognition and video description tasks.

The LRCN architecture consists of a visual feature extractor (CNN), followed by a sequence model (LSTM), followed by a prediction layer. The architecture is general and can handle three types of visual tasks by varying how the inputs and outputs interact with the sequential component: (1) Sequential input, fixed output — video classification, where each frame is processed by the CNN and the sequence of features is fed to the LSTM; (2) Fixed input, sequential output — image description, where the CNN processes a single image and the LSTM generates a sentence; (3) Sequential input, sequential output — video description, where both the visual input and the text output are sequential. In all cases, the CNN weights are shared across time steps, and the entire model is trained end-to-end.

For activity recognition, we process each video frame through a CaffeNet/VGG CNN to extract fc7 features, which are then fed as a sequence to a two-layer LSTM with 256 hidden units. The LSTM's final hidden state is used for classification. For image description, we encode the image with the CNN and condition the LSTM on this visual representation to generate a caption word by word. For video description, mean-pooled CNN features across frames serve as the visual representation, followed by LSTM caption generation. In each case, we initialize the CNN with ImageNet pre-trained weights and fine-tune jointly with the LSTM. The same fundamental architecture — CNN feeding into LSTM — is applied consistently across all three tasks, with only minor variations in how the visual and sequential components are connected.

For activity recognition, we evaluate on UCF-101 and achieve 82.9% accuracy, competitive with the state-of-the-art two-stream approach. The LSTM effectively captures temporal patterns that a single-frame CNN misses, improving accuracy by 5-10% over frame-level classification. For image description, we report results on Flickr30k, achieving comparable BLEU scores to concurrent approaches including Show and Tell. For video description, we evaluate on TACoS Multilevel and YouTube2Text, showing improvements over prior work. Ablation studies reveal that the LSTM component is crucial: replacing it with a simpler temporal pooling degrades performance significantly on all tasks.

We have proposed Long-term Recurrent Convolutional Networks (LRCNs), a unified class of models that combine convolutional and recurrent networks for visual tasks involving sequential processing. By demonstrating competitive results on activity recognition, image description, and video description, we have shown that a single architecture paradigm — CNN features processed by LSTM — is broadly applicable. The key advantage is the ability to model long-range temporal dependencies through the LSTM's gating mechanism, which is critical for understanding temporal visual patterns and generating coherent linguistic output. Future work may explore attention mechanisms and deeper recurrent architectures.