We introduce AR-Net (Adaptive Resolution Network) for efficient action recognition in videos. The key insight is that not all frames in a video require the same spatial resolution for accurate recognition. AR-Net uses a lightweight policy network to dynamically select the optimal resolution for each frame, allocating more computation to informative frames and less to redundant ones. This achieves comparable accuracy to fixed high-resolution processing with up to 50% fewer FLOPs.

Video understanding models are computationally expensive due to processing many high-resolution frames. Existing efficiency methods use frame sampling or temporal pruning, treating all spatial locations uniformly. We observe that spatial resolution requirements vary dramatically across frames: frames with large motion or fine-grained actions need high resolution, while static or transition frames can be processed at low resolution without accuracy loss.

Our approach is inspired by the observation that human visual attention naturally adjusts resolution based on content complexity. When watching a video, we focus more on key action moments and less on static backgrounds. AR-Net mimics this behavior by learning a policy that allocates computational resources proportionally to frame informativeness. This is fundamentally different from approaches that reduce temporal redundancy, as we target spatial redundancy within individual frames.

AR-Net consists of two components: a policy network and a recognition backbone. The policy network is a lightweight CNN that takes a low-resolution version of each frame and predicts the optimal resolution level from a set of candidates (e.g., 84x84, 128x128, 224x224). The selected resolution is used to resize the frame before feeding it to the backbone. The policy network is trained with reinforcement learning (Gumbel-Softmax) to maximize accuracy while minimizing computational cost.

The reward function balances accuracy and efficiency: R = accuracy - lambda * computation_cost. The computation cost is measured as the FLOPs of processing the frame at the selected resolution. Lambda controls the accuracy-efficiency trade-off, allowing users to tune the model for different deployment constraints. During inference, the policy network adds less than 1% additional FLOPs.

To stabilize training, we employ a curriculum learning strategy that starts with a high lambda (favoring efficiency) and gradually decreases it to allow the network to learn accurate resolution selection. We also use baseline subtraction in the REINFORCE gradient estimator to reduce variance. The Gumbel-Softmax temperature is annealed from 1.0 to 0.1 during training, transitioning from soft to hard selection.

On ActivityNet with ResNet-50 backbone, AR-Net achieves 72.0% mAP with 48% fewer FLOPs compared to fixed 224x224 resolution (72.4% mAP). On FCVID, AR-Net achieves 84.3% mAP with 42% fewer FLOPs. Analysis shows that the policy network correctly assigns higher resolution to action-centric frames and lower resolution to background frames.

We further analyze the resolution distribution learned by the policy network. On ActivityNet, approximately 35% of frames are assigned the lowest resolution, 40% medium, and only 25% the highest. This confirms the hypothesis that most frames do not require full resolution. Ablation studies show that the policy network outperforms random resolution assignment by 2.1% mAP and uniform low-resolution processing by 3.8% mAP.

We have presented AR-Net, which demonstrates that adaptive frame resolution is an effective and orthogonal dimension for improving video understanding efficiency. The approach is model-agnostic and can be combined with other efficiency methods like temporal sampling. AR-Net opens up a promising direction for content-adaptive computation in video understanding.