Diffusion models have revolutionized generative modeling, enabling unprecedented realism in image and video synthesis. This research systematically compares video versus image diffusion models using identical architectures. The study evaluates representations on downstream tasks including image classification, action recognition, depth estimation, and tracking. Key findings show video diffusion models consistently outperform image counterparts, with particularly large gains on motion-dependent tasks.

While image diffusion models have demonstrated effectiveness for visual understanding tasks, "the representational power of video diffusion models remains largely unexplored". The research uses the WALT architecture, which permits fair comparison between image and video training objectives through its hybrid design allowing both image and video generation modes. This is the first direct comparison of image versus video diffusion objectives for visual understanding tasks.

The paper surveys three main areas: image diffusion for visual understanding, video representation learning, and concurrent work on video diffusion models. Existing literature documents that diffusion models excel at semantic segmentation, depth estimation, and correspondence tasks. However, few studies directly compare video and image diffusion representations using identical architectures. The representational gap between generative and discriminative objectives remains an active research question.

The authors employ latent diffusion models operating in compressed latent space using a VQ-VAE encoder and denoiser network. The WALT architecture is selected for its hybrid design: I-WALT processes individual frames independently, while V-WALT (video variant) includes spatio-temporal attention blocks capturing motion dynamics across 17 frames. The probing framework extracts frozen features at various noise levels and network depths, evaluating eight downstream tasks spanning semantic understanding to spatio-temporal reasoning.

Feature visualizations via PCA reveal that V-WALT activations concentrate on moving regions, while I-WALT highlights semantically important areas regardless of motion. A brick wall experiment confirms V-WALT's motion sensitivity: when the camera pans across a uniform texture, V-WALT activations track the motion while I-WALT shows uniform responses. This demonstrates that video training fundamentally alters the model's attention patterns toward temporal dynamics.

V-WALT outperforms I-WALT across all tasks, with performance gains ranging from modest improvements (0.6% on Places365) to substantial gains (68% on point tracking, 60% on camera pose estimation). Motion-sensitive tasks show larger improvements than semantic classification, suggesting video training primarily enhances temporal and geometric understanding rather than object-level semantics. Optimal performance occurs at intermediate network depths (approximately 2/3 through the model) and moderate noise levels (timestep 200).

Training budget effects reveal that recognition tasks show consistent improvement with longer training, while tracking and depth estimation peak at earlier training stages. Camera pose estimation demonstrates performance decline after 26% training completion, suggesting potential overfitting. For scaling behavior, increasing model size from 284M to 1.9B parameters substantially improves V-WALT on semantic tasks but provides marginal gains on point tracking. Compared with other models, V-WALT demonstrates competitiveness with DINOv2 on motion understanding but semantic classification favors models like SigLIP and DINOv2.

The study establishes that video diffusion models consistently outperform their image counterparts as visual representations, with particularly large gains on motion-dependent tasks. The authors acknowledge limitations: studying a single architecture (WALT) limits generalizability. Future work should explore additional unified model architectures, investigate relationships between generation quality and representation learning, and examine the intersection of generative and perceptual capabilities in diffusion models.