Driving World Models (DWMs) have become essential for autonomous driving by enabling future scene prediction. However, existing DWMs are limited to scene generation and fail to incorporate scene understanding, which involves interpreting and reasoning about the driving environment. This paper introduces Hermes, which seamlessly integrates 3D scene understanding and future scene generation through a unified framework. It leverages Bird's-Eye View (BEV) representation to consolidate multi-view spatial information while preserving geometric relationships. The approach introduces world queries that incorporate world knowledge into BEV features via causal attention in Large Language Models. Results demonstrate reducing generation error by 32.4% and improving understanding metrics such as CIDEr by 8.0%.

Current Driving World Models excel at predicting environmental evolution but lack interpretation capabilities — they cannot describe environments, answer questions, or provide contextual information. Conversely, vision-language models demonstrate impressive capabilities in driving scene understanding but lack predictive capabilities for scene evolution. This gap creates the core research question: "how can world knowledge and future scene evolutions be seamlessly integrated into a unified world model?"

Three key challenges are identified. First, LLMs typically face token length limitations, especially in autonomous driving where multiple surrounding views must be processed. The solution leverages BEV representation, which effectively compresses surrounding views into unified latent space. Second, a straightforward approach of sharing BEV features with separate models fails to leverage potential interactions between tasks. The proposed solution uses world queries initialized from raw BEV features, enhanced with world knowledge through causal attention in the LLM.

Research on Driving World Models focuses primarily on generation in both 2D and 3D dimensions. GAIA-1 introduced learned simulators based on autoregressive models. Recent work leverages large-scale data and powerful pre-training models, "significantly enhancing generation quality regarding consistency, resolution, and controllability". For 3D spatial information, OccWorld focuses on future occupancy generation, and ViDAR uses images to predict future point clouds through self-supervision. Critical limitation: "they overlook the explicit understanding capacity of the driving environment".

Multi-view images at time t are passed through a CLIP image encoder and a single-frame BEVFormer. The obtained BEV feature captures world semantic and geometric information. "Such a feature is too large for the LLM, often containing tens of thousands of tokens." A down-sampling block reduces the feature by two times. For point cloud generation, compressed BEV features are up-sampled, reshaped by adding an extra height dimension, then processed through 3D convolutions to reconstruct the volumetric feature map. Differentiable volume rendering models the environment as an implicit signed distance function (SDF) field to compute depth for each ray.

The LLM (InternVL2-2B) processes flattened BEV to interpret driving scenarios based on user instructions. For understanding, the LLM responds to queries through auto-regressive next-token prediction. For generation, world queries are proposed — groups of n queries initialized via max pooling from BEV features, conditioned on ego-motion encoding and position embeddings. After LLM processing, a "current to future link" module employs cross-attention layers to inject world knowledge into future BEV features. This design ensures generated scenes are contextually aware and enriched with world knowledge from the LLM's understanding capability.

Training is structured into three stages. Stage 1 trains the World Tokenizer and Render to convert current images into point clouds. Stage 2 establishes BEV-text alignment via caption data, then refines using LoRA on OmniDrive scene descriptions. Stage 3 introduces future generation modules, unifying understanding and generation using nuScenes keyframes and OmniDrive conversations. The total loss combines auto-regressive language modeling loss and L1 depth supervision loss with frame-wise weights.

Experiments are conducted on nuScenes and OmniDrive-nuScenes datasets. For future point cloud generation, Hermes achieves approximately 32% Chamfer Distance reduction in 3-second point clouds compared to ViDAR. Notably, ViDAR utilizes 3-second history horizon and carefully designed latent rendering, while Hermes uses only current multi-view images and simple volumetric representation. For understanding, Hermes achieves highly competitive caption quality, notably outperforming OmniDrive by 8% on the CIDEr metric, despite using only a subset of OmniDrive's training data.

Ablation studies examine world query design. Testing n from 1 to 16 shows world queries don't negatively affect text understanding quality. Increasing the number beyond 4 leads to performance decline due to redundant information. Max pooling initialization achieves a 0.03 Chamfer Distance reduction over alternative pooling strategies. For understanding-generation interaction, the unified approach outperforms separated unification in generation results, as the latter fails to exploit potential interactions between tasks.

This paper introduces Hermes, "a simple yet effective unified Driving World Model that integrates 3D scene understanding and future scene generation within a single framework". By leveraging BEV representation and incorporating world queries enhanced through large language models, the approach effectively bridges the gap between understanding and generation. Extensive experiments validate effectiveness with "significant improvements in future scene prediction accuracy and understanding metrics, surpassing state-of-the-art methods". Future work plans to investigate multi-modal unified world models facilitating both multi-modal input and output.