We introduce a Navigation World Model (NWM), a controllable video generation model that predicts future visual observations based on past observations and navigation actions. To this end, we train a Conditional Diffusion Transformer (CDiT) on diverse egocentric videos of both human and robotic agents, scaling the model to 1 billion parameters. NWM has several useful properties: (i) planning navigation trajectories by simulating them and evaluating whether they reach the goal; (ii) incorporating dynamic constraints during planning; (iii) ranking trajectories sampled from external policies; and (iv) imagining trajectories in novel, unseen environments from a single input image.

Navigation is a fundamental capability for any agent endowed with visual-motor skills. Current state-of-the-art approaches for robotic navigation learn hard-coded policies that cannot easily incorporate new constraints after training (e.g., "no left turns"), and supervised visual navigation models cannot dynamically allocate computational resources for difficult problems. We propose Navigation World Models (NWM), trained to predict future video frame representations given past frames and actions. After training, NWM plans trajectories by simulating potential plans and verifying whether goal states are achieved.

We introduce a novel Conditional Diffusion Transformer (CDiT) architecture with computational complexity linear with respect to the number of context frames, requiring 4x fewer FLOPs compared to a standard DiT while achieving better future prediction results. We test in two scenarios: (1) known environments, assessing the ability to plan novel trajectories independently or by ranking an external navigation policy using Model Predictive Control (MPC); (2) unknown environments, leveraging unlabeled Ego4D video data. We demonstrate competitive standalone performance and state-of-the-art results when combined with existing methods.

Our key contributions are: (1) Introduction of NWM and the novel CDiT architecture scaling to 1B parameters with reduced computational requirements; (2) Training on diverse robotic agent data enabling independent planning or ranking external policies with state-of-the-art performance; (3) Demonstration of improved video prediction in unseen environments through training on action and reward-free video data.

Goal-conditioned visual navigation requires both perception and planning. Recent methods like NoMaD use diffusion policies via behavior cloning and temporal distance objectives. Earlier approaches include Active Neural SLAM, combining neural SLAM with analytical planners, and various reinforcement learning-based methods. Our work demonstrates how world models can use exploratory data to plan or improve existing navigation policies, in contrast to these end-to-end policy learning approaches that lack explicit environment modeling.

World models simulate environments by predicting next states and rewards given current states and actions. Prior work showed joint policy-world model learning improves sample efficiency in Atari, simulated robotics, and real robots. DIAMOND and GameNGen applied diffusion models to learn game engines for Atari and Doom. Our work shares their inspiration but aims for a single general diffusion video transformer that can be shared across many environments and different embodiments for navigation, rather than being confined to a single game or simulator.

Recent advances in video generation, including Sora and MovieGen, have demonstrated impressive text-to-video synthesis capabilities. Prior work controlled video synthesis using action-object categories or Action Graphs. Diffusion models have also proved useful for view synthesis approaches. Our work differs by training a single model for navigation across diverse environments that models temporal dynamics from natural videos, without relying on 3D priors.

Given an egocentric video dataset with navigation actions D = {(x₀, a₀, ..., xₚ, aₚ)}, where xᵢ represents images and aᵢ = (u, φ) represents navigation commands with translation parameter u (forward/backward and left/right motion) and yaw rotation angle φ, the goal is to learn a world model F, a stochastic mapping from previous latent observations and actions to future latent states. Visual observations are encoded via a pretrained VAE: sᵢ = enc(xᵢ), providing benefits of working with compressed latents while allowing decoding predictions back to pixel space.

The formulation extends to include a time shift input k, so the action becomes a = (u, φ, k), allowing control over temporal dynamics. Navigation actions approximate the summation of individual actions from time τ to τ+k. In practice, time shifts reach plus or minus 16 seconds. A potential challenge is entanglement where reaching specific locations always occurs at particular times, causing models to ignore either time or actions. The authors address this by sampling multiple goals for each state during training, encouraging natural counterfactuals that decouple time from actions.

The architecture employs a temporally autoregressive transformer utilizing efficient CDiT blocks. CDiT enables time-efficient autoregressive modeling by constraining the first attention block to target frame tokens only. Cross-attention layers allow current target tokens to attend to past frame tokens (keys/values), contextualizing representations via skip connections. While standard DiT attention complexity is O(m² · n² · d), quadratic with context length, CDiT achieves O(m · n² · d), linear with the number of context frames, enabling significantly longer context sizes.

Continuous actions (translation u, rotation φ, time shift k) and diffusion timestep t are conditioned by mapping each scalar to sine-cosine features, then applying a 2-layer MLP. The combined embedding is fed to an AdaLN block generating scale and shift coefficients that modulate Layer Normalization and attention layer outputs. For unlabeled data training, explicit navigation actions are omitted from the conditioning. The training objective minimizes mean-squared error between the noisy target and the predicted target: L = E[||sₚ₊₁ - F(sₚ₊₁ᵗ | sₚ, aₚ, t)||²], learning to reconstruct future states from noisy versions based on context and actions.

Given the latent encoding s₀ and navigation target s*, the task is to find an action sequence (a₀, ..., aₚ) maximizing the likelihood of reaching s*. The authors define an energy function such that its minimization corresponds to maximizing perceptual similarity while following constraints: E(s₀, a₀, ..., aₚ, sₚ) = -S(sₚ, s*) + constraint penalties. Similarity is computed by decoding states to pixels via VAE and measuring perceptual similarity (LPIPS). Constraints like "never go left then right" are encoded by constraining actions to a valid set, and indicator functions apply large penalties for violations.

The optimization problem is reformulated as a Model Predictive Control (MPC) problem, optimized using the Cross-Entropy Method (CEM), "a simple derivative-free and population-based optimization method." Alternatively, assuming access to an existing navigation policy (e.g., NoMaD), NWM can rank sampled trajectories: multiple trajectory samples are drawn from the policy, each simulated with NWM, and the one with the lowest energy (highest perceptual similarity to goal) is selected. This ranking approach yields state-of-the-art navigation performance.

Experiments use six diverse datasets: SCAND (socially compliant navigation, 8.7 hours), TartanDrive (off-road driving, 5 hours), RECON (outdoor robotics, 40 hours across 9 environments), HuRoN (social interactions, 75+ hours), GO Stanford (out-of-domain evaluation only, 25 hours), and Ego4D (large-scale egocentric, 908 hours for unlabeled training). The default model uses CDiT-XL with 1B parameters, 4-frame context, batch size 1024, trained on 8 H100 machines with 8 GPUs each. Baselines include DIAMOND (diffusion-based world model with UNet), GNM (goal-conditioned navigation), and NoMaD (diffusion policy for navigation).

Ablation studies on single-step 4-second future prediction reveal several findings. CDiT outperforms standard DiT with models up to 1B parameters while consuming less than 2x FLOPs; with equal parameters (CDiT-L vs. DiT-XL), CDiT achieves 4x faster performance with better results. Training with 4 goals leads to significantly improved prediction in all metrics compared to fewer goals. More context frames help, with single-frame models often "losing track" and producing poor predictions. Both time and action conditioning prove beneficial: time-only training performs poorly (LPIPS 0.760), while removing time causes small performance drops.

For standalone planning, the model generates 8-step trajectories using CEM with k=0.25 temporal shift, achieving competitive results: ATE 1.13 on RECON. NWM can plan while satisfying dynamic constraints (e.g., "forward-first," "left-right first"), with minor performance differences relative to unconstrained baseline and all constraints satisfied. For trajectory ranking using NoMaD with n=32 samples, NWM achieves state-of-the-art navigation performance (ATE 1.78, RPE 0.48 on RECON), with more sampled trajectories yielding better results.

Training on in-domain datasets plus unlabeled Ego4D videos (using only time-shift actions) and testing on the Stanford GO dataset (unknown environment) shows that training on unlabeled data "leads to significantly better video predictions according to all metrics". The model shows improved LPIPS: 0.658 (in-domain only) vs. 0.652 (+Ego4D) on the unknown environment. Qualitatively, the model demonstrates improved video prediction but also expectedly hallucinates paths and generates imagined environment traversals, a phenomenon the authors acknowledge as a current limitation.

We present NWM as "a scalable, data-driven approach to learning navigation policies" trained across diverse environments with the CDiT architecture, adapting flexibly to various scenarios. NWM can independently plan or rank external policies by simulating navigation outcomes, incorporating new constraints dynamically. This approach "bridges learning from video, visual navigation, and model-based planning and could potentially open the door to self-supervised systems that not only perceive but also act."

The authors identify several limitations: (1) mode collapse when applied to out-of-distribution data, where the model "slowly loses context and generates next states that resemble the training data"; (2) difficulty simulating temporal dynamics like pedestrian motion, though some cases are handled; (3) both limitations are likely solvable with longer context and more training data; (4) the current system utilizes only 3 DOF navigation actions; extending to 6 DOF and robotic arm control remain future work.