Modern autonomous driving systems are typically built in a modular fashion, where perception, prediction, and planning are designed and optimized independently. While this divide-and-conquer strategy simplifies development, it suffers from accumulated errors across stages and lacks holistic reasoning about the driving scene. An alternative is end-to-end learning, which directly maps sensor inputs to planning outputs but often sacrifices interpretability and struggles with complex multi-agent interactions.

In this work, the authors present UniAD, a Unified Autonomous Driving framework that incorporates full-stack driving tasks in one network. The key philosophy is planning-oriented: all preceding tasks — tracking, mapping, motion forecasting, and occupancy prediction — are designed to serve and benefit the ultimate planning objective. UniAD connects these nodes through query-based interfaces within transformer decoders, enabling effective inter-task feature interaction. Extensive experiments on the nuScenes benchmark demonstrate that UniAD achieves state-of-the-art performance on all tasks simultaneously, and that the joint design substantially improves planning safety and accuracy compared to both modular pipelines and naive multi-task approaches.

Autonomous driving has long been approached through a sequential pipeline: first perceive the environment via detection, tracking, and mapping; then predict the future behavior of surrounding agents; and finally plan a safe trajectory for the ego vehicle. This modular design brings clear engineering advantages — each component can be developed, tested, and improved independently. However, errors from upstream modules inevitably propagate downstream, and information loss at module boundaries prevents the planner from fully exploiting rich scene context. For instance, a missed detection in perception leads to a missing prediction, which in turn causes a dangerous planning decision.

Recent works have explored multi-task learning for autonomous driving, where a shared backbone extracts features consumed by multiple task-specific heads. While this improves efficiency and enables some implicit feature sharing, the tasks are often treated as independent objectives with separate loss functions, lacking explicit coordination. Such approaches may suffer from negative transfer — where optimizing one task degrades another — and fail to capture the causal dependencies between perception, prediction, and planning. The fundamental question remains: how should driving tasks be organized so that they collectively contribute to safe and effective planning?

The authors argue that the answer lies in a planning-oriented philosophy: every task in the driving stack should be designed with the ultimate goal of planning in mind. They introduce UniAD, which connects five key nodes — detection and tracking (TrackFormer), online mapping (MapFormer), multi-agent motion forecasting (MotionFormer), occupancy prediction (OccFormer), and ego planning (Planner) — through a unified query-based design. Each module communicates with downstream modules via learned queries that carry structured, task-specific representations, ensuring that intermediate outputs are not only accurate on their own metrics but also maximally informative for the planning task.

3D object detection from multi-camera inputs has advanced rapidly with methods like BEVFormer and DETR3D, which project image features into bird's-eye-view (BEV) representations. For multi-object tracking (MOT), traditional approaches rely on detection followed by association heuristics, introducing non-differentiable post-processing that breaks end-to-end training. Recent query-based trackers such as MOTR and TrackFormer perform joint detection and tracking using track queries that propagate across frames, eliminating the need for hand-crafted association rules. However, these methods are designed as standalone modules and do not consider how tracking representations feed into downstream prediction and planning.

Motion forecasting aims to predict future trajectories of surrounding agents. Early methods use recurrent networks or graph neural networks to model temporal and social interactions. More recent approaches adopt goal-conditioned prediction, where an agent's future trajectory is anchored to a set of possible goal locations. Scene-centric representations that reason jointly about all agents have shown advantages over agent-centric formulations that independently predict each agent without considering inter-agent coordination. Crucially, most motion forecasting methods assume perfect perception inputs (ground-truth detections and tracks), a condition rarely met in real-world deployment.

End-to-end autonomous driving methods attempt to learn a direct mapping from sensor inputs to driving actions. Pioneering works like ALVINN and more recent imitation learning approaches demonstrate feasibility, but typically lack intermediate representations, making failure diagnosis difficult. ST-P3 and PnPNet incorporate some intermediate tasks but treat them with simple concatenation or loose coupling, without explicitly modeling the task dependency graph. UniAD distinguishes itself by systematically designing inter-task connections through query-based interfaces, where each module's output queries directly serve as input to the next.

The overall architecture of UniAD follows a sequential, query-based paradigm. Multi-camera images are first processed by a shared image backbone and BEV encoder to produce a unified bird's-eye-view (BEV) feature map. This BEV representation serves as the common foundation for all downstream modules. The five task modules — TrackFormer, MapFormer, MotionFormer, OccFormer, and Planner — are arranged in a directed acyclic graph (DAG) that respects the natural dependency order of driving tasks. Each module is a transformer decoder that takes task-specific queries and cross-attends to the BEV features and outputs of preceding modules.

The core innovation is the query-based inter-task communication mechanism. In UniAD, the output queries of one module serve as the input key/value for the next module's cross-attention layers. For example, agent queries from TrackFormer carry per-object representations into MotionFormer, where they interact with map queries from MapFormer to produce motion-aware forecasts. This design ensures that information flows in a structured, task-aware manner rather than through generic shared features, enabling each downstream module to selectively attend to the most relevant upstream representations.

TrackFormer performs joint 3D object detection and multi-object tracking in a unified transformer decoder. It maintains two types of queries: detection queries for discovering new objects entering the scene, and track queries that propagate information about previously detected objects across frames. At each time step, detection queries attend to the current BEV features to localize new agents, while track queries carry forward the representation of existing agents and update their states through cross-attention with the latest BEV features.

A critical design choice is that TrackFormer eliminates all non-differentiable post-processing typically used in tracking pipelines, such as Hungarian matching at inference time or NMS-based filtering. Instead, the association between detections and tracks is implicitly learned through the attention mechanism. During training, bipartite matching is used to assign ground-truth objects to queries, and a shared query matching strategy ensures consistent instance identities across all downstream modules. This end-to-end differentiable design is essential for enabling gradient flow from the planning loss all the way back to the perception module.

MapFormer performs online BEV semantic map construction through panoptic segmentation of road elements. It uses sparse map queries, where each query represents a semantic class of road structure — such as lane dividers, road boundaries, and pedestrian crossings. These queries attend to the BEV features and produce per-class segmentation masks and embedding vectors that encode the local road topology. Unlike offline HD map approaches, MapFormer constructs the map representation on-the-fly from sensor data, removing the dependency on pre-built maps.

The map queries carry rich structural information about the driving environment that is critical for downstream motion forecasting. Agent behavior is heavily influenced by road geometry — vehicles follow lanes, respect boundaries, and yield at crossings. By passing map query embeddings into MotionFormer's cross-attention layers, the framework enables the motion predictor to reason about agent-map interactions explicitly, rather than relying on the motion model to implicitly learn road constraints from BEV features alone. This structured communication between mapping and prediction is a key differentiator from prior multi-task approaches.

MotionFormer is the central prediction module that forecasts multimodal future trajectories for all detected agents in a scene-centric manner. It receives agent queries from TrackFormer and map queries from MapFormer, and models three types of interactions through specialized attention mechanisms: agent-agent interaction (how agents influence each other's future motion), agent-map interaction (how road structure constrains trajectories), and agent-goal interaction (how intended destinations shape future paths). The output is a set of multimodal trajectory predictions with associated probability scores for each agent.

A distinctive feature of MotionFormer is its motion query design, which integrates four types of positional knowledge through sinusoidal positional encoding and MLPs: (1) scene-level anchors that provide a coarse spatial prior for possible future locations; (2) agent-level anchors derived from each agent's current state; (3) the agent's current position as a reference point; and (4) predicted goal points that represent likely destinations. These four components are fused to form rich motion queries that encode both the spatial context and the intentional direction of each agent, enabling more accurate multimodal trajectory prediction.

Unlike agent-centric approaches that independently predict each agent's trajectory without awareness of other agents' plans, MotionFormer adopts a scene-centric formulation where all agents' motion queries attend to each other through self-attention layers. This allows the model to capture coordinated behaviors — for example, when one vehicle yields, the model can predict that the other vehicle will proceed. The agent-map cross-attention further constrains predictions to be physically plausible by anchoring trajectories to the road structure encoded by MapFormer, effectively preventing predictions that violate lane boundaries or road geometry.

OccFormer predicts multi-step future occupancy grids that represent the spatial regions likely to be occupied by agents at each future time step. Unlike conventional occupancy prediction methods that produce anonymous occupancy maps without distinguishing which agent occupies which cell, OccFormer incorporates a pixel-agent interaction mechanism that preserves agent identity within the occupancy representation. This is achieved by having BEV pixel features cross-attend to agent queries from MotionFormer, with occupancy-guided masks restricting the pixel-to-agent correspondence based on predicted motion trajectories.

The occupancy prediction serves as a dense, spatiotemporal safety map for the downstream planner. While trajectory predictions from MotionFormer provide sparse future locations of agents, they may not fully capture the spatial extent of agents or account for prediction uncertainty. The occupancy grid complements trajectory prediction by providing a dense representation of future danger zones, enabling the planner to reason about collision risks in a more comprehensive manner. The multi-step nature of the prediction allows the planner to anticipate and avoid occupied regions across the entire planning horizon.

The Planner module generates the final ego-vehicle trajectory by leveraging all upstream representations. It uses an ego-vehicle query that attends to the BEV features, agent queries from MotionFormer, and occupancy predictions from OccFormer through cross-attention layers. The planner outputs a sequence of future waypoints for the ego vehicle over a predefined planning horizon. Crucially, the planning module does not operate in isolation — it benefits from the rich, structured scene understanding accumulated through the preceding modules.

To address perception uncertainty and ensure kinematically feasible trajectories, the Planner incorporates a post-optimization step using Newton's method. The raw trajectory output from the neural network may violate kinematic constraints such as maximum jerk, curvature, and acceleration limits. The optimization smooths the planned trajectory by minimizing a cost function that balances trajectory smoothness, collision avoidance (guided by OccFormer's predictions), and proximity to the neural network's initial output. Additionally, the ground-truth trajectories used for training supervision are pre-smoothed to enforce kinematic feasibility, preventing the model from learning physically impossible motions.

UniAD adopts a two-stage training strategy. In the first stage, the perception modules (TrackFormer and MapFormer) are jointly trained for 6 epochs to establish reliable detection, tracking, and mapping capabilities. In the second stage, all five modules are trained end-to-end for 20 epochs, allowing gradients from the planning loss to flow through the entire pipeline. This curriculum-style training is motivated by the observation that training all modules from scratch simultaneously leads to unstable optimization, as the prediction and planning modules require reasonable perception inputs to learn meaningful patterns.

All experiments are conducted on the nuScenes dataset, a large-scale autonomous driving benchmark containing 1000 driving scenes with 6 surround-view cameras, LiDAR, and comprehensive 3D annotations. UniAD uses only camera inputs (no LiDAR at inference), making it a vision-only approach. The framework is evaluated on five tasks simultaneously: 3D detection and tracking, online mapping, motion forecasting, occupancy prediction, and planning. Comparisons are made against both specialized single-task state-of-the-art methods and recent multi-task or end-to-end driving approaches including ST-P3, PnPNet, and various standalone baselines.

For tracking, UniAD achieves 0.359 AMOTA on the nuScenes validation set, substantially outperforming end-to-end baselines. The model produces 906 ID switches, demonstrating competitive identity preservation. For online mapping, the panoptic segmentation quality shows that MapFormer effectively learns road structure from camera-only inputs. Notably, the joint training of tracking and mapping with downstream tasks does not degrade their individual performance — in fact, the end-to-end fine-tuning stage slightly improves perception metrics, suggesting beneficial gradient signals from prediction and planning that help perception learn more task-relevant features.

The most striking results come from motion forecasting and planning. UniAD achieves 0.71m minADE and 1.02m minFDE for motion prediction, representing a 38.3% improvement over PnPNet. For the critical planning task, UniAD delivers 1.03m average L2 displacement error and 0.31% collision rate, achieving a 51.2% reduction in L2 error compared to ST-P3. These results demonstrate that the planning-oriented design philosophy yields substantial and measurable benefits — the unified framework does not merely match specialized methods but significantly surpasses them, particularly on the downstream tasks that directly impact driving safety.

Comprehensive ablation studies validate the contribution of each component. Removing the query-based inter-task connections and replacing them with simple feature concatenation leads to significant performance drops across all tasks, confirming that structured query communication is superior to generic feature sharing. Disabling individual modules (e.g., removing MapFormer from the pipeline) degrades not only the removed task but also downstream tasks, demonstrating the interdependency captured by the unified design. The motion prediction module shows the most sensitivity to upstream quality — removing tracking input from MotionFormer increases minADE by 23%.

A particularly revealing ablation examines the impact of occupancy prediction on planning. When OccFormer is removed, the planning collision rate increases from 0.31% to 0.78%, more than doubling the risk of collision. This confirms the hypothesis that dense occupancy prediction provides critical safety information that sparse trajectory prediction alone cannot offer. Furthermore, the non-linear optimization post-processing reduces the collision rate by an additional 15% and significantly improves trajectory smoothness, validating the importance of combining learned predictions with physics-based refinement.

The authors also explore alternative task orderings within the DAG structure. Placing motion forecasting before mapping, or removing the explicit dependency between tracking and motion prediction, consistently leads to worse planning outcomes. This supports the claim that the chosen task ordering — perception first, then prediction, then planning — reflects genuine causal dependencies in the driving domain, and that violating this natural ordering disrupts the information flow needed for safe planning. The results suggest that task topology design is as important as individual module architecture.

This paper presents UniAD, a unified autonomous driving framework that systematically connects perception, prediction, and planning through query-based transformer interfaces. The core contribution is the planning-oriented design philosophy, which ensures that every intermediate task is optimized not in isolation but in service of the final planning objective. Through extensive experiments on nuScenes, the authors demonstrate that this approach achieves state-of-the-art results across all five evaluated tasks and substantially improves planning accuracy and safety compared to both modular and end-to-end alternatives.

The results reveal that task coordination through structured query interfaces outperforms both isolated task optimization and naive multi-task learning with shared backbones. The planning-oriented framework demonstrates that designing upstream tasks with the downstream objective in mind leads to representations that are inherently more useful for the final goal. Looking ahead, UniAD opens several promising directions: incorporating temporal BEV features for longer-horizon prediction, integrating additional sensing modalities like LiDAR, and extending the framework to handle more complex urban scenarios with diverse agent types. The authors believe this work takes a meaningful step toward truly integrated autonomous driving systems.

In summary, UniAD validates a fundamental insight: in complex, safety-critical systems like autonomous driving, the whole is greater than the sum of its parts. Optimizing tasks jointly through principled inter-task communication yields compound benefits that cannot be achieved by improving individual components in isolation. The query-based transformer architecture provides an elegant and scalable mechanism for implementing this vision, and the consistent improvements across all metrics demonstrate its effectiveness. As the field moves toward real-world deployment, frameworks that unify perception, prediction, and planning under a coherent objective will become increasingly essential.