In this paper, we develop Position Embedding Transformation (PETR) for multi-view 3D object detection. PETR encodes the position information of 3D coordinates into image features, producing 3D position-aware features. Object queries can then perceive the 3D scene by interacting with the 3D position-aware features via attention mechanism. PETR achieves state-of-the-art performance on the nuScenes benchmark, reaching 38.3% NDS and 31.3% mAP without bells and whistles. Furthermore, PETR provides a simple and elegant framework for multi-view 3D perception that can be easily extended to temporal fusion and multi-task learning.

Multi-view 3D object detection aims to detect 3D objects from multiple camera images, which is gaining importance as a cost-effective alternative to LiDAR-based systems in autonomous driving. The key challenge lies in how to transform 2D image features into 3D-aware representations that support accurate 3D bounding box prediction. Existing approaches can be broadly categorized into two paradigms: explicit depth estimation methods that first predict depth maps and construct 3D features through view transformation (e.g., BEVDet, LSS), and implicit 3D reasoning methods that leverage attention mechanisms to reason about 3D geometry without explicit depth prediction (e.g., DETR3D).

DETR3D pioneered the use of learnable 3D object queries that interact with 2D image features through cross-attention. However, DETR3D projects 3D reference points onto 2D images and samples features at projected locations, which relies on accurate 3D-to-2D projection and suffers from quantization errors when the projected point falls between pixels. Moreover, this projection-based sampling only attends to a single point per view, missing the broader spatial context around the projected location. We propose a fundamentally different approach: instead of projecting 3D queries to 2D, we embed 3D position information directly into the 2D image features, allowing object queries to perceive 3D geometry through standard attention.

Camera-based 3D detection has evolved from monocular methods like FCOS3D and PGD that predict 3D boxes from single images, to multi-view methods that exploit the geometric constraints from multiple camera views. BEV-based methods construct bird's-eye-view representations through lift-splat-shoot (LSS) or depth estimation and back-projection, providing explicit 3D spatial structure. Query-based methods, led by DETR3D, use learnable queries to directly predict 3D boxes without constructing explicit BEV features. PETR belongs to the query-based family but differs in how 3D position information is incorporated: through position embedding transformation rather than point projection.

The core of PETR is the 3D Position Embedding (3D PE) generation. For each pixel in the multi-view images, we generate a set of 3D coordinates by back-projecting the pixel along its camera ray at multiple predefined depths. Specifically, for a pixel at location (u, v) in camera view i, we compute its 3D world coordinates at D discrete depth values using the known camera intrinsics and extrinsics. These 3D coordinates are then transformed through a small MLP network to produce the 3D position embedding, which is added to the 2D image feature at that pixel location. The resulting 3D position-aware features encode both the visual appearance from the 2D backbone and the 3D geometric information from the position embedding.

With the 3D position-aware features, we employ a standard transformer decoder where learnable object queries interact with the 3D position-aware features through cross-attention. Unlike DETR3D which samples features at specific projected locations, PETR's attention mechanism allows each object query to attend to all positions across all views simultaneously, weighted by the compatibility between the query and the 3D position-aware features. This global attention naturally handles cross-view feature aggregation without explicit view selection or fusion modules. The decoder outputs are passed through prediction heads for 3D bounding box regression and classification, trained with Hungarian matching following the DETR paradigm.

To improve detection performance, we introduce several auxiliary designs. First, a 3D coordinate generator that uses camera frustum-based sampling to create denser 3D coordinate grids at closer distances, where objects are more likely to appear. Second, feature-guided position encoding (FPE) that conditions the position embedding on the image features themselves, allowing the network to adaptively weight depth hypotheses based on visual cues. Third, we apply auxiliary supervision at intermediate decoder layers to accelerate convergence. These improvements collectively boost performance by 2.1 NDS over the base PETR model.

We evaluate PETR on the nuScenes benchmark, the standard testbed for multi-view 3D detection. Using ResNet-50 backbone, PETR achieves 38.3% NDS and 31.3% mAP on the validation set, outperforming DETR3D (34.9% NDS, 30.3% mAP) and FCOS3D (37.2% NDS, 29.5% mAP). With ResNet-101 and CBGS, PETR further achieves 44.1% NDS and 37.0% mAP. Notably, PETR significantly surpasses DETR3D in localization-related metrics: mATE (translation error) improves by 5.2%, confirming that 3D position-aware features provide superior spatial reasoning compared to point-projection-based sampling.

Ablation studies validate the key design choices. Replacing 3D position embedding with standard 2D positional encoding decreases NDS by 4.7 points, confirming the critical role of 3D spatial information. Reducing the number of depth bins from 64 to 16 decreases NDS by 1.3 points, showing that finer depth discretization is beneficial. Using feature-guided position encoding (FPE) instead of fixed position embedding improves NDS by 0.8 points, demonstrating the value of conditioning position encoding on visual features. The simplicity of PETR's architecture enables fast inference at 8.2 FPS, comparable to DETR3D and faster than most BEV-based methods.

We have presented PETR, a simple and effective framework for multi-view 3D object detection that transforms 2D image features into 3D position-aware representations through position embedding transformation. By encoding 3D coordinates directly into image features, PETR enables object queries to reason about 3D geometry through standard attention mechanisms without relying on explicit depth estimation or point projection. State-of-the-art results on nuScenes validate the effectiveness of our approach. The simplicity and modularity of PETR make it a promising foundation for future research in camera-based 3D perception, including temporal fusion (PETRv2), occupancy prediction, and multi-task autonomous driving systems.