Monocular object pose estimation is a long-standing problem in computer vision. Perspective-n-Points (PnP) is a classic method that solves pose from 2D-3D point correspondences, yet existing learning-based methods only learn partial correspondences and rely on non-differentiable PnP solvers that prevent true end-to-end training. In this paper, the authors present EPro-PnP, a generalized end-to-end probabilistic Perspective-n-Points framework that addresses this limitation. Rather than computing a deterministic pose — which is inherently non-differentiable — EPro-PnP outputs a probability density over the pose space SE(3), serving as a continuous analog to the categorical Softmax. The approach minimizes KL divergence between predicted and target pose distributions, efficiently computed via Adaptive Multiple Importance Sampling (AMIS), with 2D-3D coordinates and weights as fully learnable intermediate variables. EPro-PnP unifies existing PnP-based approaches and demonstrates state-of-the-art performance on the LineMOD 6DoF pose estimation benchmark and the nuScenes 3D object detection benchmark.

Pose estimation from monocular images is a fundamental task in computer vision with broad applications in robotics, autonomous driving, and augmented reality. The problem is typically subdivided into 6DoF object pose estimation — recovering the full rotation and translation of a known object — and 3D object detection — predicting 3D bounding boxes from images. Despite sharing the fundamental goal of estimating object pose, these two communities have developed largely independently with different methodological traditions: top-performing 3D detection methods employ direct 4DoF prediction with end-to-end learning, while 6DoF pose estimation is dominated by geometry-based approaches exploiting 3D object models.

Recent work has proposed interpreting PnP as a differentiable layer to enable end-to-end correspondence learning. However, existing approaches only learn portions of the correspondences — some learn 2D coordinates, others learn 3D coordinates, and yet others learn correspondence weights — while assuming the remaining components are given by prior knowledge. The fundamental challenge is that PnP solutions are inherently non-differentiable: given a set of 2D-3D correspondences, there may exist multiple valid pose solutions (pose ambiguity), and infinitesimal changes in correspondences can cause discontinuous jumps between solutions. This non-differentiability prevents direct gradient-based optimization of the end-to-end pipeline.

The core insight of this work is that while "a deterministic pose is non-differentiable," "the probability density of pose is apparently differentiable, just like categorical classification scores." EPro-PnP reinterprets the PnP output as a probability distribution over the pose space SE(3), parameterized by the learnable 2D-3D correspondences. Specifically, the weighted reprojection error is treated as the negative log-likelihood, from which a posterior pose distribution is derived via Bayes' theorem. During training, the KL divergence between the predicted distribution and a target distribution centered at the ground-truth pose is minimized. This is efficiently computed via Adaptive Multiple Importance Sampling (AMIS). The key contributions include: (1) a probabilistic PnP layer enabling general end-to-end pose estimation with fully learnable 2D-3D correspondences; (2) state-of-the-art 6DoF performance on LineMOD; and (3) a novel deformable correspondence network for 3D object detection that learns correspondences entirely from scratch.

Geometry-based object pose estimation methods exploit points, edges, or dense representations subject to perspective projection constraints. These can be divided into sparse keypoint methods — such as BB8 locating bounding box corners, and PVNet using farthest point sampling — and dense correspondence methods that predict pixel-wise 3D coordinate maps. Most follow a two-stage strategy where intermediate 2D-3D correspondences are learned with surrogate losses, then a separate PnP solver or RANSAC procedure recovers the pose. This decoupled training is suboptimal because the surrogate loss (e.g., coordinate regression loss) does not directly correlate with the final pose accuracy, leading to a gap between training objective and evaluation metric.

End-to-end correspondence learning methods attempt to backpropagate pose gradients through the PnP operation to the intermediate representations. Prior approaches include DSAC, which uses a differentiable RANSAC for hypothesis scoring; BPnP, which differentiates through the PnP solution via implicit differentiation of the KKT conditions; and methods that learn differentiable 3D point locations or 2D keypoint coordinates. However, these methods are typically coupled with surrogate regularization losses because deterministic pose alone lacks differentiability guarantees. Theoretically, implicit differentiation approaches are related to Laplace approximation, which assumes a locally normal posterior and fails for multi-modal distributions arising from pose ambiguity.

Probabilistic deep learning methods account for both epistemic uncertainty (model uncertainty) and aleatoric uncertainty (data uncertainty). In classification, categorical Softmax provides a smooth differentiable approximation to the argmax operation, transforming discrete logits into a probability distribution. For continuous variables, tractable parametric distributions such as Gaussian or mixture models are commonly used to capture prediction uncertainty. This paper contributes a unique perspective: "backpropagating a complicated continuous distribution derived from a nested optimization layer (the PnP layer), essentially making it a continuous counterpart of Softmax." This connection to Softmax is not merely an analogy but a rigorous mathematical parallel: just as Softmax converts logits into class probabilities, EPro-PnP converts reprojection errors into pose probabilities.

The goal is to predict N corresponding point pairs, each consisting of a 3D object coordinate, a 2D image coordinate, and a 2D correspondence weight. The classic PnP formulation finds the pose that minimizes the cumulative squared weighted reprojection error. The key insight of EPro-PnP is to treat this cumulative reprojection error as a negative log-likelihood function. Via Bayes' theorem with an uninformative prior, the error function naturally induces a posterior probability distribution over the pose space SE(3). This probabilistic reinterpretation transforms the non-differentiable argmin operation into a differentiable distribution over poses, enabling gradient flow through the entire pipeline.

Rather than backpropagating through a single non-differentiable pose solution, EPro-PnP minimizes the KL divergence between the predicted pose distribution and a target distribution. For training, the target distribution is a Dirac-like distribution centered at the ground-truth pose. Expanding the KL divergence yields a loss with two essential terms: (1) the reprojection error evaluated at the ground-truth pose, and (2) a log-partition function (normalization constant) computed over the entire predicted pose distribution. The first term alone is insufficient for learning without strict regularization — correspondences could collapse to a single point without preserving pose discrimination. The second term (normalization factor) is the crucial ingredient: it acts as an implicit regularizer ensuring that the predicted distribution is peaked around the correct pose, providing the discriminative signal necessary for learning correspondences from scratch.

This formulation stands in sharp contrast with implicit differentiation methods, which approximate the posterior locally via Laplace approximation. The Laplace approximation assumes a unimodal Gaussian posterior around the optimal pose, which fails catastrophically for objects with symmetry or self-similar appearances that induce multi-modal pose distributions. EPro-PnP's KL-based formulation makes no assumptions about the shape of the posterior and can naturally handle arbitrary distributions, including multi-modal ones arising from symmetric objects. This theoretical advantage directly translates to practical robustness: symmetric objects in LineMOD (e.g., eggbox, glue) and nuScenes (e.g., traffic cones, barriers) are handled without special treatment.

The log-partition function in the KL loss requires integration over the entire SE(3) pose space, which has no closed-form solution. EPro-PnP solves this via Adaptive Multiple Importance Sampling (AMIS), an iterative Monte Carlo method that progressively adapts its proposal distributions to the integrand. The procedure starts by estimating the mode and covariance of the predicted pose distribution using an efficient PnP solver, then iteratively refines the sampling distribution over T=4 iterations with K'=128 samples per iteration. The proposal distributions are carefully chosen for each component of the pose: multivariate t-distributions for position (providing robust heavy tails), von Mises mixtures for 1D yaw orientation (capturing rotational ambiguity), and angular central Gaussian distributions on the unit hypersphere for 3D quaternion orientation.

Analyzing the gradient of the KL loss reveals important structural insights. The gradient decomposes into the reprojection error gradient at the ground-truth pose minus the expected gradient over the predicted pose distribution. For the correspondence weights specifically, the gradient contains two competing terms: a negative-signed term penalizing high uncertainty (large reprojection error at the target pose), and a positive-signed term emphasizing sensitive correspondences that provide strong pose discrimination. This dual mechanism automatically balances uncertainty reduction and discrimination enhancement, a property absent in existing methods that only consider the uncertainty term and therefore lack discriminative ability when learning from scratch.

For inference, recovering the exact pose requires iterative PnP solvers computing first and second-order derivatives of the reprojection error. To facilitate this optimization, EPro-PnP introduces a local derivative regularization that encourages the Levenberg-Marquardt optimization step to converge toward the true pose. This is implemented by applying smooth L1 loss for position derivatives and cosine similarity for orientation derivatives. This regularization is optional but beneficial: it does not alter the probabilistic training framework but improves the numerical conditioning of the iterative solver at test time, yielding consistent improvements of 1-2 percentage points across benchmarks.

For 6DoF pose estimation, the authors modify the existing CDPN framework by expanding its confidence maps into two-channel XY weights with spatial Softmax normalization and a dynamic global weight scaling factor. The spatial Softmax is vital for stable training: it translates absolute weight values into relative measurements across all correspondence points, preventing weight collapse. The global scaling factors control the concentration of the predicted pose distribution — larger factors produce sharper distributions indicating higher confidence. The network can be trained with KL loss alone for decent performance, but adding coordinate regression as auxiliary loss introduces geometric knowledge from 3D models. Further regularization loss provides additional improvements.

For 3D object detection, the authors propose a novel deformable correspondence network that learns entire 2D-3D correspondences from scratch, extending the FCOS3D detection framework. Deformable attention layers sample key-value pairs from dense feature maps, projecting values into point-wise features for correspondence prediction while aggregating them into object-level features for auxiliary tasks. Point features predict 3D coordinates and weights (normalized by Softmax), while object features predict 3D detection score, weight scaling, bounding box size, and optional properties such as velocity. The entire network is trained with KL divergence loss, optionally combined with regularization loss. This architecture demonstrates EPro-PnP's versatility: it can learn all correspondence components from raw data without any geometric priors.

On the LineMOD benchmark (13 object sequences with ~1.2K images each, annotated with 6DoF poses), comprehensive ablation studies reveal the contribution of each component. The pure PnP baseline without end-to-end learning drops 17.46 points from the CDPN-Full baseline, confirming the importance of end-to-end training. Adding EPro-PnP with KL loss alone recovers 13.84 of those points, already surpassing CDPN-Full. Regularization loss adds 1.88 points, and initialization from pretrained CDPN contributes an additional 5.46 points. A striking finding is that "EPro-PnP manages to learn the coordinates from scratch, even outperforming CDPN without translation head (79.46 vs. 74.54)", demonstrating the power of the end-to-end probabilistic framework.

Further ablation reveals that end-to-end weight learning impacts performance more than coordinate learning (-8.69 vs. -3.08 when disabled), indicating that correspondence weighting is the most critical learnable component. The spatial Softmax normalization is found to be essential — without it, training diverges entirely. Examining the balance between uncertainty and discrimination, the authors find that the uncertainty term alone shows strong performance with intermediate supervision, but the discrimination term is indispensable for learning from scratch. This validates the theoretical analysis in Section 3.3: existing methods that only model uncertainty can function when auxiliary losses provide geometric guidance, but EPro-PnP's dual-term gradient is necessary for truly end-to-end learning.

On LineMOD state-of-the-art comparison, EPro-PnP reaches 98.54% at the 5-degree-5cm metric, which is comparable to top refinement-based methods while being more straightforward in pipeline design. On the nuScenes 3D object detection benchmark, the deformable correspondence network with EPro-PnP achieves NDS 0.425 versus the FCOS3D baseline of 0.372, a substantial improvement of 5.3 NDS points. With test-time augmentation, the method reaches NDS 0.439 versus the previous best of 0.422. The improvements are particularly pronounced in pose-related metrics: mATE (average translation error) and mAOE (average orientation error), directly demonstrating PnP's advantage in geometric reasoning. Notably, the multimodal pose distributions naturally capture orientation ambiguity for symmetric objects (barriers, traffic cones) without requiring discrete binning or mixture models.

Qualitative analysis of learned correspondence maps reveals that EPro-PnP's correspondence maps show less fine-grained detail than CDPN due to higher uncertainty around sharp edges, yet achieve comparable pose accuracy (79.46 vs. 79.96). When initialized from pretrained CDPN, the inherited geometric profiles confine weights to the foreground region, achieving the best overall performance. Visualized pose distributions on nuScenes clearly capture multimodal orientations for symmetric objects — barriers and traffic cones exhibit bimodal or uniform rotational distributions, while uncertain observations (e.g., heavily occluded pedestrians) show broader, more diffuse distributions. These visualizations provide compelling evidence that the probabilistic formulation captures meaningful geometric uncertainty.

EPro-PnP translates the non-differentiable deterministic PnP into a differentiable probabilistic layer, enabling unprecedented flexibility in end-to-end 2D-3D correspondence learning. The theoretical connections to prior work — including implicit differentiation as Laplace approximation, uncertainty-only gradients as special cases of the full KL gradient, and the Softmax analogy for continuous pose distributions — are thoroughly discussed both theoretically and experimentally. The approach can either be inserted into existing PnP-based networks to improve their end-to-end capability, or inspire entirely novel architectures like the deformable correspondence network for 3D detection. The underlying principles generalize beyond pose estimation to any declarative network with nested optimization layers, suggesting broad applicability to geometric reasoning tasks in computer vision.

Looking forward, several promising directions emerge. The AMIS-based Monte Carlo estimation could potentially be replaced with more efficient sampling strategies or even learned proposal distributions. The deformable correspondence framework could be extended to multi-view and temporal settings, where pose distributions across frames could be fused coherently. Furthermore, the probabilistic output of EPro-PnP naturally interfaces with downstream uncertainty-aware planning systems in autonomous driving and robotics, where knowing not just the most likely pose but also its confidence is critical for safe decision-making. The success of EPro-PnP across both 6DoF estimation and 3D detection suggests that bridging the gap between geometric reasoning and end-to-end learning remains a fertile research direction.