Finding correspondences across images and recovering camera poses and scene geometry are key tasks of Structure-from-Motion (SfM). The accuracy of this process is limited by the coarseness of keypoint detection. This work proposes to refine keypoint locations and camera poses by directly aligning dense neural network features, bridging the gap between feature matching and direct alignment methods. The approach operates as a post-processing step that can improve any SfM pipeline, significantly improving accuracy across various keypoint detectors and viewing conditions while maintaining scalability to large image collections.

Structure-from-Motion reconstructs 3D scenes from unordered image collections by establishing correspondences, estimating camera poses, and triangulating 3D points. Traditional feature-based SfM pipelines (such as COLMAP) detect sparse keypoints, describe them with local descriptors, and match across images. However, keypoint positions are typically estimated with limited accuracy — often at integer pixel coordinates, which introduces quantization errors that propagate through the entire reconstruction pipeline. On the other hand, direct methods align images by minimizing photometric errors, achieving sub-pixel accuracy, but they require good initialization and are sensitive to appearance changes.

The authors propose a featuremetric refinement approach that combines the strengths of both paradigms: it optimizes keypoint positions and camera poses by aligning dense learned features extracted by a convolutional neural network. Unlike photometric alignment that operates on raw pixel intensities, featuremetric alignment leverages learned feature representations that are robust to appearance changes such as illumination variation and viewpoint shift. The refinement is formulated as two complementary stages: keypoint adjustment before SfM (refining tentative correspondences) and bundle adjustment after SfM (jointly optimizing poses and structure).

Feature-based methods such as SIFT, SuperPoint, and D2-Net detect and describe keypoints that are then matched and fed into geometric verification. These methods are robust and scalable but limited in localization accuracy. Direct methods such as DSO and LSD-SLAM achieve superior accuracy by minimizing photometric reprojection error, but they are sensitive to large baselines and illumination changes. Learned feature matching approaches like SuperGlue improve matching quality but still rely on initial keypoint locations. This work is unique in that it refines the geometric estimates by optimizing in learned feature space rather than pixel space.

The core idea is to replace the traditional geometric reprojection error with a featuremetric objective that measures alignment in learned dense feature space. Given a pair of images, a CNN extracts dense feature maps F_i at multiple scales. For corresponding keypoints p_u in image i(u) and p_v in image i(v), the objective minimizes: sum of weighted norms ||F_i(u)[p_u] - F_i(v)[p_v]||, where confidence weights w_uv prevent drift while allowing track separation. The features capture both local texture details and broader contextual information, providing robustness that raw pixel intensities cannot achieve.

Before running the full SfM pipeline, tentative correspondences are refined by adjusting keypoint locations to minimize the featuremetric cost. For each track (a set of corresponding keypoints across images), the optimization jointly adjusts all keypoint positions to minimize the pairwise feature distance. This is solved via Gauss-Newton optimization on the sub-pixel keypoint coordinates. The confidence weights serve a dual purpose: they downweight unreliable correspondences and can split tracks when the optimization detects inconsistencies. This pre-refinement step ensures that the subsequent SfM pipeline starts with significantly more accurate correspondences.

After the initial SfM reconstruction, a featuremetric bundle adjustment further refines camera poses and 3D point positions. For each track, a reference feature is selected as "the observation closest to the robust mean" in feature space. All other observations are then aligned against this reference, reducing memory complexity from O(N^2) pairwise residuals to O(N) reference-based residuals. The optimization jointly adjusts camera extrinsics, intrinsics, and 3D point positions to minimize the featuremetric reprojection cost. This formulation is scalable to large scenes because the reference-based parameterization avoids the combinatorial explosion of pairwise comparisons.

Experiments are conducted on the ETH3D benchmark for triangulation and camera pose estimation. For triangulation accuracy at 1cm threshold, the method improves SIFT from 75.62% to 82.82%, SuperPoint from 75.76% to 89.33%, and D2-Net from 47.18% to 82.49%. The improvements are particularly dramatic for D2-Net, where the +35.31 percentage point improvement demonstrates the method's ability to compensate for poor initial localization. For camera pose estimation, SuperPoint's AUC at 1cm improves from 15.38% to 40.00%. The scalability is validated on a collection of 7,000 images processed in under 2 hours, demonstrating practical applicability for large-scale reconstruction.

This work demonstrates that optimizing in learned feature space rather than pixel space can significantly improve the accuracy of Structure-from-Motion. The featuremetric refinement approach — operating both before and after the SfM pipeline — bridges the gap between feature matching and direct alignment methods. The method consistently improves results across different keypoint detectors and scales to thousands of images. The authors envision that featuremetric optimization could become a standard component in reconstruction pipelines, complementing both classical and learning-based approaches to keypoint detection and matching.