Conventional autofocus systems in cameras assume a single global focal plane, which means that for scenes with objects at varying depths, only one depth layer can be in sharp focus at a time. This paper introduces Spatially-Varying Autofocus, a method that enables per-pixel focal control across the image sensor. By leveraging a programmable spatial light modulator (SLM) in the optical path, we can impose spatially-varying phase shifts on the incoming wavefront, effectively assigning different focal lengths to different image regions. Combined with a learned optimization framework, the system determines the optimal per-pixel phase pattern to maximize sharpness across all depth layers simultaneously. We demonstrate that our approach produces all-in-focus images from a single capture, outperforming traditional autofocus and existing computational photography methods on both synthetic and real scenes.

Autofocus is one of the most fundamental functions in modern cameras. Despite decades of advancement, current autofocus systems fundamentally operate under the assumption of a single focal plane. In multi-depth scenes, photographers must choose which depth to focus on, leaving other regions blurred. Focus stacking addresses this by capturing multiple images at different focal distances and merging them, but it requires multiple exposures and is susceptible to motion artifacts. Computational depth-of-field extension methods use coded apertures or phase masks to encode depth information, but they rely on post-capture deconvolution that introduces noise and artifacts.

We propose Spatially-Varying Autofocus, which replaces the global focus mechanism with a spatially-adaptive one. The key hardware enabler is a spatial light modulator (SLM) placed in the pupil plane of the optical system. By programming pixel-wise phase patterns on the SLM, we effectively create a spatially-varying lens that focuses different image regions to different depths simultaneously. A learned optimization algorithm takes a depth estimate of the scene and computes the optimal phase pattern that maximizes the modulation transfer function (MTF) across all spatial frequencies and depth layers.

Traditional autofocus methods — contrast-detection AF (CDAF) and phase-detection AF (PDAF) — determine the optimal position for a single movable lens element. Multi-focus imaging captures a focal stack and merges in post-processing, requiring temporal multiplexing that precludes dynamic scenes. Wavefront coding with phase masks extends depth of field but trades off spatial resolution for depth invariance. Computational photography approaches using coded apertures or light field cameras capture richer information, yet face resolution-versus-angular-sampling tradeoffs. Recent neural optics methods co-optimize optical elements with reconstruction networks but typically design static optical elements rather than adaptive ones. Our approach uniquely combines adaptive wavefront modulation with learned per-scene optimization.

Our optical system places a phase-only spatial light modulator at the pupil plane of a standard imaging lens. The SLM modulates the phase of the wavefront passing through each pupil location without affecting amplitude. For a scene point at depth d imaged through pupil location (u, v) with SLM phase φ(u, v), the resulting point spread function (PSF) at image point (x, y) is given by the Fourier transform of the generalized pupil function: P(u, v) = A(u, v) · exp(j[W(u, v; d) + φ(u, v)]), where W is the defocus aberration dependent on depth and A is the aperture function. By choosing φ(u, v) to locally cancel the defocus aberration W for different depths at different image locations, we achieve spatially-varying focus.

Given a depth map of the scene (obtained from a depth sensor or monocular estimation), we formulate the phase pattern optimization as maximizing the integrated MTF across all image regions. The objective is: max_φ ∑_(x,y) MTF(x, y; d(x,y), φ), where d(x,y) is the depth at pixel (x, y). We solve this using a differentiable optical forward model combined with gradient-based optimization. The forward model simulates image formation through the SLM, and gradients with respect to φ are computed via automatic differentiation. The entire pipeline — from depth input to optimized phase pattern — runs in under 50 milliseconds on a modern GPU, enabling real-time operation.

We evaluate our method on both synthetic scenes with ground-truth depth and real captured scenes using a prototype with a liquid-crystal SLM. On synthetic benchmarks, our approach achieves significantly higher PSNR and SSIM compared to fixed-focus, focus stacking (with 3 and 5 captures), and coded aperture methods. On real scenes, we demonstrate all-in-focus results from single captures of indoor tabletop scenes with depth ranges spanning 0.3m to 2m. The method handles both static and dynamic scenes, with the single-capture advantage being particularly pronounced for the latter. Ablation studies confirm the importance of the learned optimization over hand-crafted phase patterns.

Spatially-Varying Autofocus represents a paradigm shift from global to per-pixel focal control. By combining an SLM-based adaptive optical element with differentiable optimization, we demonstrate that all-in-focus imaging from a single capture is achievable in real-time. This opens up new possibilities for computational cameras that actively co-design optics and algorithms on a per-scene basis. Future work includes miniaturizing the SLM module for integration into mobile devices and extending the approach to video capture with temporal consistency.