We present the bilateral solver, a novel algorithm for edge-aware smoothing that combines the flexibility of optimization-based approaches with the speed of fast bilateral filtering. Our technique is capable of solving a broad class of optimization problems by working in a "bilateral space" defined by the input reference image. The bilateral solver can be applied to numerous problems including depth superresolution, stereo, colorization, and semantic segmentation. The solver produces results that are comparable to or better than state-of-the-art domain-specific techniques, while being fast enough for real-time applications. Our fast bilateral solver can process a 1-megapixel image in under 200 milliseconds.

Many computer vision tasks require some form of edge-aware smoothing — a process that smooths an image or signal while preserving sharp edges. The bilateral filter is perhaps the most well-known edge-aware filter, but it is inherently limited: it can only be applied as a fixed filter and cannot easily incorporate data-dependent constraints or regularization terms. On the other hand, optimization-based methods like CRF-based approaches can incorporate arbitrary constraints but are typically slow and require problem-specific implementations. We bridge this gap by formulating edge-aware smoothing as an optimization problem that can be solved efficiently in a bilateral space using a compact representation derived from the bilateral grid.

The bilateral solver minimizes an objective that consists of a data fidelity term and a bilateral smoothness term. The key insight is to reformulate the problem in bilateral space: instead of solving for a value at every pixel, we solve for values at vertices of a simplified bilateral grid, which has far fewer unknowns than the number of pixels. The bilateral smoothness term is defined as a quadratic form using the bistochastic normalized bilateral affinity matrix, which encodes edge-aware connections between pixels based on both spatial proximity and color similarity. This formulation allows us to express the optimization as a sparse linear system that can be solved efficiently using preconditioned conjugate gradient.

The fast bilateral solver exploits the structure of the bilateral grid to achieve computational efficiency. The bilateral grid reduces the dimensionality of the problem from the number of pixels (potentially millions) to the number of grid vertices (typically thousands). We use the simplified bilateral grid representation where the grid resolution is controlled by spatial and range parameters. The resulting sparse linear system is solved using preconditioned conjugate gradient (PCG) with a Jacobi preconditioner, typically converging in 25-50 iterations. The total computational cost is O(N) in the number of pixels, making the solver practical for megapixel images.

We demonstrate the bilateral solver on four diverse applications. For depth superresolution, we upsample low-resolution depth maps guided by high-resolution RGB images, achieving lower error than domain-specific methods. For stereo, we use the solver as a post-processing step to refine disparity maps, improving quality while adding only 50ms of processing time. For colorization, we solve for ab channels in Lab space given sparse user scribbles, producing high-quality results in real-time. For semantic segmentation, we use the solver as an alternative to dense CRF post-processing, achieving comparable IoU scores while being significantly faster.

We have presented the fast bilateral solver, a general-purpose edge-aware optimization technique that bridges the gap between fast bilateral filtering and flexible optimization-based approaches. By operating in a compact bilateral space, the solver achieves linear computational complexity while producing results that respect image edges. The bilateral solver is a versatile tool that can improve the output of many computer vision algorithms as a simple post-processing step.