This paper proposes a method for converting a single RGB-D image into a 3D photo — a multi-layered representation that enables novel view synthesis with motion parallax from a single input image. The key contribution is "a learning-based inpainting model that synthesizes new local color-and-depth content into the occluded region in a spatial context-aware manner." The method uses a Layered Depth Image (LDI) representation that stores multiple color and depth values per pixel, enabling efficient rendering with realistic motion parallax effects.

Generating 3D experiences from 2D images is a long-standing goal in computer vision. Recent advances in monocular depth estimation have made it possible to obtain depth maps from ordinary photographs, but simply warping an image using its depth map reveals disoccluded regions — areas that were hidden behind foreground objects in the original view. These disocclusion artifacts appear as "holes and stretched textures" that break the 3D illusion. The core challenge is to plausibly fill in these missing regions with both color and depth information.

Existing approaches for novel view synthesis include multi-plane images (MPI) which require "a fixed number of depth planes" and struggle with continuous depth variation, and mesh-based methods that produce "stretching artifacts at depth discontinuities." The authors propose instead a layered depth image approach with learned context-aware inpainting, which separates the scene into depth layers and synthesizes plausible content behind foreground objects.

Prior work spans three areas: single-image view synthesis methods that learn to predict novel views directly, image inpainting techniques that fill missing regions in 2D, and depth inpainting/completion methods that reconstruct missing depth. Most image inpainting methods operate in 2D and do not jointly reason about color and depth. Depth completion methods focus on sparse-to-dense depth but do not address the color synthesis needed for rendering. This work uniquely combines joint color-and-depth inpainting with spatial context awareness in a layered 3D representation.

The method constructs a Layered Depth Image (LDI) from the input RGB-D image. The process begins by detecting depth discontinuities to identify occlusion boundaries between foreground and background objects. The scene is then decomposed into multiple depth layers, where each layer contains pixels at similar depths. Foreground objects are separated from their backgrounds, and the occluded regions — areas behind foreground objects that are invisible in the input view — are marked for inpainting. This layered structure enables independent rendering of each layer with proper depth ordering during novel view synthesis.

The core of the method is a context-aware inpainting network that simultaneously synthesizes color and depth in the occluded regions. Unlike standard image inpainting that operates on the full image, this network processes local regions around each depth edge, maintaining spatial context awareness by conditioning on the surrounding visible content. The inpainting proceeds in an edge-guided manner: starting from the outermost boundary of the occluded region and progressively filling inward, ensuring each newly synthesized strip is informed by both the original visible content and previously synthesized content. This iterative edge-based synthesis naturally handles varying sizes of occluded regions without requiring fixed-size inputs.

The method is evaluated on diverse real-world images including indoor scenes, outdoor landscapes, and portraits. Comparisons against baselines including Niklaus et al.'s 3D Ken Burns method and mesh-based warping show that the proposed approach produces fewer artifacts at depth discontinuities and more plausible disoccluded content. User studies indicate that participants preferred the proposed method's results in the majority of comparisons. The method processes an image in approximately 3-5 minutes on a single GPU, generating a complete LDI representation that enables real-time rendering of novel views.

This work presents a complete pipeline for generating 3D photographs from single RGB-D images. The context-aware layered depth inpainting approach effectively synthesizes plausible color and depth content in occluded regions, producing Layered Depth Images that enable real-time rendering with convincing motion parallax. The method demonstrates robustness across diverse scene types and opens new possibilities for immersive photo viewing experiences.