We introduce a real-time, high-resolution background matting method that operates at 4K resolution at 30fps and HD at 60fps on a modern GPU. Our method requires an additional captured background image, which is typically available in video conferencing and virtual production scenarios. We propose a two-stage architecture: a base network that processes the entire image at low resolution to produce a coarse alpha matte, foreground, and an error map, followed by a refinement network that selectively processes only high-resolution patches identified by the error map. This design enables real-time performance at very high resolutions while maintaining matting quality comparable to or exceeding offline methods.

Image matting refers to the problem of estimating the alpha matte and foreground color of a subject from an input image. It is a fundamental operation in image and video editing, with applications in virtual backgrounds for video conferencing, film production, and augmented reality. Traditional methods rely on user-supplied trimaps, which are impractical for real-time applications. Recent deep learning approaches achieve impressive quality but are computationally expensive, often requiring seconds per frame, making them unsuitable for video.

Background matting is a special case where an additional clean background image is available. This extra information provides a strong signal for foreground-background separation, as the difference between the captured image and the background reveals the foreground region. Our prior work, Background Matting V1, demonstrated this concept but operated at low resolution and could not achieve real-time performance. In this work, we propose a fundamentally different architecture that achieves both real-time speed and high resolution through a novel selective refinement strategy.

Classical matting methods such as Bayesian matting and KNN matting require trimaps or scribbles as user input. Deep learning methods like Deep Image Matting and Context-Aware Matting achieve superior quality but still require trimaps and operate at low speed. MODNet proposes trimap-free matting but at limited resolution. Green screen matting in film production uses a uniform colored background for easy chroma keying, but requires controlled studio environments. Our approach takes the middle ground: requiring only a casually captured background image, which is far easier to obtain than a trimap or green screen, while achieving real-time high-resolution performance.

The base network takes as input the source image and captured background, both downsampled to low resolution (e.g., 512x288). It produces three outputs: a coarse alpha matte, a coarse foreground color estimation, and an error map that predicts where the coarse result is likely to be inaccurate. The architecture uses a ResNet-based encoder-decoder with skip connections. The error map is trained with L1 loss between predicted alpha and ground truth, learning to identify regions such as hair boundaries, semi-transparent regions, and areas with color spill that require high-resolution refinement.

The refinement network operates on selected high-resolution patches identified by the error map. Specifically, patches where the error map exceeds a threshold are cropped from the original high-resolution image and processed individually. The refinement network takes as input the high-resolution patch, its corresponding coarse predictions, and the background patch, and outputs refined alpha and foreground for that patch only. This selective strategy means that only a small fraction of the image (typically 5-20%) needs high-resolution processing, dramatically reducing computation. The refined patches are then composited back into the coarse full-resolution output to produce the final result.

We evaluate on both synthetic composites from the Adobe Matting dataset and real-world captures using webcam setups. On the synthetic benchmark, our method achieves comparable or superior alpha quality (SAD, MSE, Grad metrics) to state-of-the-art offline methods including FBA Matting and Index Matting. For speed, our method runs at 4K (3840x2160) at 30fps, HD (1920x1080) at 60fps, and 512x288 at over 100fps on an NVIDIA RTX 2080 Ti. In comparison, FBA Matting requires approximately 4 seconds per 4K frame. Our method is approximately 120x faster than FBA at 4K resolution. User studies confirm that participants rated our results comparably to offline methods for video conferencing scenarios.

We present a real-time, high-resolution background matting method that achieves 4K at 30fps through a novel two-stage architecture with selective refinement. By leveraging an error-map-guided patch selection strategy, our method processes only the critical regions at high resolution, enabling unprecedented speed-quality trade-off for matting applications. The requirement of an additional background image is minimal in typical deployment scenarios such as video conferencing. Our method enables real-time virtual background replacement at resolutions previously achievable only by offline methods.