Deep generative models allow for photorealistic image synthesis at high resolutions. But for many applications, this is not enough: content creation also needs to be controllable. While several recent works investigate how to disentangle underlying factors of variation in the data, most of them operate in 2D and therefore miss that our world is three-dimensional. Others do consider the 3D nature but do not scale to complex, multi-object scenes. In this paper, the authors propose GIRAFFE, which "represents scenes as compositional generative neural feature fields," allowing to disentangle individual objects from the background as well as individual object shape and appearance without additional supervision.

Recent advances in generative adversarial networks (GANs) have led to photorealistic image synthesis at unprecedented quality. However, most approaches treat the image generation process as a "black box" where the user has little control over the generated content. Incorporating a 3D representation into the generative model is a promising direction for achieving controllable image synthesis, as it enables explicit control over camera viewpoint and object arrangement.

Prior works on 3D-aware image synthesis either use voxel-based representations that are "limited in resolution due to their cubic memory growth", or mesh-based approaches that require template meshes and category-specific knowledge. Recent neural implicit representations such as NeRF have shown remarkable quality but focus on single-scene reconstruction rather than generative modeling. A key limitation is that most existing methods represent a scene as a single entity, making it difficult to achieve compositional control.

The authors propose GIRAFFE — Generative Implicit Representation as Compositional Feature Fields for Scenes. The key idea is to represent scenes as compositions of neural feature fields, where each object and the background are modeled by individual feature fields. These are combined via a composition operator and volume-rendered at a low resolution, then upsampled using a 2D neural renderer to produce the final high-resolution image. This design allows disentangled control over individual objects' shape, appearance, and pose, as well as the camera viewpoint.

Generative Adversarial Networks have achieved remarkable results in high-resolution image synthesis. StyleGAN and its variants enable some control through latent space manipulation, but the disentanglement is "implicit and incomplete — entangled changes in viewpoint, shape, and appearance are common." Works on 3D-aware GANs incorporate 3D representations but typically model scenes as monolithic entities without compositional structure.

Neural Radiance Fields (NeRF) represent scenes as continuous volumetric functions parameterized by neural networks, mapping 3D coordinates to color and density. While producing state-of-the-art novel view synthesis, NeRF requires per-scene optimization from many posed images and does not naturally support generative modeling or compositional scene manipulation. GRAF extends this to a generative setting but models the entire scene as a single field.

Each entity in the scene (objects and background) is represented by a neural feature field. Given a 3D point x and viewing direction d, the network predicts a volume density sigma and a feature vector f instead of RGB color. The feature field for each object is conditioned on shape code z_s and appearance code z_a, sampled from standard Gaussian distributions. This formulation "allows us to independently sample shape and appearance for each object," achieving disentangled control at the object level.

Individual objects are placed in the scene via affine transformations that control translation, rotation, and scale. The scene is composed by "combining the individual feature fields using a density-weighted mean" of the feature vectors. Specifically, at each 3D point, the composite density is the sum of all entity densities, and the composite feature is the density-weighted average. This simple composition operator enables adding or removing objects and controlling their individual poses without retraining.

The rendering pipeline consists of two stages. First, the composite neural feature field is volume-rendered at a low resolution (e.g., 16x16 or 64x64) using classical volume rendering integration. This produces a 2D feature map rather than an image. Second, a 2D convolutional neural network acts as a neural renderer that upsamples the low-resolution feature map to the final high-resolution image (e.g., 256x256). This two-stage design is critical: "it allows us to keep the expensive 3D volume rendering at low resolution while still producing high-fidelity images," making the approach computationally tractable.

Experiments are conducted on CompCars, FFHQ, CLEVR, and custom multi-object datasets. The model is trained using only unposed 2D images without 3D supervision. Results demonstrate that GIRAFFE achieves disentangled control over camera pose, object translation, rotation, shape, and appearance. On FFHQ at 256x256 resolution, GIRAFFE achieves FID scores competitive with state-of-the-art methods while additionally providing controllability. Compared to GRAF, the method shows significant improvements in image quality and controllability on multi-object scenes. Ablation studies confirm that both the compositional representation and the neural renderer are essential components.

GIRAFFE demonstrates that incorporating a compositional 3D scene representation into the generative model leads to more controllable image synthesis. By representing scenes as compositions of neural feature fields and combining 3D volume rendering with a 2D neural renderer, the method achieves disentangled control over scene properties while maintaining competitive image quality. The approach learns this compositional structure from raw, unposed image collections without explicit 3D supervision, suggesting that 3D-aware generative models are a fruitful direction for controllable content creation.