This paper introduces a methodology for image generation from scene graphs that distinguishes between layout and appearance embeddings. This dual approach produces images with improved scene graph alignment, enhanced visual quality, and capability for complex scene representations. The technique enables diverse output variations per scene graph with user-directed control through two mechanisms: importing elements from other images and navigating object space via appearance archetypes. The system supports real-time rendering for interactive novel scene creation.

Inspired by David Marr's definition of vision -- discovering from images what exists and where -- this work employs scene graphs with object location and appearance attributes as an accessible interface for users to express image synthesis intentions. The "what" aspect captures object class hierarchies with appearance attributes from predefined clusters or sample images. The "where" aspect uses scene graphs representing spatial relationships like "above" or "left of." The methodology employs dual encoding: layout embedding captures relative positioning, while appearance embedding can be replaced independently, enabling object copying between images without affecting others.

Conditional image generation techniques have evolved from class-based and text-description synthesis to more structured inputs. Image translation methods like Pix2pix require paired matching samples. Scene graph-based image generation recently emerged, but prior work synthesizes images from input bounding box layouts at small resolutions without distinguishing layout from appearance. Interactive tools based on GAN dissection enable neuron manipulation but offer only class-level control rather than full instance control. This work provides more semantic manipulation relating to spatial object relations and more precise instance-level control.

The neural architecture comprises multiple components: a graph convolutional network (G) converting scene graphs to per-object layout embeddings; a CNN (M) converting layout embeddings to object masks; a parallel network (B) generating bounding box coordinates; an appearance embedding CNN (A) converting image information to vectors; a multiplexer combining masks and appearance information; and an encoder-decoder residual network (R) generating output images. Each scene graph object node contains object class encoding, location attributes (25 bits for 5x5 grid positioning + 10 bits for size), and appearance embeddings. Stochasticity is introduced via per-object random vectors enabling mask variation.

The optimization combines multiple loss terms: reconstruction loss (L1 difference), bounding box loss (MSE), perceptual loss using VGG network activations, and three discriminators. The mask discriminator uses Least Squares GAN conditioned on object class. The image discriminator ensures generated images match ground truth layouts while penalizing counterfactual appearance vectors -- a novel technique where mismatched appearance embeddings are used to train adversarial robustness. The object discriminator ensures individual generated objects appear realistic by comparing cropped regions. Feature matching losses based on discriminator activations provide additional supervisory signal.

The GUI enables selecting preexisting object appearances via 100 archetypes per class, obtained by applying the learned appearance network to training set objects and employing k-means clustering. Archetypes are arranged linearly using 1-D t-SNE embedding. Users place objects on a schematic layout depicted as strings in ten font sizes capturing size information. Edge labels auto-infer from relative object positioning, eliminating unnecessary user intervention. This coarse user placement proves more intuitive and less laborious than explicit scene graph specification.

Experiments use the COCO-Stuff dataset (approximately 25,000 train, 1,000 validation, 2,000 test images) at 64x64, 128x128, and 256x256 resolutions. The method demonstrates significant performance advantages over baselines across inception score, FID, and classification accuracy at both ground truth and inferred layouts. A user study with 20 participants shows 83.3% rated outputs as more realistic and 80.7% found better scene graph adherence. Ablation analysis reveals that removing perceptual loss is extremely detrimental, and mask discriminator removal most damages the discriminator components.

The authors present an interactive image generation tool accepting scene graphs with optional location information. Each object receives both location and appearance embeddings; the latter can transfer from other images, enabling object duplication with drastically changed layouts. Beyond the dual encoding, novel architecture and loss terms -- including counterfactual training and multi-scale discriminators -- produce improved baseline performance. The system enables a new form of semantic-level image manipulation that is more intuitive than pixel-level editing.