We present ControlNet, a neural network architecture to add spatial conditioning controls to large, pretrained text-to-image diffusion models. ControlNet locks the production-ready large diffusion model and reuses its deep and robust encoding layers as a strong backbone to learn diverse conditional controls. The trainable copy is connected to the locked model via "zero convolution" layers — convolution layers initialized with zero weights — that progressively grow the parameters from zero and ensure that no harmful noise affects the finetuning. We test ControlNet with Stable Diffusion on various conditioning inputs such as edges, depth, segmentation, and human pose, showing that the training is robust even with limited data (<50k) and scales well to large datasets (>1M).

Large text-to-image diffusion models such as Stable Diffusion have demonstrated remarkable capabilities in generating high-quality images from text descriptions. However, text prompts alone provide limited control over the spatial composition of generated images. Users cannot precisely specify where objects should appear, what pose a person should take, or how edges in the scene should be arranged. Enabling finer-grained spatial control by letting users provide additional images that directly specify their desired image composition — such as edge maps, pose skeletons, or depth maps — is a crucial next step for practical creative applications.

A core challenge is that task-specific datasets for spatial conditioning are typically orders of magnitude smaller than the billions of image-text pairs used to train models like Stable Diffusion. Directly finetuning the entire model on such small datasets risks catastrophic forgetting — degrading the model's original generative capabilities. Existing approaches such as HyperNetworks, Adapters, and LoRA offer partial solutions but may not fully preserve the quality of the pretrained backbone when learning complex spatial conditions. We propose ControlNet to address this challenge with a dedicated architecture that fundamentally prevents degradation through zero-initialized connections.

Finetuning large neural networks has been studied through various paradigms. HyperNetworks train small networks to influence larger ones; Adapters embed new modules without modifying original weights; LoRA prevents forgetting through low-rank matrix learning; and zero-initialized layers prevent harmful noise during training initialization. In the domain of image diffusion, text-to-image systems including GLIDE, Disco Diffusion, and Stable Diffusion have achieved remarkable quality. Prior conditional generation works — from conditional GANs to PITI — address image-to-image translation but typically require training from scratch or do not leverage the full capacity of pretrained diffusion backbones.

The core innovation of ControlNet involves creating a trainable copy of neural network blocks while keeping the originals frozen. Given a trained block with function F(x; Theta) that transforms input x to output y, ControlNet adds a trainable copy with parameters Theta_c. These connect via zero convolution layers Z(.; .) — 1x1 convolutions initialized entirely to zero. The complete ControlNet computes: y_c = F(x; Theta) + Z(F(x + Z(c; Theta_z1); Theta_c); Theta_z2). This initialization ensures that initially, both zero terms evaluate to zero, preserving original outputs while preventing harmful noise from corrupting the backbone during early training.

Applied to Stable Diffusion's U-Net architecture, ControlNet creates trainable copies of 12 encoding blocks plus 1 middle block across four resolutions (64x64, 32x32, 16x16, 8x8). Outputs are added to corresponding skip connections in the decoder. Input conditioning images are first encoded from 512x512 pixel space to 64x64 feature space using a small four-layer convolutional encoder with 4x4 kernels, 2x2 strides, and channels [16, 32, 64, 128]. The training overhead is minimal: approximately 23% more GPU memory and 34% more time per iteration compared to standard Stable Diffusion optimization on a single NVIDIA A100.

ControlNet uses the standard diffusion objective. A key technique involves randomly replacing 50% of text prompts with empty strings during training, forcing the model to recognize semantic content directly from the conditioning inputs. A notable phenomenon emerges: the model does not gradually learn the control conditions but "abruptly succeeds in following the input conditioning image; usually in less than 10K optimization steps." This "sudden convergence" occurs because zero convolutions prevent noise accumulation, maintaining constant output quality throughout training.

We demonstrate results across eight conditioning types: Canny edges, depth maps, normal maps, M-LSD lines, HED edges, ADE20K segmentation, OpenPose, and user sketches. ControlNet robustly interprets content semantics in diverse input conditioning images, even without text prompts. In ablative studies, standard convolutions with Gaussian initialization perform poorly, and removal of zero convolutions destroys the pretrained backbone. In user studies, ControlNet achieved 4.22/5 for result quality and 4.28/5 for condition fidelity. Remarkably, ControlNet trained on only 200k depth samples with one RTX 3090Ti over 5 days achieved results indistinguishable from Stable Diffusion V2 Depth-to-Image trained on 12M images across A100 clusters.

An important finding is the transfer learning capability of trained ControlNets. Models trained on standard Stable Diffusion can be directly applied to community-finetuned models like Comic Diffusion and Protogen without retraining. This demonstrates that ControlNet learns condition-to-feature mappings that generalize across model variants sharing the same architecture. Additionally, multiple ControlNets can be composed by simply adding their outputs, enabling multi-condition generation without additional weighting or interpolation.

ControlNet establishes an efficient architecture for adding spatial controls to pretrained diffusion models. By freezing the original backbone and connecting trainable copies through zero convolutions, the method "reuses the large-scale pretrained layers of source models to build a deep and strong encoder to learn specific conditions." The approach enables practical applications across diverse conditioning types with varied dataset sizes and computational budgets, demonstrating that pretrained diffusion models can be effectively repurposed as powerful backbones for spatial-conditional image generation.