We present a method for rewriting the rules encoded by a deep generative model. A generative adversarial network (GAN) learns rich semantics about the visual world, encoding rules such as "trees have leaves" or "doors are on buildings." We propose a method to directly modify specific rules in the model's weights through a constrained optimization that changes a small number of model parameters to achieve a desired visual effect while preserving the model's other capabilities.

Deep generative models like StyleGAN produce remarkably realistic images, but controlling their output semantics remains challenging. Existing methods for GAN editing operate in the latent space, modifying individual samples rather than the model's learned rules. We take a fundamentally different approach: instead of editing individual images, we edit the model itself to change the rules it has learned. For example, we can make a model that always generates churches with domes, or horses with stripes.

This distinction has profound implications: model-level editing changes apply to all future generated samples, not just one. This enables systematic modifications like "remove all windows from churches" or "add grass to all outdoor scenes." Such operations would require editing each generated image individually with latent-space methods, which is impractical for large-scale generation.

Our approach works by identifying a specific layer and set of neurons in the generator that are responsible for the rule to be changed. We then solve a constrained optimization problem that modifies the weights of that layer to produce the desired effect on a set of exemplar images, while minimizing changes to the network's behavior on other inputs. The key constraint is a rank-1 update to the weight matrix, which limits the modification to a single semantic direction.

Formally, given a layer with weight matrix W, we compute the optimal rank-1 update W' = W + d * k^T, where d is the desired output direction and k is the key that selects when to apply the change. The key k is computed to activate on the specific visual pattern (e.g., tree canopy) while remaining inactive on other patterns. This formulation allows precise, localized editing of the model's rules without retraining.

We demonstrate model rewriting on StyleGAN trained on churches, horses, and kitchens. Examples include: adding domes to all churches, changing tree species, adding stripes to horses. User studies show that edited models produce coherent changes that are preferred over latent-space editing in 78% of cases. The changes are consistent across all generated samples, confirming that we are editing rules rather than individual images. Editing takes only seconds per rule change.

We also study the scope of modifications. The rank-1 constraint ensures that changes are localized to the targeted semantic concept. Quantitatively, FID scores on the unmodified concepts change by less than 1 point, confirming minimal collateral damage. We compare against fine-tuning the entire layer, which achieves similar target modifications but degrades FID by 5-10 points on other concepts.

We have shown that the rules encoded by a deep generative model can be directly rewritten through targeted modification of model weights. Our rank-1 rewriting approach provides a new tool for understanding and controlling the knowledge stored in neural networks. This opens up possibilities for model debugging, creative applications, and studying how neural networks encode semantic knowledge.