Large text-to-image models achieved a remarkable leap in the evolution of AI, enabling high-quality and diverse synthesis of images from a given text prompt. However, these models lack the ability to mimic the appearance of subjects in a given reference set and synthesize novel renditions of them in different contexts. In this work, we present a new approach for "personalization" of text-to-image diffusion models. Given as input just a few images of a subject, we fine-tune a pretrained text-to-image model so that it learns to bind a unique identifier with that specific subject. Once the subject is embedded in the output domain of the model, the unique identifier can be used to synthesize fully-novel photorealistic images of the subject contextualized in different scenes, while preserving their key visual features.

We leverage the semantic prior that is embedded in the model with a new autogenous class-specific prior preservation loss, which encourages it to generate diverse instances of the same class as the subject. This enables synthesizing the subject in diverse scenes, poses, views, and lighting conditions that do not appear in the reference images. We present applications including subject recontextualization, text-guided view synthesis, appearance modification, and artistic rendering, all while preserving the subject's key features. We also provide a new dataset and evaluation protocol for this new task of subject-driven generation.

Imagine being able to take your dog on a walk through the streets of every city in the world, display your favorite bag in the most exclusive Parisian showrooms, or feature your parrot as a character in a storybook illustration. Synthesizing such scenes requires the ability to render specific subjects in new, previously unseen contexts, where the subject naturally blends into the environment. Recent large text-to-image models demonstrate unprecedented capabilities, enabling high-quality, diverse image synthesis based on natural language prompts. A primary advantage involves the strong semantic priors learned from large-scale image-caption datasets, binding concepts like "dog" with various dog instances across different poses and contexts.

Despite these remarkable capabilities, these models lack the ability to mimic the appearance of subjects in a given reference set and synthesize novel renditions of them in different contexts. The main reason is that the output domain of the model is limited in its expressiveness; even the most detailed textual description of a subject yields varied instances with different appearances rather than a consistent subject identity. Even models with shared language-vision spaces cannot accurately reconstruct the appearance of given subjects, only creating variations of the content described by the text.

In this work, we introduce a new approach for "personalization" of text-to-image diffusion models, expanding the language-vision dictionary to bind new "words" with specific user-desired subjects. Once a subject is embedded in the model, these "words" can be used to synthesize novel photorealistic images of the subject in different scenes, while preserving their identifying features — functioning like a "magic photo booth" where a few subject images can generate infinite photos in different conditions, guided by intuitive text prompts. Formally, given approximately 3-5 casually captured images of a subject, our objective involves implanting the subject into the model's output domain such that it can be synthesized via a unique text identifier.

Our technique represents subjects with rare token identifiers and fine-tunes a pretrained diffusion-based text-to-image framework. Fine-tuning uses the input images paired with text prompts containing the unique identifier followed by a class name (e.g., "A [V] dog"). This enables leveraging the model's prior knowledge of the class while binding class-specific instances to the unique identifier. To prevent language drift — where the model progressively associates class names exclusively with the specific target instance — we propose an autogenous class-specific prior preservation loss that leverages the semantic prior embedded in the model, encouraging generation of diverse instances of the same class.

Image composition techniques aim to clone subjects into new backgrounds with seamless integration. Applying 3D reconstruction for novel poses typically works on rigid objects, requiring larger numbers of views. Drawbacks include difficulties with scene integration (lighting, shadows, contact) and inability to generate novel scenes. Text-driven image manipulation achieved significant progress using GANs combined with image-text representations like CLIP, yielding realistic manipulations. However, these methods work well on structured scenarios (e.g., human face editing) but struggle over diverse datasets with varied subjects. While most editing approaches enable global property modification or localized editing, none previously enabled generating novel subject renditions in new contexts.

Recent large text-to-image synthesis models like Imagen, DALL-E 2, Parti, CogView2, and Stable Diffusion demonstrate unprecedented semantic generation capabilities. However, these models provide limited fine-grained control using text guidance only, making subject identity preservation challenging or impossible across synthesized images. Various approaches attempt to control generative models for subject-driven, prompt-guided image synthesis. Some diffusion-based techniques allow image variations guided by reference images or text. Inversion methods can preserve subjects while modifying context. However, these methods fall short of identity-preserving novel sample generation.

Within GANs, Pivotal Tuning enables real image editing through fine-tuning with inverted latent code anchors. Extended work on GAN fine-tuning for faces trains personalized priors, requiring around 100 images and limiting to face domains. Concurrent work by Gal et al. (Textual Inversion) proposes representing visual concepts through new embedding space tokens in frozen text-to-image models, resulting in small personalized token embeddings. While limited by the expressiveness of the frozen model, their approach demonstrates the potential of concept-level personalization. In contrast, our fine-tuning approach embeds subjects within the model's output domain, generating novel images that preserve key visual features with higher fidelity.

Diffusion models are probabilistic generative models that learn a data distribution by gradually denoising a variable sampled from a Gaussian distribution. Specifically, a pretrained text-to-image diffusion model generates images x̂_θ given initial noise ε ~ N(0, I) and conditioning vector c = Γ(P) produced by a text encoder Γ from prompt P, such that x_gen = x̂_θ(ε, c). Training uses the squared error loss on denoising variably-noised images or latent codes z_t := α_tx + σ_tε.

The training objective can be expressed as: E_x,c,ε,t[w_t ||x̂_θ(α_tx + σ_tε, c) - x||²₂], where x represents ground-truth images, c is the conditioning vector, and α_t, σ_t, w_t are functions of the diffusion process time t ~ U([0,1]) that control the noise schedule and sample quality. This formulation provides the foundation upon which subject-specific fine-tuning can be applied, as the same objective is used but with subject-specific input images and identifier-augmented text prompts.

The goal involves "implanting" new (unique identifier, subject) pairs into the diffusion model's "dictionary". Rather than detailed image descriptions, we label all input images as "a [identifier] [class noun]", where [identifier] is a unique identifier linked to the subject and [class noun] is a coarse class descriptor of the subject (e.g., cat, dog, watch). The class descriptor can be obtained from the user or through a classifier. Using class descriptors tethers class-specific priors to the unique subject; using wrong descriptors or omitting them increases training time, language drift, and decreases performance. Essentially, the method leverages model class priors, entangling them with the subject's unique identifier embeddings for generating novel subject poses and articulations in different contexts.

Existing English words (e.g., "unique," "special") prove suboptimal as identifiers since models must learn to disentangle them from their original meanings and then re-entangle them to reference subjects. This motivates identifiers that have weak priors in both the language model and the diffusion model. Random character concatenation (e.g., "xxy5syt00") is also hazardous because tokenizers split characters separately, each carrying strong diffusion priors. Our approach finds rare tokens in the vocabulary and inverts them back to text space, minimizing the prior probability of the identifier. We perform rare-token lookup in the tokenizer vocabulary and obtain sequences f(V̂), where f is the tokenizer. For Imagen using T5-XXL tokenizer, uniformly sampling tokens corresponding to 3 or fewer Unicode characters (without spaces) in the range {5000,...,10000} works well.

The best results for maximum subject fidelity are achieved by fine-tuning all layers of the model, including text-embedding-conditioned layers. However, this causes language drift — a phenomenon observed in language models where models pre-trained on large text corpora progressively lose syntactic and semantic knowledge during task-specific fine-tuning. Similarly, in diffusion models, the model slowly forgets how to generate diverse same-class subjects as the target. Another problem involves reduced output diversity: fine-tuning on small image sets risks reducing subject output variability in viewpoints, poses, and articulations, sometimes "snapping" to the few-shot training views.

To mitigate these issues, we propose an autogenous class-specific prior preservation loss. The key idea is to supervise the model with its own generated samples, enabling it to retain prior knowledge during few-shot fine-tuning. Specifically, we generate data x_pr = x̂(z_t1, c_pr) using ancestral sampling from the frozen pretrained model with random initial noise z_t1 ~ N(0, I) and conditioning c_pr := Γ(f("a [class noun]")). The full loss becomes: E[w_t||x̂_θ(α_tx + σ_tε, c) - x||² + λ w_t'||x̂_θ(α_t'x_pr + σ_t'ε', c_pr) - x_pr||²], where the second term is the prior-preservation regularizer and λ controls the relative weighting.

Despite its simplicity, this prior-preservation loss effectively encourages output diversity and overcomes language drift, enabling longer model training without overfitting risks. In practice, approximately 1000 iterations with λ = 1 and learning rate 10^-5 for Imagen (or 5 × 10^-6 for Stable Diffusion) with 3-5 subject images achieves good results. During the process, approximately 1000 "a [class noun]" samples are generated from the frozen model — though fewer can suffice. Training requires about 5 minutes on a single TPUv4 for Imagen and a single NVIDIA A100 for Stable Diffusion, making the approach remarkably efficient for personalization.

We collected a dataset of 30 subjects including unique objects and pets: backpacks, stuffed animals, dogs, cats, sunglasses, cartoons, etc. Subjects are separated into objects (21 of 30) and live subjects/pets (9 of 30). We collected 25 prompts per category: 20 recontextualization and 5 property modification prompts for objects; 10 recontextualization, 10 accessorization, and 5 property modification prompts for live subjects. For the evaluation suite, four images per subject-prompt pair are generated, totaling 3,000 images. This setup robustly measures performance and generalization. Datasets and evaluation protocols are publicly available for future subject-driven generation research.

Subject fidelity — preserving subject details in generated images — is computed via two metrics. CLIP-I computes the average pairwise cosine similarity between CLIP embeddings of generated and real images. Though previously used, this metric cannot distinguish different subjects with similar text descriptions (e.g., two different yellow clocks). We therefore propose a new metric, DINO, computing average pairwise cosine similarity between ViT-S/16 DINO embeddings of generated and real images. Unlike supervised networks, DINO's self-supervised training objective encourages distinguishing unique subject features. The second aspect, prompt fidelity, is measured as average cosine similarity between prompt and image CLIP embeddings, denoted CLIP-T.

Results are compared with Textual Inversion (Gal et al.), the most comparable concurrent work for subject-driven, text-guided image generation. Table 1 shows sizeable gaps between DreamBooth and Textual Inversion in both subject and prompt fidelity. DreamBooth (Imagen) achieves higher subject and prompt fidelity scores than DreamBooth (Stable Diffusion), approaching the upper bound of real image subject fidelity, attributed to Imagen's larger expressive power and higher output quality. Further comparisons via user studies with 72 participants and 1,800 comparative answers show overwhelming DreamBooth preference for both subject fidelity (88.5%) and prompt fidelity (81.6%), demonstrating that DINO differences around 0.1 and CLIP-T differences of 0.05 are perceptually significant.

Ablation studies are conducted on two key components. For the prior preservation loss (PPL), fine-tuning with and without PPL on 15 subjects reveals that PPL substantially counteracts language drift, helping retain the ability to generate diverse prior-class images. A prior preservation metric (PRES) measures how much random class subjects resemble specific subjects — higher scores signal prior collapse. Models trained with PPL also achieve higher diversity (DIV) measured by LPIPS, with slightly diminished subject fidelity, as PPL-trained models overfit less to reference image environments. For the class-prior ablation, comparing correct class nouns, incorrect class nouns, and no class nouns shows that correct class nouns yield substantially higher subject fidelity. Incorrect nouns cause contention between subjects and priors, while omitting nouns fails to leverage class priors entirely.

The method enables a wide range of applications. Recontextualization generates subjects in novel contexts with descriptive prompts (e.g., "a [V] dog on a beach"), producing realistic scene-subject integration with contact, shadows, and reflections. Art renditions use prompts like "a painting of a [V] dog in the style of [famous painter]" to generate meaningful artistic variations while preserving subject identity — unlike style transfer which merely applies style to existing structure. Novel view synthesis extrapolates class-prior knowledge to render subjects under previously unseen viewpoints from only 4 frontal input images. Property modification enables cross-species combinations (e.g., "a cross of a [V] dog and a [target species]") and material changes (e.g., "a transparent [V] teapot"), preserving key identity features throughout transformations.

Several failure modes are identified. First, inaccurate generation of prompted contexts: possible reasons include weak context priors or difficulty generating both subjects and specified concepts together due to low co-occurrence probability in training sets. Second, context-appearance entanglement, where the subject's appearance changes due to the prompted context — for example, a backpack's color changing. Third, overfitting to real images occurs when prompts resemble the original subject-capture settings, causing generated images to replicate training environments rather than creating novel ones.

Other limitations include varying difficulty in learning different subjects — common categories (dogs, cats) are easier than rarer subjects, and occasionally rarer subjects cannot support as many variations. Additionally, subject-fidelity variability exists across generated images: some may contain hallucinated subject features depending on model-prior strength and semantic-modification complexity. These limitations suggest that stronger base models with broader training distributions would further improve personalization results, pointing toward future scaling directions.

This work presents an approach for synthesizing novel renditions of subjects using just a few reference images and text-prompt guidance. The key idea involves embedding given subject instances in the output domain of text-to-image diffusion models by binding subjects to unique identifiers. Remarkably, fine-tuning works with only 3-5 subject images, making the technique particularly accessible. The work demonstrates various photorealistic-scene applications with animals and objects, producing results mostly indistinguishable from real images.

The approach leverages rare-token identifiers to minimize prior interference, the class-noun prompt design to harness class-level semantic priors, and the autogenous class-specific prior preservation loss to combat language drift and preserve output diversity. Together, these components enable a remarkably simple yet effective pipeline for subject-driven generation that opens new creative possibilities — from subject recontextualization and novel view synthesis to artistic rendering and property modification. The technique's low data requirement and fast training time make it accessible to a broad range of users and applications, fundamentally expanding the creative capabilities of text-to-image models.