The paper presents a lightweight adapter enabling image prompts for pretrained text-to-image diffusion models. Key claim: "An IP-Adapter with only 22M parameters can achieve comparable or even better performance to a fully fine-tuned image prompt model." The method uses decoupled cross-attention, remains compatible with text prompts and structural controls, and achieves multimodal image generation capabilities. This enables users to guide image generation with reference images rather than complex text engineering.

While text-to-image diffusion models are powerful, writing effective text prompts requires complex engineering. The authors propose image prompts as an alternative, noting that "an image is worth a thousand words." Previous approaches either fully fine-tune models (requiring substantial resources and losing text compatibility) or replace text encoders (preventing multimodal use). The core innovation involves "a decoupled cross-attention mechanism that separates cross-attention layers for text features and image features." This design allows the adapter to work with custom models, integrate with structural controls like ControlNet, and support combined text-image prompting.

The review covers text-to-image diffusion models (GLIDE, DALL-E 2, Imagen, Stable Diffusion) and adapters for large models (ControlNet, T2I-Adapter, Uni-ControlNet). Prior adapters for image prompts have drawbacks: they eliminate text capability, require extensive computation, are not reusable across model variants, and lack compatibility with existing tools. The paper distinguishes its approach: "Although the aforementioned adapters are lightweight, their performance is hardly comparable to that of the fine-tuned image prompt models." IP-Adapter achieves superior results through its decoupled cross-attention design.

The approach uses a frozen CLIP image encoder to extract features from image prompts. A small projection network converts the global image embedding into a sequence of N=4 tokens matching the text feature dimension. The projection network comprises "a linear layer and a Layer Normalization." This minimal design keeps the adapter lightweight while providing sufficient representation capacity for image content and style transfer.

The core innovation addresses a fundamental limitation: "The key and value projection weights of the cross-attention layer in the pretrained diffusion model are trained to adapt the text features." Rather than concatenating image and text features, the method adds separate cross-attention layers for image features. For each original cross-attention layer, a new one processes image features using the same query but separate key-value projections. The outputs combine additively: Z_new = Attention(Q, K_text, V_text) + Attention(Q, K'_image, V'_image). Only the new key-value weight matrices (W'_k, W'_v) are trainable, initialized from the original weights for faster convergence. At inference, users can adjust the image feature weight lambda: "the model becomes the original text-to-image diffusion model if lambda=0."

Training uses the standard diffusion objective on approximately 10 million image-text pairs from LAION-2B and COYO-700M. The method randomly drops conditions to enable classifier-free guidance. Implementation is built on Stable Diffusion v1.5 with OpenCLIP ViT-H/14 as the image encoder. The adapter totals "about 22M parameters" across 16 added cross-attention layers. Training used 8 V100 GPUs for 1 million steps with batch size 8 per GPU. A key advantage is that "IP-Adapter trained on the base diffusion model can be generalized to other custom models fine-tuned from the same base," enabling direct reuse on community models like Realistic Vision and Anything v4.

Evaluation on COCO validation (5,000 images) shows IP-Adapter achieving 0.588 CLIP-T and 0.828 CLIP-I, "comparable or even better than the fine-tuned model with only 22M parameters." Ablation studies confirm the importance of decoupled cross-attention: compared against a simple concatenation baseline, "the IP-Adapter not only can generate higher quality images, but also more consistent images with image prompts." The method integrates seamlessly with ControlNet and T2I-Adapter for combined image prompt and structural guidance. For multimodal prompts, users adjust lambda to balance image and text guidance, enabling hybrid prompting where "we can use additional text prompt to generate more diverse images."

IP-Adapter demonstrates that image prompt capability can be achieved through a lightweight, modular design. Key contributions include the decoupled cross-attention strategy yielding competitive performance with only 22M parameters, reusability across model variants and tools, and multimodal prompt compatibility. Limitations acknowledged: the method "can only generate images that resemble the reference images in content and style" rather than pixel-perfect subject consistency like DreamBooth. Future work aims to enhance subject fidelity in generated outputs.