As Text-to-Image (T2I) diffusion models become increasingly powerful, the scientific community is developing methods to limit their potential misuse, such as generating copyrighted content or harmful imagery. These concept inhibition methods aim to remove specific concepts from a model's generative capability. In this work, we test these safety measures as adversaries to assess their robustness. We leverage the compositional property of diffusion models, which allows combining multiple prompts in a single image generation. Our key insight is that even when a target concept has been inhibited, its generative direction in the model's latent space can be approximately reconstructed by combining other, non-inhibited concepts. We call this approach Concept Arithmetics, drawing an analogy to the well-known arithmetic properties of word embeddings.

Text-to-Image diffusion models such as Stable Diffusion and DALL-E can generate photorealistic images from natural language descriptions. While this capability enables remarkable creative applications, it also raises significant concerns about misuse, including the generation of deepfakes, copyrighted art styles, and violent or explicit content. In response, researchers have developed various concept inhibition techniques that modify the model to prevent generation of specific target concepts. Methods like Erased Stable Diffusion (ESD), Forget-Me-Not, and Concept Ablation fine-tune the model to unlearn particular concepts while preserving its general generative ability.

However, a critical question remains: how robust are these concept inhibition methods against adversarial circumvention? If an inhibited concept can be easily recovered through simple prompt engineering or model manipulation, the safety guarantees provided by these methods would be insufficient. In this work, we demonstrate that the compositional nature of diffusion models creates a fundamental vulnerability in concept inhibition approaches. Specifically, because diffusion models can compose multiple concepts through mechanisms like classifier-free guidance blending and prompt interpolation, an adversary can reconstruct an inhibited concept by arithmetically combining related, non-inhibited concepts.

Our approach, Concept Arithmetics, exploits the compositional property of diffusion models. In standard diffusion sampling, the noise prediction at each step can be expressed as a linear combination of conditional and unconditional predictions through classifier-free guidance. We extend this by observing that multiple conditional predictions can be combined with different weights, effectively performing arithmetic operations in the model's prediction space. Given a target concept T that has been inhibited, we seek non-inhibited concepts A, B, C, ... such that T can be approximated as a weighted sum: T ≈ w_A * A + w_B * B + w_C * C + ... where the weights are optimized to maximize the similarity between the generated output and the target concept.

We evaluate Concept Arithmetics against three state-of-the-art concept inhibition methods: Erased Stable Diffusion (ESD), Forget-Me-Not (FMN), and Concept Ablation (CA). Our experiments cover two categories of inhibited concepts: artistic styles (e.g., Van Gogh, Picasso) and object categories (e.g., specific celebrities, trademarked characters). For artistic style recovery, we find that Concept Arithmetics can reconstruct the inhibited style with a CLIP similarity score of 0.78 to the original, compared to 0.31 achieved by direct prompting of the inhibited model. Human evaluators rated 72% of our reconstructed images as successfully capturing the target style, demonstrating the effectiveness of the compositional attack.

For object category recovery, the results are similarly concerning. When attempting to generate inhibited celebrity faces, direct prompting yields unrecognizable outputs, but Concept Arithmetics can recover identifiable likenesses in 63% of cases as judged by a face verification model. We also demonstrate that the attack generalizes across different concept inhibition methods, suggesting that the vulnerability is not specific to any particular defense approach but is inherent to the compositional nature of diffusion models. Importantly, our attack requires no access to the model's internal weights — it operates purely through the inference API by manipulating the guidance signals.

Our findings have important implications for AI safety. The fact that concept inhibition can be circumvented through the model's own compositional properties suggests that current approaches to content safety in diffusion models may need fundamental rethinking. Simply removing a concept from the model's repertoire is insufficient if the concept can be approximately reconstructed from its constituent parts. We suggest that future safety measures should account for the compositional nature of these models, potentially by monitoring the composition of guidance signals during inference or by developing inhibition methods that also address related concept combinations. We emphasize that the goal of this work is to strengthen AI safety by identifying weaknesses, not to enable misuse.

We have presented Concept Arithmetics, a method that reveals a fundamental vulnerability in current concept inhibition approaches for diffusion models. By leveraging the compositional property of these models, we demonstrate that inhibited concepts can be approximately reconstructed by combining non-inhibited concepts through weighted guidance signal manipulation. Our attacks are effective across multiple concept inhibition methods, require no weight access, and achieve high success rates in recovering both artistic styles and object categories. These findings underscore the need for more robust content safety mechanisms that account for the compositional nature of generative models.