The emergence of Large Language Models (LLMs) has unified language generation tasks and revolutionized human-machine interaction. However, in the realm of image generation, a unified model capable of handling various tasks within a single framework remains largely unexplored. In this work, we introduce OmniGen, a new diffusion model for unified image generation.

Unlike popular diffusion models (e.g., Stable Diffusion), OmniGen no longer requires additional modules such as ControlNet or IP-Adapter to process diverse control conditions. OmniGen is characterized by the following features: 1) Unification: OmniGen not only demonstrates text-to-image generation capabilities but also inherently supports various downstream tasks, such as image editing, subject-driven generation, and visual-conditional generation. Additionally, OmniGen can handle classic computer vision tasks by transforming them into image generation tasks. 2) Simplicity: The architecture is highly simplified, eliminating the need for additional text encoders. 3) Knowledge Transfer: Benefit from learning in a unified format, OmniGen effectively transfers knowledge across different tasks, manages unseen tasks and domains, and exhibits novel capabilities.

The pursuit of Artificial General Intelligence (AGI) has intensified the demand for generative foundation models capable of handling a wide variety of tasks within a single framework. In the field of Natural Language Processing (NLP), Large Language Models (LLMs) have become exemplary in achieving this goal, demonstrating remarkable versatility across numerous language tasks such as question answering, text summarization, and code generation. However, the field of visual generation has yet to reveal a counterpart that mirrors the universality of LLMs.

Current image generation models have demonstrated proficiency in specialized tasks. For instance, state-of-the-art models such as the Stable Diffusion series, DALL-E, and Imagen have made significant strides in text-to-image generation. Meanwhile, many efforts have been proposed to extend diffusion models for specific tasks: ControlNet and T2i-Adapter design additional networks for visual conditions; InstructPix2Pix is trained for image editing tasks. Despite their strengths, those models are limited by their task-specific nature and do not exhibit the comprehensive perceptual understanding and generative capabilities required for a universal model.

Is it possible to address various image generation tasks within a single diffusion framework, akin to how GPT handles language tasks? If a universal model is available, the need for training additional modules (e.g., ControlNet, IP-Adapter, T2I-Adapter) in practical applications can be eliminated. Motivated by this potential, we explore a unified framework for image generation, named OmniGen. Unlike popular diffusion models, OmniGen features a very concise structure, comprising only two main components: a VAE and a transformer model, without any additional encoders. OmniGen supports arbitrarily interleaved text and image inputs as conditions to guide image generation.

The design principles of OmniGen are as follows: 1) Universality: accepting any form of image and text inputs for various tasks; 2) Conciseness: avoiding overly complex structural designs and numerous additional components. The OmniGen framework adopts an architecture comprised of a Variational Autoencoder (VAE) and a pre-trained large transformer model. Specifically, VAE extracts continuous visual features from images, while the transformer model generates images based on input conditions. We use the VAE from SDXL and freeze it during training. We use Phi-3 to initialize the transformer model, inheriting its excellent text processing capabilities.

Unlike state-of-the-art diffusion models that require additional encoders to pre-process conditional information (such as CLIP text encoder and image encoder), OmniGen inherently encodes conditional information by itself, significantly simplifying the pipeline. Furthermore, OmniGen jointly models text and images within a single model, rather than independently modeling different input conditions with separate encoders as in existing works, which lacks interaction between different modality conditions.

The input to the model can be multimodal interleaved text and images in free form. We utilize the tokenizer of Phi-3 to process text without any modifications. For images, we employ a VAE with a simple linear layer to extract latent representations, then flatten them into a sequence of visual tokens. We encapsulate each image sequence with two special tokens: "<img>" and "</img>" before inserting it into the text tokens sequence. Different from text, which can be decomposed into discrete tokens, we argue that images should be modeled as a whole. Therefore, we modify the common causal attention mechanism, integrating it with bidirectional attention: causal attention is applied to each element in the sequence, but bidirectional attention is applied within each image sequence.

In this work, we use rectified flow to optimize the parameters of the model. Different from DDPM, flow matching conducts the forward process by linearly interpolating between noise and data in a straight line. The model is trained to directly regress the target velocity given the noised data, timestep, and condition information. For image editing tasks, the difference between input and target images is often small, which allows the model to learn an unexpected shortcut: simply copying the input image. To mitigate this, we amplify the loss in regions where changes occur, assigning higher weights to altered regions than those without changes.

We gradually increase the image resolution during the training process. Low resolution is data-efficient, while high resolution can enhance the aesthetic quality. The training pipeline consists of five stages: starting from 256x256 with 500K steps, progressing through 512x512 (300K steps), 1024x1024 (100K steps), 2240x2240 (30K steps), and finally multi-resolution training (80K steps). We adopt the AdamW optimizer with beta=(0.9, 0.999). All experiments are conducted on 104 A800 GPUs.

To achieve robust multi-task processing capabilities, it is essential to train models on large-scale and diverse datasets. In this work, we have constructed the first large-scale unified image generation dataset, which we refer to as the X2I dataset, meaning "anything to image". We have converted these data into a unified format. The entire dataset comprises approximately 0.1 billion images. It includes: text-to-image data from multiple open-source datasets (Recap-DataComp, SAM-LLaVA, ShareGPT4V, LAION-Aesthetic, etc.), multi-modal to image data for editing, virtual try-on, and style transfer, as well as computer vision tasks such as human pose estimation, edge detection, and image deblurring.

For subject-driven image generation, we constructed both a large-scale foundational dataset (GRIT-Entity, 6 million pairs) and a high-quality advanced dataset (Web Images, 533,000 pairs). The GRIT-Entity dataset leveraged the GRIT dataset with Grounding DINO for text-to-bounding-box grounding and SAM for segmentation. The Web Images dataset was built using natural images of well-known individuals, starting with 2,000 names expanded to approximately 10,000 name pairs, with InternVL for cross-verification filtering. We also constructed a few-shot to image dataset to stimulate the model's in-context learning capabilities.

We evaluate text-to-image generation capability on the GenEval benchmark. Our model achieved an overall score of 0.70, compared to SD3's 0.68 (current state-of-the-art). Notably, our model has only 3.8 billion parameters, whereas SD3 has a total of 12.7 billion parameters (more than three times ours). Current diffusion models typically adopt an encoder-decoder architecture with an additional text encoder that alone is larger than our entire model. Besides, we employed only 0.1 billion image data, whereas SD3 used over 1 billion (more than ten times ours), highlighting the role of multitask data X2I in enhancing text-to-image capabilities.

For image editing, we compare on the EMU-Edit dataset with seven different operations. OmniGen achieves CLIP-I of 0.836, CLIP-T of 0.233, and DINO of 0.804, significantly outperforming InstructPix2Pix and exhibiting comparable performance to the state-of-the-art EMU-Edit. For subject-driven generation on DreamBench, OmniGen achieves DINO of 0.801 and CLIP-I of 0.847, significantly outperforming Re-Imagen and Kosmos-G and demonstrating superior subject fidelity relative to SuTI.

For visual conditional controls, OmniGen achieves optimal results on segmentation mask (mIoU: 44.23) and HED edge map (SSIM: 0.8237), and obtains competitive results for canny edge map (F1: 35.54) and depth map (RMSE: 28.54). Compared to ControlNet and ControlNet++, our model demonstrates strong spatial controllability without requiring any additional control modules. Furthermore, OmniGen can handle various computer vision tasks such as deraining, deblurring, inpainting, human pose recognition, and depth estimation, directly using the generation model to complete traditional vision tasks in a single step.

By standardizing all tasks into a unified format and training on the X2I dataset, OmniGen can acquire universal knowledge and allow knowledge transfer across different scenarios and tasks, thus enabling generation capabilities on unseen tasks and domains. We illustrate several emerging capabilities: Task Composition — the model can simultaneously process multiple instructions, including those for different tasks (image inpainting and color change) as well as multiple instructions for the same task. Implicit Combination — the model can extract relevant conditional information from reference images and generate new images based on captured conditions, with all processing completed internally, negating the need for explicit conditional extraction using other models.

We explored the reasoning capabilities of the model. When given an instruction without explicitly specifying the object, such as "Where can I wash my hands? Please help me find the right place," the model can recognize image contents and infer that a sink is needed, consequently identifying the area of the sink in the image. We also explored a Chain-of-Thought (CoT) approach for image generation, inspired by human drawing behavior: drawing the basic outline, incrementally adding details, making modifications, and applying colors. Unfortunately, the quality of the final generated images does not surpass that of the original model. The step-by-step approach may incorporate erroneous modifications. However, we posit that supervising the drawing process of images is a promising direction.

In-context Learning enables the model to complete novel tasks by providing an example: the model can successfully generate images based on scribble data, which is not encountered during training. We show examples from the FSS dataset, which contains objects never seen or annotated in previous datasets. When provided with an example, the model makes accurate predictions, demonstrating that in-context learning can enhance the model's generalization ability across different domains. Furthermore, the end-to-end workflow enabled by OmniGen significantly simplifies the existing multi-model pipeline: users can specify objects within images through textual instructions and generate new images without preliminary operations such as image cropping or loading additional models for preprocessing.

We summarize the limitations of the current model: 1) Similar to existing diffusion models, OmniGen is sensitive to text prompts; detailed text descriptions result in higher-quality images. 2) The current model's text rendering capabilities are limited; it can handle short text segments but fails to accurately generate longer texts. Due to resource constraints, the number of input images during training is limited to a maximum of three. 3) Generated images may contain erroneous details, especially small and delicate parts; facial features occasionally do not fully align in subject-driven tasks, and incorrect depictions of hands also occur. 4) OmniGen cannot process unseen image types (e.g., surface normal estimation).

We believe that most limitations can be addressed by training the model on more related data. Moreover, compared to most models, fine-tuning OmniGen for downstream tasks is simpler, as it inherently supports various image generation tasks without the need for extensive efforts and costs to build additional networks. This work represents the first attempt at a general-purpose image generation model, and there remain several unresolved issues. We will open-source the related resources to foster advancements in this field.

The generative foundation model serves as the core of many contemporary AI systems. The GPT series demonstrated that language models can learn numerous tasks via training on large-scale datasets. Multimodal large language models such as LLaVA have been proposed to integrate vision and language capabilities. However, despite their ability to handle mixed text and image inputs, they lack the capability to generate images. Recently, works such as Chameleon, TransFusion, and Show-O have explored unified models supporting both text and image generation. Nonetheless, like most existing diffusion models, they can only perform text-to-image tasks and cannot handle more complex and various visual generation tasks.

Recent advancements in diffusion models have been remarkable. Models like ControlNet and T2i-Adapter introduce supplementary networks for visual-conditioned generation. InstructPix2Pix addresses image editing by augmenting the model with additional input channels. SEED-X and Kosmos-G employ an MLLM to replace the CLIP encoder. However, these methods are task-specific, extending capabilities by modifying the model architecture. In contrast, OmniGen natively supports various image generative tasks, unifying all tasks into a single framework. Multi-task learning enhances capabilities and also leads to the emergence of new abilities. Furthermore, OmniGen no longer requires any preprocessing steps or assistance from other models.