In this paper, we introduce Janus, an autoregressive framework that unifies multimodal understanding and generation. Prior research often relies on a single visual encoder for both tasks, such as Chameleon. However, due to the differing levels of information granularity required by multimodal understanding and generation, this approach can lead to suboptimal performance, particularly in multimodal understanding. To address this issue, we decouple visual encoding into separate pathways, while still leveraging a single, unified transformer architecture for processing. The decoupling not only alleviates the conflict between the two tasks but also enhances the framework's flexibility. Janus surpasses previous unified models and even outperforms task-specific models in both multimodal understanding and visual generation.

Building a unified model capable of both multimodal understanding and visual generation has been a long-standing goal in AI research. Recent advances in large language models (LLMs) and visual tokenization have made it possible to process both text and images within a single autoregressive framework. However, prior unified approaches such as Chameleon typically use a single vision encoder to process inputs for both understanding and generation tasks. This design choice leads to an inherent conflict: multimodal understanding demands high-dimensional semantic representations that capture abstract concepts, while visual generation requires low-dimensional encoding capable of expressing fine-grained spatial structure and textural detail.

To address this fundamental tension, we propose Janus, named after the two-faced Roman god who looks simultaneously in opposite directions. The key idea of Janus is to decouple the visual encoding into two independent pathways: a SigLIP vision encoder for multimodal understanding that produces high-level semantic features, and a VQ tokenizer for visual generation that encodes images into discrete tokens suitable for autoregressive prediction. Both pathways feed into a single, unified autoregressive transformer, sharing the same language model backbone. This design preserves the simplicity and scalability of a unified architecture while eliminating the representational conflict between understanding and generation.

Despite its simplicity, Janus achieves strong performance on both fronts. On multimodal understanding benchmarks, Janus outperforms the previous state-of-the-art unified model with a 41% improvement on MME (949 to 1338) and a 30% improvement on GQA compared to Show-o. On visual generation, Janus achieves a GenEval accuracy of 61%, surpassing SDXL (55%) and DALL-E 2 (52%), and obtains an FID of 8.53 on COCO-30K. Remarkably, these results are achieved with only 1.3 billion parameters, demonstrating the efficiency and effectiveness of the decoupling approach.

Visual generation has witnessed rapid progress through two major paradigms. Diffusion-based methods, including DALL-E 2, Stable Diffusion, and SDXL, generate images through iterative denoising of Gaussian noise, achieving remarkable quality and diversity. In parallel, autoregressive approaches such as DALL-E and Parti treat image generation as next-token prediction over discrete visual tokens, produced by VQ-VAE or VQ-GAN tokenizers. The autoregressive paradigm is particularly attractive for unification because it naturally aligns with the sequential token prediction used in large language models, enabling a shared architectural backbone.

Multimodal large language models (MLLMs) such as LLaVA, InternVL, and Qwen-VL have established a dominant paradigm for multimodal understanding: a pretrained vision encoder (typically CLIP or SigLIP) extracts visual features, which are then projected into the language model's embedding space via an adaptor module. This architecture excels at visual question answering, image captioning, and visual reasoning. Some recent works attempt to extend these understanding models with generation capability by connecting them to pretrained diffusion decoders, but such hybrid approaches introduce additional complexity and lose the elegance of a truly unified model.

The quest for unified multimodal understanding and generation has produced models such as Chameleon, Show-o, and Emu. These approaches typically use a single visual representation for both tasks: Chameleon employs discrete tokens from a VQ tokenizer for both understanding and generation inputs, while Show-o uses a similar shared tokenization scheme. However, this single-encoder paradigm creates an inherent bottleneck: optimizing the visual encoder for generation quality (requiring fine spatial details) inevitably degrades understanding performance (requiring high-level semantics), and vice versa. Janus addresses this by maintaining separate, task-optimized visual encoders while sharing the transformer backbone.

The Janus architecture consists of four key components. For multimodal understanding, images are processed by a SigLIP-Large-Patch16-384 vision encoder, which produces dense semantic feature maps. These features pass through an understanding adaptor (a linear projection layer) that maps them into the language model's embedding space. For visual generation, images are tokenized by a VQ tokenizer with a codebook of 16,384 entries and 16x downsampling, converting each image into a sequence of discrete codes. These codes pass through a separate generation adaptor (an embedding lookup layer). Both adapted representations are then processed by a unified autoregressive transformer based on DeepSeek-LLM (1.3B parameters).

The critical insight behind the decoupling is that multimodal understanding and visual generation impose fundamentally different requirements on visual encoding. Understanding tasks require a high-dimensional, semantically rich representation where spatial locality is less important than capturing object categories, relationships, and scene context. Generation tasks, in contrast, require a spatially precise, low-level representation that preserves fine-grained details such as textures, edges, and color distributions. By decoupling the encoding while sharing the processing backbone, Janus allows each encoder to be independently optimized for its specific task without compromise.

Janus adopts a three-stage training procedure. In Stage I (Adaptor Training), only the understanding and generation adaptors along with the image head are trained, while the vision encoders and LLM backbone remain frozen. This stage establishes the mapping between visual representations and the shared embedding space. In Stage II (Unified Pretraining), the entire model is trained end-to-end on a mixture of text-only, multimodal understanding, and visual generation data, enabling cross-task knowledge transfer within the shared transformer. In Stage III (Supervised Fine-tuning), the model is fine-tuned with instruction-following data to align with user intent for both understanding and generation tasks.

Janus is an autoregressive model, and the training simply adopts the cross-entropy loss. For multimodal understanding, the loss is computed on the text response tokens, given the image features and text prompt as context. For visual generation, the loss is computed on the image token sequence, conditioned on the text description. Notably, the two tasks share the same loss function and the same next-token prediction paradigm, which is key to maintaining architectural simplicity. The understanding loss and generation loss are simply summed without task-specific weighting, relying on the natural data ratio to balance the two objectives.

During inference, Janus operates in two modes depending on the task. For multimodal understanding, the image is encoded by SigLIP into visual tokens, concatenated with the text query, and the model autoregressively generates text tokens as the response. For visual generation, the text prompt is processed first, and the model autoregressively predicts discrete image tokens, which are then decoded by the VQ tokenizer's decoder into pixel space. To improve generation quality, classifier-free guidance (CFG) is applied with a default scale parameter of 5. The unified inference pipeline requires no task-specific heads or routing mechanisms beyond the initial encoding pathway selection.

The decoupled design of Janus offers natural extensibility along multiple dimensions. The understanding encoder can be upgraded to stronger vision models such as EVA-CLIP or InternViT without affecting the generation pathway. Similarly, the generation tokenizer can be replaced with finer-grained encoders that support higher resolutions or continuous representations. The framework can also accommodate dynamic resolution handling by adapting the understanding encoder's input processing. Furthermore, additional modalities such as audio, video, or 3D data can be incorporated by adding new encoding pathways with dedicated adaptors, each independently optimized for its modality-specific requirements while sharing the same transformer backbone.

Janus is built upon DeepSeek-LLM with 1.3 billion parameters and a context length of 4096 tokens. The understanding encoder is SigLIP-Large-Patch16-384, processing images at 384 x 384 resolution. The generation encoder uses a VQ tokenizer with a codebook size of 16,384 and 16x spatial downsampling. Training is conducted on 128 NVIDIA A100 GPUs (16 nodes x 8 GPUs) for approximately 7 days. The training data spans three stages: Stage I uses ShareGPT4V caption data and ImageNet-1k, Stage II incorporates WikiHow, WIT, and various large-scale caption datasets totaling millions of samples, and Stage III uses curated instruction-following data for both tasks.

On multimodal understanding benchmarks, Janus demonstrates substantial improvements over prior unified models. Compared to the previous best unified model Show-o, Janus achieves POPE 87.0 (vs. Show-o's comparable score), MMBench 69.4, and SEED-Bench 63.7. On the comprehensive MME benchmark, Janus scores 1338, a 41% improvement over Show-o's 949. On GQA, Janus achieves a 30% improvement over Show-o. Remarkably, Janus even outperforms some task-specific understanding models that do not handle generation at all, demonstrating that the decoupled encoding eliminates the understanding performance penalty typically associated with unified models.

On visual generation benchmarks, Janus achieves competitive results against both unified and task-specific models. On the GenEval benchmark, which evaluates compositional text-to-image generation, Janus achieves an overall accuracy of 61%, surpassing SDXL (55%) and DALL-E 2 (52%). On MSCOCO-30K, Janus obtains an FID of 8.53, demonstrating competitive image quality. On the MJHQ-30K aesthetic quality benchmark, Janus achieves an FID of 10.10. These results are particularly notable given that Janus uses only 1.3B parameters, while competing generation models like SDXL employ significantly larger architectures. Qualitative examples further reveal that Janus exhibits superior instruction-following capability and emergent multilingual generation despite being trained on English-only generation data.

Comprehensive ablation experiments validate the effectiveness of the decoupled design. First, a single VQ tokenizer baseline (using VQ tokens for both understanding and generation, as in Chameleon) shows significantly degraded understanding performance, confirming that discrete visual tokens lack the semantic richness needed for comprehension. Second, a semantic tokenizer baseline (using SigLIP features for both tasks) shows better understanding but degraded generation quality, confirming that high-level features lack spatial precision. Third, the full decoupled Janus achieves optimal performance on both tasks simultaneously. Additionally, comparing understanding-only and generation-only training variants with the unified model reveals that unified training achieves performance comparable to task-specific training, demonstrating that the two tasks do not interfere when properly decoupled.

In this paper, we introduced Janus, a simple, unified, and extensible multimodal understanding and generation model. The core idea of Janus is to decouple visual encoding for multimodal understanding and generation, which alleviates the conflict arising from the differing demands that the two tasks place on the visual encoder. By maintaining independent, task-optimized encoding pathways while sharing a unified autoregressive transformer backbone, Janus achieves state-of-the-art performance among unified models and competitive results against task-specific models. We believe the decoupling principle is broadly applicable to future multimodal model designs and that the extensibility of Janus's architecture positions it well for incorporating additional modalities and scaling to larger model sizes.