Recent endeavors in Multimodal Large Language Models (MLLMs) aim to unify visual comprehension and generation by combining LLM and diffusion models, the state-of-the-art in each task, respectively. Existing approaches rely on spatial visual tokens, where image patches are encoded and arranged according to a spatial order (e.g., raster scan). However, we show that spatial tokens lack the recursive structure inherent to languages, hence form an impossible language for LLM to master. In this paper, we build a proper visual language by leveraging diffusion timesteps to learn discrete, recursive visual tokens. Our proposed tokens recursively compensate for the progressive attribute loss in noisy images as timesteps increase, enabling the diffusion model to reconstruct the original image at any timestep. This approach allows us to effectively integrate the strengths of LLMs in autoregressive reasoning and diffusion models in precise image generation, achieving seamless multimodal comprehension and generation within a unified framework. Extensive experiments show that we achieve superior performance for multimodal comprehension and generation simultaneously compared with other MLLMs.

Multimodal Large Language Models (MLLMs) strive to unify the comprehension and generation of data across various modalities within the same next-token prediction paradigm of LLM. Specifically, given a user query about comprehension — "What kind of dog is in this picture [IMG]", or generation — "Turning the picture [IMG] into a sketch", the model can complete the task by sequentially predicting the appropriate text or image tokens. The challenge lies in the conflicting objectives of the two tasks. Comprehension pursues a many-to-one mapping that abstracts visual details (e.g., many photos of corgi dogs result in recognition of "corgi"). On the other hand, generation finds a one-to-one mapping that preserves visual details (e.g., the sketched image specific to the query image). To bridge these conflicting objectives, the most straightforward way is to combine LLMs and diffusion models (DMs), which excel in comprehension and generation respectively.

Two main integration strategies have emerged. Cascading approaches let LLMs process queries and pass instructions to diffusion models, while tokenization-based approaches discretize visual modalities into tokens for LLM processing before decoding back to images via diffusion models. Recent work like Transfusion aims to combine both into unified frameworks. However, a critical limitation persists: existing methods rely on spatial visual tokens, where image patches are encoded and arranged in a spatial order. These tokens lack the recursive structure inherent to natural language, forming what the authors term an "impossible language" for LLMs. Perturbation analysis in Figure 1 demonstrates that spatial tokens are robust to sequence disruption — shuffling their order barely affects generation quality — proving they lack the sequential dependency that defines language.

In this paper, the authors propose Discrete Diffusion Timestep (DDT) tokenization to learn recursive tokens that reflect how diffusion progressively corrupts images. The DDT tokens form expanding token sequences that compensate for incrementally lost visual attributes as noise increases across timesteps. Three main contributions are outlined: (1) proposing DDT as a principled method to create recursive visual tokens with language-like properties; (2) introducing an integration method that combines LLMs with diffusion models through these tokens, enabling unified comprehension and generation; and (3) demonstrating state-of-the-art results across text-to-image generation, image editing, and vision-language understanding, notably with a tokenizer trained only on ImageNet at 256x256 resolution that surpasses methods using larger datasets like LAION.

Research on achieving unified understanding and generation in multimodal models has primarily focused on two main strategies: cascading architectures and tokenization-based methods. Cascading architectures integrate separate modality-specific encoders and decoders, each pre-trained independently, and then fuse their representations through projection layers to create combined models for multimodal tasks. Notable examples include models such as EMU2, which uses pre-trained language models augmented with EVA-02-CLIP-E-plus for comprehension tasks, and cascades an SDXL-initialized diffusion model for visual generation tasks.

In contrast, tokenization-based methods aim to create a unified framework by converting visual and textual inputs into a discrete space, and then jointly training a single transformer based solely on next-token prediction. Moreover, recent advances such as TransFusion and Show-o explore a blend of diffusion and autoregressive models within a single transformer for enhanced performance. However, although these methods provide a step towards unification, most of them focus on spatial tokens for vision, which are extracted from image patches and arranged in a spatial order. These spatial tokens "lack the traits of human language, resulting in an inability to seamlessly integrate with human natural language" within an MLLM. Consequently, existing MLLMs still lag behind specialized architectures like SDXL in visual generation tasks and LLaVA-1.6 in visual comprehension tasks. These indicate the need for further exploration into more holistic tokenization methods that go beyond spatial representations.

The authors present a tokenizer architecture with three core components. The goal is to "train an image tokenizer that encodes an image to a recursive sequence of discrete tokens" while enabling decoding back to images. The system comprises: (1) an encoder mapping noise-free images to continuous features, (2) a vector quantizer assigning features to token embeddings from a fixed dictionary, and (3) a diffusion model decoder reconstructing images from tokens and noisy inputs. Unlike conventional image tokenizers that produce spatially arranged tokens, the DDT tokenizer produces temporally structured tokens aligned with diffusion timesteps, establishing a recursive dependency between tokens.

The encoder uses a transformer-based architecture with learnable query tokens. Input consists of patchified images (similar to Vision Transformers) and T learnable query tokens. The system applies "1D and 2D sinusoidal position embeddings on the query and image tokens, respectively." Following the SD3 architecture, it employs "two independent transformers to process the query and image tokens, and join them for the attention operation." Only the transformed query tokens are retained as output. A standard Vector Quantization module with a fixed-size dictionary of 65,536 entries quantizes encoder outputs. The system projects encoder outputs from 256 dimensions to 16 dimensions before lookup, using an EMA-variant of VQ for training stability. Dead entries are monitored and reset to random tokens at each training step.

The decoder is a diffusion model based on the MMDiT architecture from SD3 with modifications. Rather than text conditioning tokens, it "inputs a sequence of quantized tokens" with "a linear layer to project the m-dimensional vector to the latent dimension." The system learns recursive token sequences where later tokens compensate for attribute loss in progressively noisier images. The training loss minimizes reconstruction error across timesteps, following Rectified Flow sampling methods. At timestep t, the decoder receives the first t tokens (masking remaining tokens) and attempts to reconstruct the clean image from the noisy input. A commitment loss regularizes encoder outputs to match quantized vectors. This design ensures that each successive token encodes the residual visual information lost at the corresponding noise level, creating a natural recursive hierarchy.

The paper describes converting images into "a 1D recursive discrete sequence like a foreign language that LLM can read." Visual tokens are concatenated with text tokens in multimodal sequences, with special markers [BOV] and [EOV] distinguishing modalities. The model initializes from LLaMA-3-8B, expanding vocabulary by 65,536 visual codes. Training employs a unified next-token-prediction objective across image-text data pairs using cross-entropy loss, where both modalities share a prediction head. This design means that no separate diffusion loss or auxiliary objectives are needed during MLLM training — the standard language modeling loss suffices for both text and image generation.

Training proceeds in two stages. Stage 1 (Pre-training) uses 200 million image-text pairs from LAION and COYO datasets, structured as [BOS] <caption> [BOV] <DDT tokens> [EOV] [EOS]. Pure text data comprises 10% of training to prevent textual capability degradation. Training occurred on 512 Ascend 910B NPUs for nearly two weeks. Stage 2 (Instruction Tuning) applies supervised fine-tuning on visual comprehension and generation tasks using the format "USER: <Instructions> ASSISTANT: <Answers>", scoring only assistant responses. During inference, after predicting [BOV], the model generates T sequential DDT tokens. These feed into the decoder, which denoises random noise over T timesteps via DDPM, providing only the first t tokens at timestep t while masking remaining tokens.

The researchers evaluated DDT-LLaMA using three benchmarks: GenEval, T2I-CompBench, and DrawBench. Results showed the model "significantly outperforms SEED-X, Emu3, and the specialist method SDXL" on GenEval overall scores. The model achieved a GenEval score of 0.66, compared to 0.54 for Emu3 and 0.49 for SEED-X. DDT-LLaMA demonstrated particular strength in "tasks related to color, counting, and position," excelling at "understanding object attributes such as color, quantity, and the spatial relationships between objects." Qualitative examples show DDT-LLaMA "effectively follows various types of instructions, including complex ones such as generating surreal images and multi-condition combined prompts." However, the researchers acknowledge that "there is room for improvement in the aesthetic quality of the images. This limitation stems from the fact that our current tokenizer was only trained on ImageNet with a resolution of 256x256."

Evaluation across three editing datasets (EVR, MA5K, MagicBrush) revealed superior performance. On MagicBrush, DDT-LLaMA achieved an L1 score of 7.1 versus 8.2 for MGIE and 6.6 for UltraEdit. The model "demonstrates significant superiority over existing MLLMs across all test datasets and metrics," supporting "a wide spectrum of editing operations including both local change (e.g., removal, replacement) and global change (change time, manipulation)," while achieving "a great trade-off between fidelity and editability." For vision-language comprehension, comparisons across nine benchmarks (NoCaps, Flickr30K, VQA, GQA, OKVQA, VizWiz, MME, SEEDBench, POPE) show DDT-LLaMA achieves competitive or superior results. On NoCaps, it scored 124.2 versus 117.5 for Emu3 and 114.2 for LaVIT. The model "significantly surpasses its counterparts across multiple benchmarks, even without relying on a specialized pretrained CLIP."

Several in-depth analyses validate the core claims. For class-conditional generation, DDT achieves reconstruction PSNR of 21.7 on ImageNet validation and FID of 6.1, surpassing MoVQ's 7.1 and RQ-Transformer's 7.6. Counterfactual interpolation experiments demonstrate that "DDT tokens, with their disentangled representation, ensure that only the attributes captured by the substituted tokens change in the generated counterfactuals." Expanding token subset experiments show "with the number of tokens increasing, the image's attributes are progressively recovered — initially the decoder reconstructs fine details, with contours and color information gradually completed." Critically, training with DDT tokens "results in a decrease in perplexity for text generation, whereas training with VQGAN tokens yields the opposite" — demonstrating DDT tokens are "better suited for unified autoregressive modeling with text tokens." In A/B testing using Gemma2-2B, DDT-Gemma outperforms MoVQ-Gemma in 65 editing cases while MoVQ-Gemma surpasses DDT-Gemma in only 10 cases. Preliminary results also indicate scaling laws in DDT-based MLLM, with improvements across model sizes (2B, 8B).

The authors propose Discrete Diffusion Timestep (DDT) tokenization to "learn discrete, recursive visual tokens, which recursively compensate for the progressive attribute loss in noisy images as timesteps increase." They trained DDT-LLaMA on vast image-text corpora for vision-language alignment. The approach achieves "superior performance for multimodal comprehension and generation simultaneously compared with other MLLMs," excelling across text-to-image generation, image editing, and vision-language understanding. The paper notes they are "currently working on scaling up the training of our DDT tokenizer and the MLLM," with plans to release enhanced versions. This work demonstrates that the key to unifying multimodal comprehension and generation lies not in model architecture alone, but in designing a visual token language that is structurally compatible with the autoregressive nature of LLMs.