We propose MaskGIT, a novel image synthesis paradigm using a bidirectional transformer decoder. During training, MaskGIT learns to predict randomly masked tokens by attending to tokens in all directions. At inference, the model begins by generating all tokens simultaneously, then iteratively refines the image by re-masking and re-predicting low-confidence tokens. MaskGIT significantly outperforms the state-of-the-art transformer model on the ImageNet dataset, while accelerating autoregressive decoding by up to 64x. We further demonstrate that MaskGIT can be easily extended to various image editing tasks, such as inpainting, extrapolation, and image manipulation.

Existing generative transformers treat images as sequential 1D token sequences following raster scan ordering (left-to-right, line-by-line). This approach is fundamentally problematic because images are inherently 2D and non-sequential. The sequential decoding strategy creates quadratically long sequences and requires substantial computation — approximately 30 seconds per GPU-generated image with 32x32 tokens. This computational bottleneck limits practical deployment and restricts the achievable image resolution.

MaskGIT adopts a non-autoregressive approach inspired by how artists create paintings: starting with a sketch and progressively refining details. The model predicts all tokens in parallel at each iteration, retaining only the most confident predictions while masking uncertain ones for refinement in subsequent iterations. This constant-iteration approach completes 256-token images in 8 steps versus 256 steps for autoregressive methods. Key technical innovations include: (1) bidirectional self-attention enabling context from all directions, (2) a novel mask scheduling strategy determining which tokens to predict at each iteration, and (3) iterative refinement balancing quality and efficiency.

Our work is inspired by BERT's masked language modeling for bidirectional representation learning. While masked modeling has been extended to vision tasks (MAE, BEiT) for representation learning, few works have applied it successfully to image generation on standard benchmarks. We provide the first evidence demonstrating the efficacy of masked modeling for image generation on the common ImageNet benchmark. Our approach borrows from bidirectional machine translation but introduces novel masking strategies and decoding algorithms specifically designed for generation rather than representation learning.

The model learns Masked Visual Token Modeling (MVTM), inspired by BERT's approach. Given latent tokens Y and binary mask M, the training objective minimizes the negative log-likelihood of masked tokens. Training randomly masks a variable number of tokens determined by a mask scheduling function: the number of masked tokens equals the ceiling of gamma(r) times N, where gamma is the scheduling function, r is a random ratio, and N is the total sequence length. The bidirectional transformer predicts probability distributions for masked positions, utilizing context from all directions rather than only preceding tokens as in autoregressive models.

At inference, the iterative decoding algorithm proceeds in T steps: (1) Predict: the model generates probability distributions for all masked positions in parallel; (2) Sample: tokens are sampled with prediction confidence serving as confidence scores; (3) Mask Schedule: compute the number of tokens to mask next using gamma(t/T) times N; (4) Mask: retain only the most confident predictions and mask lower-confidence positions for refinement. This process continues until all tokens are generated. The mask scheduling function critically impacts generation quality — it must be continuous, bounded in (0,1], and monotonically decreasing with gamma(0) approaching 1 and gamma(1) approaching 0.

The paper evaluates three families of scheduling functions: linear (equal tokens masked per iteration), concave (cosine, square, cubic, exponential — emphasizing early iterations with fewer correct predictions), and convex (square root, logarithmic — requiring most tokens to be finalized early). Ablations show that concave functions outperform others, with cosine achieving the best results. The authors hypothesize that concave functions succeed by challenging training with difficult cases and appropriately prioritizing a less-to-more prediction progression — generating fewer tokens with high confidence initially, then progressively committing to more tokens as context accumulates.

The implementation uses 24 transformer layers, 8 attention heads, 768 embedding dimensions, and 3072 hidden dimensions. Images are compressed by factor 16 using a tokenizer with 1024-token codebook. Training uses 4x TPU devices, batch size 256, Adam optimizer. On ImageNet 256x256, MaskGIT achieves FID of 6.18 versus VQGAN's 15.78 and Inception Score 182.1 versus 78.3. At 512x512 resolution, FID reaches 7.32, establishing new state-of-the-art on classification accuracy score and FID metrics, exceeding BigGAN's 8.43 FID at this resolution.

MaskGIT requires 8-12 inference steps versus VQGAN's 256-1024 steps. Wall-clock runtime comparisons demonstrate MaskGIT accelerates VQGAN by 30-64x, with speed advantages increasing at higher resolutions due to token sequence length growth. Using Classification Accuracy Score (CAS), MaskGIT achieves 63.14 (top-1) and 84.45 (top-5) on 256x256 versus VQGAN's 53.10 and 76.18. Precision/Recall analysis shows MaskGIT balances quality and coverage better than single-mode GANs, offering competitive sample diversity with superior coverage compared to BigGAN.

The model's bidirectional nature enables a novel class-conditional image editing task: replacing bounding box content with target class objects while preserving context — a task infeasible for autoregressive methods due to their sequential left-to-right generation constraint. For image inpainting on Places2 (center 50% masking), MaskGIT achieves FID 7.92, comparable to dedicated inpainting methods like CoModGAN (7.13) without task-specific training. For image outpainting (right 50% extrapolation), MaskGIT achieves FID 6.78 and IS 11.69, beating all baselines including InfinityGAN (10.60 FID), handling arbitrary-direction outpainting with a single model.

MaskGIT introduces a paradigm shift in generative transformers through bidirectional masked modeling and iterative parallel decoding. The method achieves state-of-the-art image synthesis results on ImageNet while demonstrating substantial speed improvements (30-64x) and versatility across editing tasks. The cosine mask scheduling strategy provides the optimal less-to-more prediction progression, and the iterative refinement mechanism resolves the training-inference discrepancy inherent in single-pass non-autoregressive generation. The authors acknowledge limitations in handling complex structures like faces and text, and propose extending the approach to additional synthesis tasks as future work.