We introduce a novel 3D generation method for versatile and high-quality 3D asset creation. At the core of our method is a unified Structured LATent (SLat) representation that allows decoding to different output formats, such as Radiance Fields, 3D Gaussians, and meshes. This representation is designed by integrating a sparsely-populated 3D grid with dense multiview visual features extracted from a powerful vision foundation model. We employ rectified flow transformers tailored for the structured latent representation and train models with up to 2 billion parameters on a large-scale dataset of 500K diverse 3D objects. Our model generates high-quality results with text or image conditions, and the resulting 3D assets can be directly extracted to various final representations, and further edited to create diverse variations.

While 2D image generation has achieved remarkable progress through large-scale models, 3D generative models still fall short in generation quality compared to their 2D predecessors. A core challenge is that 3D data encompasses diverse representations like meshes, point clouds, Radiance Fields, and 3D Gaussians, each optimized for specific applications yet difficult to adapt across tasks. Geometry-focused approaches often falter in detailed appearance modeling compared to those relying on representations equipped with advanced volumetric rendering capabilities, while appearance-focused methods excel in rendering high-quality appearances but struggle with plausible geometry extraction.

We propose developing a unified and versatile latent space that facilitates high-quality 3D generation across various representations. Our strategy involves two key design choices: (1) introducing explicit sparse 3D structures in the latent space design, enabling diverse decoding targets; and (2) equipping these sparse structures with a powerful vision foundation model for detailed information encoding. The resulting system, TRELLIS, is trained on 500K carefully-collected assets using rectified flow transformers as backbone models adapted for sparse structures. Key capabilities include high-quality generation, versatile format selection, flexible editing without parameter tuning, and training without 3D-specific data fitting.

Some recent methods have leveraged 2D diffusion models for 3D creation, starting with DreamFusion and progressing to multiview generation approaches. However, these 2D-assisted approaches often yield lower geometry quality compared to native 3D models learned from 3D data collections, due to inherent multiview inconsistency. In contrast, our approach directly learns from large-scale 3D data, ensuring geometric fidelity and multiview coherence that 2D-lifting methods fundamentally cannot guarantee.

Early approaches used Generative Adversarial Networks (GANs) to model 3D distributions but faced scaling challenges. Later methods employed diffusion models for various representations like point clouds, voxel grids, Triplanes, and 3D Gaussians. While efficient latent-space approaches emerged, most methods focused either on shape modeling, often requiring an additional texturing phase, or on appearance-rich formats that struggle with plausible geometry extraction. This dichotomy underscores the need for a unified representation that can serve both geometry and appearance.

An alternative paradigm leverages 2D generative models for 3D creation. Score Distillation Sampling (SDS), introduced by DreamFusion, optimizes a 3D representation to match a 2D diffusion prior. Subsequent works explored multiview generation followed by 3D reconstruction. However, these 2D-assisted approaches often yield lower geometry quality compared to native 3D models due to inherent multiview inconsistency. Rectified flow models have recently emerged as a novel generative paradigm that challenges the dominance of diffusions, demonstrating effectiveness for large-scale image and video generation, motivating their adoption for 3D generation.

The core contribution is a representation called Structured Latents (SLat), defined as z = {(z_i, p_i)} for i = 1 to L, where p_i represents active voxel positions in a 3D grid and z_i denotes local latents at those positions. The active voxels outline the coarse structure of the 3D asset, while the latents capture finer details of appearance and shape. The representation leverages sparsity: the number of active voxels L is significantly smaller than the total grid size N³, allowing construction at a relatively high resolution. Default settings use N = 64 resolution yielding approximately L = 20K active voxels.

A critical design insight is the separation between structure (active voxel positions) and content (local latents). This decoupling yields two major advantages: first, the structure can be generated independently and efficiently as a compact binary occupancy grid; second, the latents are generated conditioned on the structure, allowing detail variation and region-specific editing without affecting the overall coarse geometry. The locality of the latents further enables spatial editing by altering voxels and latents in targeted areas while leaving others unchanged.

The encoding process converts 3D assets into voxelized features by aggregating features extracted from dense multiview images using a pre-trained DINOv2 encoder. For each active voxel, features are gathered from randomly sampled camera views on a sphere, projected onto multiview feature maps to retrieve features at corresponding locations, and their average is used as the voxel's visual feature. A transformer-based VAE architecture then processes these voxelized features, serializing input features from active voxels and adding sinusoidal positional encodings to create tokens with variable context length L.

The structured latents can be decoded into versatile output formats. For 3D Gaussians, each z_i is decoded into K Gaussians with position offsets, colors, scales, opacities, and rotations, with final positions constrained to the vicinity of their active voxel using p_i + tanh(o_i). For Radiance Fields, the decoder outputs a CP-decomposition of a local radiance volume at 8³ per voxel. For meshes, the process decodes FlexiCubes parameters and signed distance values, upsampling to 256³ and extracting meshes from 0-level isosurfaces. Notably, the encoder and decoder are trained end-to-end using Gaussians, while other format decoders are simply trained from scratch with frozen encoders, demonstrating strong extensibility.

The Sparse VAE architecture incorporates shifted window attention in 3D space to enhance local information interaction among neighboring voxels. This design choice balances computational efficiency with expressive power: global self-attention across all 20K tokens would be prohibitively expensive, while purely local operations would miss inter-region dependencies. The shifted window mechanism, adapted from the 2D Swin Transformer paradigm to 3D sparse structures, allows information to propagate across window boundaries while maintaining manageable computational costs.

The generation pipeline follows a two-stage approach. The first stage generates the sparse structure {p_i} by converting sparse active voxels into a dense binary 3D grid O, then compressing it via a simple VAE with 3D convolutional blocks into a low-resolution feature grid S. This makes structure generation computationally efficient while converting discrete occupancy values into continuous features suited for rectified flow training. The rectified flow formulation uses linear interpolation x(t) = (1-t)x_0 + t*epsilon and learns a vector field through conditional flow matching.

The second stage generates local latents {z_i} conditioned on the structure. A transformer G_L is designed specifically for sparse structures: instead of directly serializing all 20K tokens, the method improves efficiency by packing them into a shorter sequence using sparse convolutions to aggregate latents within a 2³ local region, followed by multiple time-modulated transformer blocks. Both structure and latent generators incorporate conditions through cross-attention layers: text conditions use CLIP features, while image conditions use DINOv2 visual features. Both models are trained separately using the conditional flow matching objective.

The separation between structure and latents enables two forms of editing. Detail variation is accomplished by preserving the asset's structure and executing the second generation stage with different text prompts, producing variants that adhere to the overall shape while exhibiting diverse appearance and geometry details. Region-specific editing leverages the locality of the representation: altering voxels and latents in targeted areas while leaving others unchanged, accomplished through adapting the Repaint technique to the two-stage generation pipeline and specifying bounding boxes for voxels to be edited. This enables detailed local modifications such as adding rivers and bridges to island models, all guided by text or image conditions.

Training involved approximately 500K high-quality 3D assets from four public datasets: Objaverse (XL), ABO, 3D-FUTURE, and HSSD. The pipeline renders 150 images per asset and employs GPT-4o for captioning. Three model scales were trained: 342M (Basic), 1.1B (Large), and 2B (X-Large) parameters. The X-Large model was trained with 64 A100 GPUs for 400K steps with a batch size of 256. At inference, classifier-free guidance strength is set to 3, sampling steps to 50, and generation time is approximately 10 seconds. Evaluation uses the Toys4k dataset (4K objects not in the training set).

In reconstruction evaluation, the method outperforms all baselines across all evaluated metrics including PSNR, LPIPS, Chamfer Distance, and F-score, with even geometry quality surpassing CLAY which focuses solely on shape encoding. For generation, qualitative comparisons demonstrate superiority over both 2D-assisted methods (InstantMesh, LGM) and 3D approaches (GaussianCube, Shap-E, 3DTopia-XL, LN3Diff), exhibiting not only more vivid appearances and finer geometries but also more precise alignment with provided text and image prompts. Generated assets demonstrate an unprecedented level of quality with vibrant colors, vivid details, complex structures, and flat faces with sharp edges.

Quantitative evaluation on Toys4k using Frechet Distance, Kernel Distance, and CLIP Score confirms the method significantly surpasses previous methods across all evaluated metrics. A user study with over 100 participants across 68 text and 67 image prompts shows the method is strongly preferred by users with clear margins. Ablation studies reveal three critical findings: 64³ resolution provides a significant boost over 32³; replacing diffusion with rectified flow improves both quality and prompt alignment at any generation stage; and increasing model size consistently improves generation performance on both training distribution and held-out test sets, demonstrating favorable scaling behavior up to 2 billion parameters.

We present TRELLIS, introducing a structured latent representation that allows decoding to versatile output formats by comprehensively encoding both geometry and appearance information into localized latents anchored on a sparse 3D grid. Paired with rectified flow transformers scaled to 2 billion parameters and trained on 500K diverse 3D objects, the approach demonstrates superiority in 3D generation in terms of quality, versatility, and editability. The strong scaling behavior and favorable comparison to existing methods highlight strong potential for real-world applications in digital production, suggesting that structured sparse latent representations paired with large-scale flow-based generation provide a promising foundation for the future of 3D content creation.