We introduce LegoGPT, the first approach for generating physically stable LEGO brick models from text prompts. To achieve this, we construct a large-scale, physically stable dataset of LEGO designs, along with their associated captions, and train an autoregressive large language model to predict the next brick to add via next-token prediction. To improve the stability of the resulting designs, we employ an efficient validity check and physics-aware rollback during autoregressive inference, which prunes infeasible token predictions using physics laws and assembly constraints. Our experiments show that LegoGPT produces stable, diverse, and aesthetically pleasing LEGO designs that align closely with the input text prompts.

While 3D generative models have advanced significantly, applying them to create real-world objects remains challenging due to assembly and physical stability constraints. LEGO design generation serves as an accessible benchmark with standardized components, yet existing methods do not account for the unique physical constraints and assembly requirements of real-world object designs. The gap between digital 3D generation and physically realizable construction motivates our work.

We propose LegoGPT, which formulates LEGO generation as a sequential token prediction problem. Each brick placement is encoded as a token, and a fine-tuned LLaMA-3.2-1B-Instruct model learns to predict the next brick given the current assembly state and a text description. During inference, we integrate physical stability checks and a physics-aware rollback mechanism that rejects unstable placements and backtracks to find stable alternatives, ensuring all generated designs are both structurally sound and buildable in the real world.

Text-to-3D generation has progressed rapidly with methods like DreamFusion and Score Distillation Sampling, but these approaches produce continuous 3D representations that cannot be directly manufactured. Autoregressive 3D modeling treats shape generation as token sequences, yet prior work focuses on voxels or point clouds rather than structured assemblies with physical constraints. In the LEGO domain, classical legolization algorithms convert 3D meshes to brick layouts through voxelization and merging, but they do not incorporate text conditioning or learned generative priors. Physics-aware generation methods enforce stability in rigid body systems, yet none addresses the combined challenge of text-guided design and physical buildability in brick structures.

We construct StableText2Lego, a dataset containing over 47,000 LEGO structures of over 28,000 unique 3D objects accompanied by detailed captions. The construction pipeline consists of four stages: (1) converting 3D shapes from ShapeNetCore into LEGO structures through voxelization; (2) applying a split-and-remerge legolization algorithm to produce valid brick layouts; (3) assessing physical stability through simulation; and (4) generating captions using GPT-4o from multi-view renderings. This pipeline ensures that every sample in the dataset is both a valid LEGO assembly and physically stable.

LegoGPT fine-tunes LLaMA-3.2-1B-Instruct on the StableText2Lego dataset. Each LEGO structure is serialized as a sequence of brick tokens, where each token encodes the brick type, position (x, y, z), and orientation. The model is trained with standard next-token prediction loss, learning to generate brick-by-brick given the text prompt and the previously placed bricks as context. This formulation naturally captures spatial dependencies between bricks and enables the model to learn structural patterns from the training data.

During inference, we integrate two mechanisms to enforce physical validity. First, a brick-by-brick rejection sampling step checks whether each predicted brick satisfies assembly constraints (no collision, valid connection). Second, a physics-aware rollback mechanism monitors the cumulative structural stability of the growing assembly: if adding a brick causes the structure to become physically unstable (e.g., cantilevered without support), the system backtracks several steps and resamples alternative brick placements. This combination ensures that the final design is not only geometrically valid but also physically buildable — it can stand on its own without external support.

We evaluate LegoGPT on quantitative metrics including physical stability rate, text-design alignment (CLIP score), and aesthetic quality. Compared to baseline methods that apply legolization to text-to-3D outputs, LegoGPT achieves significantly higher stability rates while maintaining comparable or better visual quality. Ablation studies confirm that both the physics-aware rollback and the rejection sampling components contribute meaningfully to the final design quality. We further demonstrate real-world buildability through robotic assembly and manual construction, validating that generated designs can be physically assembled using standard LEGO bricks.

LegoGPT demonstrates that large language models can be effectively repurposed for physically-grounded 3D design tasks by formulating brick assembly as token prediction. The combination of learned generative priors with physics-based constraints during inference ensures that outputs are not merely plausible-looking but genuinely buildable. Limitations include the fixed grid size (20x20x20), the restricted brick library, and computational costs of the rollback mechanism. Future work may explore larger design spaces, more diverse brick types, and integration with robotic assembly systems for fully automated text-to-physical-object pipelines.