The paper introduces ControlMM, a method that incorporates spatial control signals into generative masked motion models. The authors claim to achieve "real-time, high-fidelity, and high-precision controllable motion generation simultaneously". Key innovations include masked consistency modeling and inference-time logit editing. Results show superior motion quality with FID scores of 0.061 versus 0.271 for state-of-the-art methods, with generation speeds 20 times faster than diffusion-based approaches.

Text-driven human motion generation has gained significant attention due to natural language's semantic richness. Applications span animation, film, VR/AR, and robotics. However, text descriptions alone struggle to provide precise spatial control over specific joints like the pelvis and hands, limiting natural environmental interaction and 3D space navigation. Existing solutions face difficulties generating high-fidelity motion with precise, flexible spatial control while ensuring real-time inference.

The paper identifies three key problems: unsatisfied spatial flexibility and accuracy, suboptimal motion quality in controllable models, and slow generation speeds from motion-space diffusion models. ControlMM addresses these by integrating spatial control into generative masked motion models, described as "the first method capable of achieving real-time, high-fidelity, and high-precision controllable motion generation simultaneously".

Text-driven motion generation methods have evolved from KL divergence and contrastive loss approaches (Language2Pose, TEMOS, T2M) to diffusion models operating in motion space, VAE latent space, or quantized space. Token-based models explore autoregressive GPTs and masked motion modeling, predicting multiple tokens simultaneously and generating sequences in as few as 15 steps, achieving state-of-the-art performance on generation quality and efficiency.

For controllable motion synthesis, PriorMDM fine-tunes for end effector control, GMD incorporates root joint guidance, and OmniControl extends to any joint. MotionLCM applies control in latent space, while DNO introduces optimization on diffusion noise. Existing methods are "either limited to specific signals, achieve imprecise control, or rely on optimizing over the diffusion model which is costly and impractical for real-time applications".

Masked Motion Models consist of two stages: a Motion Tokenizer and a Text-conditioned Masked Transformer. The tokenizer learns discrete motion representations by quantizing encoder output embeddings into a codebook. For motion sequence X, the encoder compresses it into latent embedding z, which is quantized into codes c from codebook C. In the second stage, quantized motion tokens are corrupted with [MASK] tokens and fed into the text-conditioned masked transformer to reconstruct the original sequence. During inference, the transformer masks tokens with least confidence and predicts them in parallel.

The architecture consists of a pre-trained text-conditioned masked motion model and a trainable motion control model. Each Transformer layer pairs with a corresponding trainable layer connected via zero-initialized linear layer, ensuring no initial effect during training. The spatial control signal S specifies targeted 3D joint coordinates among 22 total joints, with uncontrolled joints zeroed out. To guarantee controllability, the paper employs consistency training that extracts spatial control signals from generated motion and optimizes consistency loss between input controls and extracted outputs.

Integrating consistency loss into the generative masked model requires converting motion tokens from latent space to Euclidean space, necessitating categorical sampling which is non-differentiable. The authors leverage the straight-through Gumbel-Softmax technique with temperature parameter tau and Gumbel noise. The motion consistency loss assesses alignment between extracted joint control signals from generated motion and input spatial control, normalized by the number of controlled joints and frames.

To achieve accurate, generalizable spatial control, the authors optimize motion token classifier logits and codebook while keeping the network frozen. This reduces discrepancy between generated motion and desired control objectives without requiring pretraining on specific signals, enabling arbitrary out-of-distribution signals and zero-shot tasks like obstacle avoidance. Core idea involves updating logits through gradient-guided optimization during inference. Combining joint logits and codebook editing produces best performance.

Comprehensive experiments are conducted on the HumanML3D dataset containing 14,616 motion sequences with 44,970 text descriptions. Evaluation combines quality metrics (FID, R-Precision, Diversity, Foot Skating Ratio) and trajectory error metrics. Compared to TLControl, FID decreased from 0.271 to 0.061. R-Precision increased from 0.779 to 0.809. Trajectory and Location Errors dropped to zero, with average error at 0.91 cm. For multi-joint configurations, ControlMM outperforms all methods; compared to OmniControl's FID of 0.624, ControlMM achieves FID 0.049.

Ablation studies test eight configurations. Without the Motion Control Model, controlled joints deviate from spatial signals. Logits Editing alone brings root positions closer but wrist accuracy remains poor. Embedding Editing alone provides closer alignment but lacks motion realism. The full model generates realistic motion with high precision matching control signals. Applications include any-joint any-frame control, obstacle avoidance via SDF loss, and body part timeline control — all achieved without retraining.

ControlMM presents a method incorporating spatial control signals into Masked Motion Models, described as "the first model that enables precise control over quantized motion tokens while maintaining high-quality motion generation at faster speeds, consistently outperforming diffusion-based controllable frameworks". Two key innovations drive the results: Masked Consistency Modeling ensures high-fidelity generation while reducing inconsistencies, and Inference-Time Logit and Codebook Editing enhances precision and adaptability for various tasks including any-joint any-frame control, obstacle avoidance, and body part timeline control.