The paper introduces SceneSplat, described as "the first large-scale 3D indoor scene understanding approach that operates natively on 3DGS". It addresses the challenge of effectively integrating semantic reasoning into 3DGS in a generalizable manner. Key contributions include SceneSplat-7K, a new dataset containing 7,916 scenes derived from seven established datasets totaling 8.15 billion 3D Gaussians. The authors propose a self-supervised learning scheme called GaussSSL that enables rich 3D feature learning from unlabeled scenes, achieving state-of-the-art performance in zero-shot semantic segmentation.

Recognizing arbitrary or previously unseen categories is essential for comprehensive real-world 3D scene understanding. Current systems trained on fixed closed-set categories from datasets like ScanNet fail to capture real-world diversity. The fundamental challenge is "the absence of a robust model for processing 3D data end-to-end for semantic learning, along with the lack of sufficient data for training such a model". 3D Gaussian Splatting has emerged as the de facto standard for 3D scene representation, but integrating semantic reasoning remains problematic. Prior approaches like optimizing additional semantic features are inefficient and limited to a single scene.

Existing 3D datasets (ScanNet, ScanNet++, Hypersim, ARKitScenes) are essential for 3D perception but lack large-scale support for emerging 3D representations like 3DGS. For open vocabulary scene understanding, LERF integrated language queries into NeRF but requires time-consuming preprocessing. LangSplat combined 3DGS with open-vocabulary embeddings from SAM and CLIP. SceneSplat's distinction: it enables "feed-forward open-vocabulary understanding of 3DGS" without requiring time-consuming preprocessing.

The approach uses SAMv2 for object-level segmentation and SigLIP2 for feature extraction, then employs Occam's LGS to efficiently lift 2D feature maps to a 3D Gaussian feature field. Rather than aligning with text embeddings, the method directly aligns Gaussians with the image embedding space of vision-language models, preserving richer latent semantic information. A dynamic weighting mechanism combines three feature types: global features capturing full scene context, local features from crops with background, and masked features focusing solely on objects.

The model architecture adapts a transformer encoder-decoder backbone to map input Gaussians to language features. Three training objectives are employed: Cosine Similarity Loss minimizes angular difference, L2 Loss enforces feature similarity in Euclidean space, and Aggregated Contrastive Loss encourages class-level feature separation through class-wise mean pooling. Notably, "applying the contrastive loss later during training helps promote early feature learning while effectively refining class distinctions".

Three components are integrated: Masked Gaussian Modeling (MGM) samples Gaussian subsets, masks them, and reconstructs via L2 loss. Self-Distillation follows the DINO framework with student-teacher networks using EMA updates. Language-Gaussian Alignment uses precomputed language features compressed via autoencoder to regularize self-supervised learning. The high dimensionality problem is addressed by replacing original features with compressed representation learned via an autoencoder.

On ScanNet200, SceneSplat achieves a 5.9% f-mIoU increase over prior work when trained on single sources. On ScanNet++, results show 11.1% f-mIoU improvement while using significantly less training data. Remarkably, inference results can "even be better than using the collected labels", demonstrating that large-scale pretraining is able to filter noise and learn meaningful patterns. For efficiency, SceneSplat requires 0.24 minutes per scene versus 107 minutes for Occam's LGS — approximately 445.8 times faster.

Ablation studies reveal a clear positive trend between input 3DGS quality (PSNR) and resulting mIoU, highlighting the importance of data curation. When comparing 3DGS parameters versus point cloud properties, the model trained on 3DGS parameters consistently outperforms the point cloud variant. The paper also identifies temporal inconsistency in the SAMv2+SigLIP2 pipeline, particularly for large background objects, resulting in corrupted feature fields.

SceneSplat establishes "the first large-scale 3D scene understanding model for indoor environments operating directly on 3D Gaussian splats". It enables "open-vocabulary scene recognition without relying on 2D fusion" through vision-language pretraining, and unlocks "label-free 3DGS pretraining at the scene level" through self-supervised techniques. The approach "achieves state-of-the-art performance in zero-shot semantic segmentation, establishing new benchmarks" for future 3D understanding research.