We present RayZer, a model that takes unposed and uncalibrated images as input, recovers camera parameters, reconstructs a scene representation, and synthesizes novel views. The system trains without 3D supervision, using self-predicted camera poses to render target views rather than ground-truth annotations. The framework achieves comparable or even superior novel view synthesis performance than "oracle" methods that require pose annotations. The model employs a ray structure, connecting camera, pixel, and scene simultaneously.

While self-supervised learning has advanced LLMs and VLMs, 3D Vision models still rely heavily on ground-truth 3D geometry and camera pose labels. The authors propose breaking from this paradigm by asking: "How far can we push a 3D Vision model without any 3D supervision?" RayZer processes unposed and uncalibrated multi-view images through three stages: camera parameter recovery, scene representation reconstruction, and novel view rendering. During training, the model uses camera poses predicted by RayZer itself to render views that provide photometric supervision.

The framework interprets the task as "3D-aware image auto-encoding", where input images are disentangled into camera and scene representations, then re-entangled through rendering. The model uses transformers with no 3D representation, no hand-crafted rendering equation, and no 3D-informed architectures — the only 3D prior is the ray structure, incorporating pixel-aligned Plucker ray maps to guide scene reconstruction. Evaluation across three datasets shows RayZer demonstrates comparable or even better novel view synthesis performance than "oracle" methods, suggesting that potentially noisy pose annotations from COLMAP can limit the performance of supervised models.

Large-scale 3D Vision Models like LRM, DUSt3R, and LVSM have incorporated transformers to convert 2D images into 3D representations. However, these approaches still require ground-truth camera poses for supervised training and/or accurate camera annotations during inference. Self-supervised 3D representation learning methods either work for a specific category or can only recover partial observations. The most relevant prior work is RUST, which learns latent scene representations from unposed imagery. RayZer differs in three key aspects: it initially estimates camera poses rather than reconstructing first; it employs explicit pose representations for better information disentanglement; and it uses pure self-attention in transformers.

RayZer's input consists of a set of unposed and uncalibrated multi-view images. To enable self-supervised training, the framework controls the data information flow by splitting images into two non-overlapping subsets. One subset predicts scene representation while the other provides photometric supervision. The loss function combines MSE and perceptual loss. A key design element involves cascaded prediction of camera and scene representations, motivated by the fact that even noisy cameras can be a strong condition for better scene reconstruction, providing mutual regularization during training.

RayZer builds a pure transformer-based model with three components. The Camera Estimator uses learnable camera tokens combined with image features via full self-attention to predict SE(3) camera poses (using continuous 6D rotation representation) and intrinsics. Camera parameters are then converted to pixel-aligned Plucker ray maps. The Scene Reconstructor fuses raw image tokens with ray information via MLP and uses self-attention transformer layers to predict a latent set scene representation z. Critically, it uses raw image tokens rather than pose transformer output to prevent information leakage. The Rendering Decoder represents target images as Plucker rays, fuses them with scene tokens via self-attention, and decodes RGB patches through an MLP.

RayZer is evaluated on DL3DV, RealEstate10k (scene-level), and Objaverse (object-level). The model achieves performance comparable to the best oracle model LVSM and even outperforms LVSM on DL3DV and RealEstate10k while performing slightly worse on Objaverse. The authors conjecture this is because camera poses in DL3DV and RealEstate are annotated by COLMAP, which can be imperfect, while Objaverse has perfect pose annotations from the rendering tool. Analysis shows predicted poses are interpolatable and 3D-aware. Ablation studies verify that the latent set representation, Plucker ray maps, and cascaded pose-first paradigm are all essential. Notably, replacing the latent representation with 3D Gaussian Splatting causes training to not converge, confirming the flexibility advantage of learned representations.

The authors introduce RayZer, a self-supervised large multi-view 3D Vision model trained with zero 3D supervision — no 3D geometry and no camera annotations. Results verify the feasibility of breaking free from supervised learning in 3D vision tasks. The model's ability to learn a pose space that is potentially better suited for novel view synthesis than noisy ground-truth annotations represents a paradigm shift in how we think about 3D supervision.