We introduce pixelSplat, a feed-forward model that learns to reconstruct a 3D Gaussian splat parameterization of a 3D scene from a pair of images. Our model features real-time and memory-efficient rendering for scalable training as well as fast 3D reconstruction at inference time. To overcome local minima inherent to sparse representations, we predict dense probability distributions over 3D and sample Gaussian means from these distributions. We make this sampling operation differentiable via a reparameterization trick, allowing us to back-propagate gradients through the Gaussian splatting representation. We benchmark our method on wide-baseline novel view synthesis on the RealEstate10k and ACID datasets, where we outperform state-of-the-art light field transformers and accelerate rendering by 2.5 orders of magnitude while reconstructing an interpretable and editable 3D radiance field.

The problem of generalizable novel view synthesis from sparse image observations has seen tremendous progress through differentiable rendering. However, differentiable rendering inherits a key weakness: training, reconstruction, and rendering are notoriously memory- and time-intensive because differentiable rendering requires evaluating dozens or hundreds of points along each camera ray. While light-field transformers represent progress, they remain far from real-time and do not reconstruct 3D scene representations that can be edited or exported for downstream tasks in vision and graphics.

We present pixelSplat, which brings the benefits of a primitive-based 3D representation — fast and memory-efficient rendering as well as interpretable 3D structure — to generalizable view synthesis. We identify two key challenges. First, scale ambiguity: in real-world datasets, camera poses are only reconstructed up to an arbitrary scale factor. We design a multi-view epipolar transformer that reliably infers this per-scene scale factor. Second, local minima in optimization: optimizing primitive parameters directly via gradient descent suffers from local minima. While single-scene methods use non-differentiable pruning and division heuristics, the generalizable case requires gradient flow, precluding such operations.

Our proposed solution: we parameterize the positions (i.e., means) of Gaussians implicitly via dense probability distributions predicted by our encoder. In each forward pass, we sample Gaussian primitive locations from this distribution, enabling implicit spawning or deletion of primitives during training while maintaining gradient flow through a reparameterization trick. We demonstrate that a 3D Gaussian splatting representation can be predicted in a single forward pass from just a pair of images, showing that a Gaussian splatting representation can be turned into a building block of any end-to-end differentiable system. Results show significant outperformance over light field transformers on real-world datasets while drastically reducing both training and rendering cost.

Recent advances in single-scene novel view synthesis employ neural fields and volume rendering as standards. However, these methods suffer from high computational demands requiring dozens of queries of the neural field per ray. Discrete data structures accelerate rendering but still significantly fall short of real-time rendering at high resolutions. The recently proposed 3D Gaussian splatting offers an efficient alternative by sparsely representing the radiance field with 3D Gaussian primitives that can be rendered via rasterization. However, single-scene optimization requires dozens of images per scene. Our work trains neural networks to estimate 3D Gaussian primitive scene representation from just two images in a single forward pass.

The generalizable novel view synthesis direction seeks 3D reconstruction and novel view synthesis from only a handful of images per scene. Neural networks can be trained to regress multi-plane images for small-baseline synthesis, but large-baseline novel view synthesis requires full 3D representations. Methods preserving end-to-end locality and shift equivariance between encoder and scene representation via pixel-aligned or otherwise local features enabled generalization to room-scale scenes and beyond. Recent work combines encoders with cost volumes to match features across views, inspired by classical multi-view stereo. While methods inferring interpretable 3D representations exist, recent light field scene representations trade interpretability for faster rendering. Our method presents the best of both worlds: it infers an interpretable 3D scene representation in the form of 3D Gaussian primitives while accelerating rendering by three orders of magnitude compared to light field transformers.

Prior work recognizes the challenge of scene scale ambiguity in machine learning for multi-view geometry. In monocular depth estimation, state-of-the-art models rely on sophisticated scale-invariant depth losses. In novel view synthesis, single-image 3D diffusion models trained on real-world data rescale 3D scenes according to heuristics on depth statistics and condition their encoders on scene scale. We instead build a multi-view encoder that can infer the scale of the scene, using an epipolar transformer that finds cross-view pixel correspondences and associates them with positionally encoded depth values.

3D Gaussian Splatting parameterizes a 3D scene as a set of 3D Gaussian primitives {g_k = (mu_k, Sigma_k, alpha_k, S_k)}, where each has a mean mu_k, a covariance Sigma_k, an opacity alpha_k, and spherical harmonics coefficients S_k. These primitives parameterize the 3D radiance field and can be rendered via an inexpensive rasterization operation. Unlike dense neural field representations that require evaluating dozens of samples per ray, this approach is significantly cheaper in terms of time and memory.

Fitting 3D-GS models relates to fitting Gaussian mixture models, where we seek the parameters of a set of Gaussians that maximize the likelihood of a set of samples. This problem is famously non-convex. Local minima arise when Gaussian primitives initialized at random locations have to move through space to arrive at their final location. Two issues prevent convergence: first, Gaussian primitives have local support, meaning gradients vanish if the distance to the correct location exceeds more than a few standard deviations. Second, even if a Gaussian is close enough to receive substantial gradients, there still needs to exist a monotonically decreasing loss path to its final location. 3D-GS addresses this via non-differentiable pruning and splitting operations, which are incompatible with the generalizable setting where primitive parameters are predicted by a neural network that must receive gradients.

In practice, structure-from-motion (SfM) reconstructs each scene only up to scale, meaning scenes are scaled by individual, arbitrary scale factors s_i. A given scene provides camera data C_i = {(I_j, s_i * T_j)}, where s_i * T_j denotes a metric pose whose translation component is scaled by the unknown scalar s_i. Critically, recovering s_i from a single image is impossible due to the principle of scale ambiguity. This means a neural network making predictions about the geometry of a scene from a single image cannot possibly predict the depth that matches the poses reconstructed by structure-from-motion.

Our solution uses two reference views. For each pixel in image I, we annotate points along its epipolar line in the second image with their corresponding depths in I's coordinate frame. These depth values are computed from camera poses and encode the scene's scale s_i. Each view is encoded separately into feature volumes F and F-tilde. For pixel coordinate u from I, the epipolar line in the second image is sampled at coordinates {u-tilde_l}. For each epipolar sample, the distance to I's camera origin is computed by triangulation. Queries, keys, and values for epipolar attention are then computed as: s = F-tilde[u-tilde_l] concatenated with gamma(d-tilde), where gamma denotes positional encoding. We then perform epipolar cross-attention: F[u] += Att(q, {k_l}, {v_l}).

After the epipolar cross-attention layer, each pixel feature F[u] contains a weighted sum of the depth positional encodings, where we expect that the correct correspondence gained the largest weight, and thus each pixel feature now encodes the scaled depth that is consistent with the arbitrary scale factor s_i. Following epipolar cross-attention, a residual convolution layer F += Conv(F) and a per-image self-attention layer F += SelfAttention(F) are applied. These enable our encoder to propagate scaled depth estimates to parts of the image feature maps that may not have any epipolar correspondences, such as occluded or textureless regions.

The method leverages scale-aware feature maps to predict Gaussian primitive parameters. Every pixel in an image samples a point on a surface in the 3D scene. We thus choose to parameterize the scene via pixel-aligned Gaussians: for each pixel at coordinate u, we take the corresponding feature F[u] as input and predict the parameters of M Gaussian primitives. The baseline approach directly regresses the Gaussian center mu = o + d_u * d, where d = g(F[u]) and the ray direction d is computed from camera extrinsics T and intrinsics K. However, directly optimizing Gaussian parameters is susceptible to local minima, and we cannot rely on spawning and pruning heuristics since they require gradients.

Rather than predicting depth d directly, our method predicts the probability distribution of the likelihood that a Gaussian (i.e. surface) exists at a depth d along the ray u. This is implemented as a discrete probability density over a set of depth buckets. Near and far planes d_near and d_far are set, and depth is discretized into Z bins represented by vector b, where each element is defined in disparity space. A discrete probability distribution p_phi(z) is defined over index variable z, where phi_z is the probability that a surface exists in depth bucket b_z. Probabilities are predicted by a fully connected neural network f from per-pixel feature F[u] and normalized via softmax. A per-bucket center offset delta is also predicted, allowing fine-grained depth adjustment within each bucket.

To backpropagate gradients into depth bucket probabilities phi, the sampling operation z ~ p_phi(z) must be made differentiable. However, the sampling operation is inherently not differentiable. We employ a reparameterization trick inspired by variational autoencoders: we set the opacity alpha of a Gaussian to be equal to the probability of the bucket that it was sampled from. If z ~ p_phi(z), then alpha = phi_z. This means in each backward pass, we assign the gradients of the loss with respect to the opacities to the gradients of the depth probability buckets. Intuitively: if sampling produces a correct depth, gradient descent increases the opacity, leading it to be sampled more often, eventually concentrating probability mass in the correct bucket. If an incorrect depth is sampled, gradient descent decreases the opacity, lowering the probability of further incorrect predictions.

The method is trained and evaluated on RealEstate10k, a dataset of home walkthrough videos downloaded from YouTube, and ACID, a dataset of aerial landscape videos. Both include camera poses computed by SfM software, necessitating the scale-aware design. Three novel-view-synthesis baselines are compared: pixelNeRF (conditions neural radiance fields on 2D image features), GPNR (image-based light field rendering using epipolar lines), and Du et al. (combines light field rendering with epipolar transformer). To present a fair comparison, all baselines were retrained using the same data loaders and training curriculum, where the inter-frame distance between reference views is gradually increased during training. Evaluation uses PSNR, SSIM, and LPIPS metrics. Implementation employs a ResNet-50 and a ViT-B/8 vision transformer, both pre-trained using DINO objective, with outputs summed pixel-wise.

Quantitative results demonstrate that pixelSplat outperforms all baselines on every metric. On ACID, pixelSplat achieves 28.14 dB PSNR; on RealEstate10k, it achieves 25.89 dB PSNR, with especially significant LPIPS improvements. Compared to Du et al., improvements are 1.26 dB on ACID and 1.11 dB on RealEstate10k. Qualitative results demonstrate that the method not only produces more accurate and perceptually appealing images, but also generalizes better to out-of-distribution examples. The method is also significantly less resource-intensive: for a single scene encoding followed by 100 image renderings (approximate sequence length), the cost is about 650 times less than the next-fastest baseline.

Point cloud visualizations from out-of-distribution views show the method produces structured 3D representations. However, the authors acknowledge that while the resulting Gaussians facilitate high-fidelity novel-view synthesis for in-distribution camera poses, they suffer from the same failure modes as 3D Gaussians optimized using the original 3D Gaussian splatting method. Specifically, reflective surfaces are often transparent, and Gaussians appear billboard-like when viewed from out-of-distribution views. These limitations are inherent to the Gaussian splatting representation itself, not unique to the proposed method.

The ablation study addresses three questions. Q1a: Is the epipolar encoder responsible for handling scale ambiguity? Comparing against a variant without epipolar encoding shows significant performance drops. Qualitatively, this produces ghosting and motion blur artifacts that are evidence of incorrect depth predictions. Visualization of epipolar attention scores demonstrates that the epipolar transformer successfully discovers cross-view correspondences. Q1b: Does frequency-based positional encoding of depth matter? Testing without depth encoding yields a performance drop of approximately 2 dB PSNR, highlighting that beyond simply detecting correspondence, the encoder uses the scene-scale encoded depths it triangulates to resolve scale ambiguity.

Q2: Does probabilistic depth prediction alleviate local minima? Ablating the sampling approach and directly regressing depth shows a performance drop of approximately 1 dB in PSNR, validating that predicting the depth of a Gaussian probabilistically is necessary. Detailed ablation metrics on RealEstate10k: Full model achieves 25.89 PSNR / 0.858 SSIM / 0.142 LPIPS; without epipolar encoder: 19.98 / 0.641 / 0.289; without depth encoding: 24.09 / 0.803 / 0.181; without probabilistic sampling: 24.44 / 0.820 / 0.175; with additional depth regularization: 25.28 / 0.847 / 0.152. These results confirm that every proposed component contributes to the final performance.

We have introduced pixelSplat, a method that reconstructs a primitive-based parameterization of the 3D radiance field of a scene from only two images. Our method is significantly faster than prior work on generalizable novel view synthesis while producing an explicit 3D scene representation. To solve local minima in primitive-based function regression, we introduced a novel parameterization of primitive location via a dense probability distribution and a novel reparameterization trick to backpropagate gradients into the parameters of this distribution. This framework is general, and we hope that our work inspires follow-up work on prior-based inference of primitive-based representations across applications.

Future directions include leveraging our model for generative modeling by combining it with diffusion models, or removing the need for camera poses to enable large-scale training on uncurated internet data. The ability to predict 3D Gaussian representations in a single forward pass opens opportunities for real-time 3D reconstruction in augmented reality and robotics applications, where both speed and interpretable 3D structure are critical requirements.