A diffusion model learns to predict a vector field of gradients. We propose to apply the chain rule on the learned gradients, and back-propagate the score of a diffusion model through the Jacobian of a differentiable renderer, which we instantiate to be a voxel radiance field. This approach aggregates 2D scores at multiple camera viewpoints into a 3D score, and repurposes a pretrained 2D model for 3D data generation. We identify a technical challenge of distribution mismatch that arises in this application, and propose a novel estimation mechanism to resolve it. We run our algorithm on several off-the-shelf diffusion image generative models, including the recently released Stable Diffusion trained on the large-scale LAION dataset.

Diffusion models have emerged as a powerful class of generative models, achieving state-of-the-art results in image generation, inpainting, super-resolution, and text-to-image synthesis. These models learn to reverse a gradual noising process by predicting a score function — the gradient of the log probability density with respect to the data. The success of large-scale text-conditioned models such as DALL-E 2, Imagen, and Stable Diffusion has demonstrated the remarkable capability of diffusion models to capture complex visual distributions when trained on billions of image-text pairs.

However, training 3D generative models directly is fundamentally more challenging. The scarcity of large-scale, high-quality 3D datasets compared to their 2D counterparts presents a significant bottleneck. While neural radiance fields (NeRF) and other differentiable rendering techniques have enabled impressive 3D reconstruction from multi-view images, these methods typically require per-scene optimization and do not generalize to open-domain 3D generation. A natural question arises: can we leverage the rich visual priors captured by large-scale 2D diffusion models to guide the creation of 3D content?

In this work, we introduce Score Jacobian Chaining (SJC), a principled framework that lifts pretrained 2D diffusion models for 3D generation by chaining the score function through the Jacobian of a differentiable renderer. Our key insight is that a 3D score can be computed as the vector-Jacobian product of the 2D score and the renderer Jacobian, aggregated over different camera viewpoints. We instantiate the 3D representation as a voxel radiance field and optimize it by following the estimated 3D score. To address the out-of-distribution problem that arises when evaluating the score on clean rendered images, we propose Perturb-and-Average Scoring (PAAS), which perturbs the rendered image with noise and averages the resulting scores to obtain a robust estimate.

Diffusion probabilistic models, also known as score-based generative models, define a forward process that gradually corrupts data with Gaussian noise and a reverse process that learns to denoise. The score function — the gradient of the log data density — is the central object learned by these models. Denoising score matching provides a tractable training objective, and recent advances in architecture design and training strategies have led to diffusion models surpassing GANs on image generation benchmarks such as FID on ImageNet. Classifier-free guidance has further enabled high-quality text-conditioned generation, powering systems like DALL-E 2, Imagen, and Stable Diffusion.

Neural implicit representations such as NeRF represent scenes as continuous functions mapping 3D coordinates to radiance and density. While highly effective for novel view synthesis from posed images, extending these to generative modeling has proven challenging. 3D-aware GANs such as pi-GAN, EG3D, and GRAF incorporate neural radiance fields into adversarial training but are typically limited to single object categories and require category-specific training data. Point-E and Shap-E train diffusion models directly on 3D representations, but require large paired 3D datasets that are expensive to curate.

Concurrent with our work, DreamFusion proposes Score Distillation Sampling (SDS) to optimize a NeRF using gradients derived from a pretrained Imagen text-to-image diffusion model. While DreamFusion and SJC share the high-level goal of lifting 2D diffusion priors to 3D, they differ in their theoretical foundations. DreamFusion derives SDS from a probabilistic density distillation perspective, whereas SJC provides a score-based derivation grounded in the chain rule of calculus, yielding a clear mathematical interpretation of the 2D-to-3D lifting process. Our formulation predated DreamFusion's public release and offers a complementary theoretical lens on this family of techniques.

Consider a 3D scene parameterized by theta and a differentiable renderer g that maps the scene parameters to a 2D image: x = g(theta). A pretrained 2D diffusion model provides the score function of the image distribution: the gradient of log p(x) with respect to x. Our goal is to obtain the 3D score — the gradient of log p(theta) with respect to theta — so that we can optimize the 3D scene to match the distribution learned by the 2D model. By the chain rule of calculus, the 3D score decomposes as the product of the Jacobian of the renderer and the 2D score.

To account for the fact that a single 2D view provides only partial information about the 3D scene, we aggregate the score over multiple camera viewpoints. Specifically, for each optimization step, we sample a camera pose pi, render the scene from that viewpoint to obtain a 2D image x_pi = g(theta, pi), compute the 2D score at x_pi, and back-propagate it through the renderer Jacobian. The aggregated gradient across viewpoints provides a Monte Carlo estimate of the full 3D score, steering the 3D scene parameters toward a configuration that appears realistic from all angles according to the 2D diffusion prior.

It is instructive to compare our formulation with Score Distillation Sampling (SDS) from DreamFusion. Both approaches yield similar update rules in practice: the gradient used to update the 3D parameters involves back-propagating a 2D diffusion model's prediction through the renderer. However, the derivation paths differ fundamentally. SDS is motivated by minimizing a KL divergence between the rendered image distribution and the diffusion model's learned distribution, dropping a Jacobian term that corresponds to the score of the rendering distribution. SJC derives the same gradient from the chain rule perspective, providing a direct probabilistic interpretation: the optimization follows the score of the 3D distribution induced by the 2D prior through the rendering process.

A critical technical challenge arises when applying the above framework in practice. The diffusion model's denoiser is trained to operate on noisy inputs — images corrupted by Gaussian noise at various levels. However, in our pipeline, the renderer produces clean, noise-free images. When we directly evaluate the denoiser on these clean renderings, it receives out-of-distribution inputs that it has never encountered during training. The pixel values of rendered images stay within a bounded range (e.g., [-1, 1]), while the training data for the denoiser exists in a numerically larger range due to the added noise. This distribution mismatch causes the score estimates to be unreliable and can lead to catastrophic failure in the optimization.

To understand this more concretely, recall that the denoiser is trained via denoising score matching at noise level sigma: given a clean image x_0 and a noisy version x_sigma = x_0 + sigma * epsilon, the denoiser learns to predict x_0 from x_sigma. At noise level sigma approaching zero, the denoiser has essentially only seen noisy inputs during training. Evaluating it on a clean rendered image x = g(theta) is therefore extrapolation beyond the training distribution. This problem is not merely theoretical — in our experiments, naive score evaluation on clean images produces incoherent, artifact-ridden 3D outputs.

To resolve the distribution mismatch, we propose Perturb-and-Average Scoring (PAAS). The idea is intuitive: instead of evaluating the score directly on a clean rendered image, we first perturb the image by adding Gaussian noise, then evaluate the score on the noisy version, and finally average over multiple noise perturbations. Formally, given a rendered image x = g(theta), we compute the score estimate as the expectation of the score evaluated at x + sigma * epsilon, where epsilon is drawn from a standard Gaussian. This ensures that the denoiser always receives inputs within its training distribution, while the averaging recovers a valid score estimate for the original clean image.

Mathematically, we show that PAAS approximates the score at an inflated noise level of square root of 2 times sigma. This can be understood as follows: perturbing the rendered image with noise at level sigma and then evaluating the denoiser (which was trained at noise level sigma) is equivalent to evaluating the score of a smoothed version of the data distribution. The key theoretical contribution is proving that this averaged score provides a consistent estimator of the true score, up to the smoothing induced by the noise perturbation. In practice, we adopt a coarse-to-fine annealing schedule for sigma, starting with large noise levels for global structure and gradually reducing it for finer details.

The PAAS mechanism also connects to Tweedie's formula in statistics. Under the Gaussian perturbation model, the denoiser's output can be interpreted via Tweedie's formula as the posterior mean estimate of the clean image. The score is then recovered as (denoiser output minus noisy input) divided by sigma squared. This interpretation provides additional theoretical grounding for our approach and connects it to a well-established statistical framework. The resulting algorithm is straightforward to implement: render, perturb, denoise, compute score, and back-propagate through the renderer.

We represent the 3D scene as a voxel radiance field — a dense 3D grid where each voxel stores an opacity value and a color (or feature) vector. This representation is chosen for its simplicity and full differentiability: the rendering operation (ray marching through the voxel grid) is straightforward to implement with automatic differentiation, and the Jacobian of the renderer with respect to the voxel parameters is well-defined everywhere. We use a grid resolution of 64 cubed in our default configuration, which balances detail and computational cost.

The optimization is performed using gradient descent on the voxel parameters, guided by the SJC score at each step. We employ text-conditioned generation via classifier-free guidance when using text-to-image diffusion models: the score is computed as a weighted combination of the conditional and unconditional score estimates, with a guidance scale of 100 found to work well empirically. The optimization typically runs for 10,000 iterations with the Adam optimizer. We also apply total variation regularization on the voxel grid to encourage spatial smoothness and reduce noise artifacts. The entire pipeline requires no 3D training data — only a pretrained 2D diffusion model and a text prompt.

For the diffusion backbone, we demonstrate results using several off-the-shelf models: DeepFloyd-IF, Stable Diffusion v1.5, and a custom-trained model on specific categories. When using Stable Diffusion, which operates in a latent space via a VAE encoder-decoder, we apply the score Jacobian chaining through both the decoder and the diffusion model, effectively chaining three Jacobians: the voxel renderer, the VAE decoder, and the diffusion UNet. This demonstrates the generality of the chain rule approach — it naturally extends to multi-stage generative pipelines without requiring architectural modifications.

We evaluate SJC on text-guided 3D generation using Stable Diffusion as the backbone. Given text prompts such as "a DSLR photo of a peacock on a surfboard," "a car made out of sushi," and "a temple", our method produces coherent 3D voxel representations that can be rendered from novel viewpoints. The generated objects exhibit recognizable geometry and plausible texture that align with the text descriptions. We observe that the coarse-to-fine noise schedule is critical: early iterations with high noise levels establish the global shape, while later iterations with lower noise refine surface details and textures.

We compare SJC with DreamFusion both qualitatively and in terms of computational efficiency. Since DreamFusion uses the proprietary Imagen model (not publicly available), direct comparison requires care. Using the same Stable Diffusion backbone for both methods, SJC produces results of comparable visual quality. Notably, SJC's voxel-based representation enables faster optimization than NeRF-based representations used in DreamFusion, as voxel rendering avoids the per-ray network queries that dominate NeRF's computational cost. On the other hand, the fixed voxel resolution limits the geometric detail achievable by SJC compared to the continuous NeRF representation.

Ablation studies validate the importance of key design choices. Without PAAS (i.e., evaluating the score directly on clean images), the optimization diverges rapidly, producing noisy and incoherent 3D structures. This confirms the severity of the distribution mismatch problem and the necessity of our proposed solution. Removing total variation regularization leads to noisier voxel grids with floating artifacts. The noise annealing schedule also proves critical: using a fixed noise level throughout optimization produces either overly smooth results (high sigma) or noisy artifacts (low sigma), while the annealing schedule captures both global structure and fine detail.

We have presented Score Jacobian Chaining, a method that lifts pretrained 2D diffusion models to 3D generation by applying the chain rule to propagate learned 2D scores through a differentiable renderer. Our framework provides a principled, score-based perspective on the emerging paradigm of distilling 2D generative priors for 3D content creation. The identification of the out-of-distribution problem and its resolution via PAAS constitute important practical contributions that enable reliable optimization.

Limitations remain. The voxel representation constrains the geometric resolution, and generated objects sometimes exhibit the "Janus problem" — multi-faced artifacts where the model generates a front-facing view from multiple angles due to the lack of explicit view-dependent reasoning in the 2D diffusion prior. The optimization process is also computationally expensive, requiring several hours on a single GPU. Future work may address these through higher-capacity 3D representations, view-conditioned diffusion models, and more efficient sampling strategies. We believe the score-based formulation opens doors to incorporating additional priors — such as depth, normal, or symmetry constraints — in a principled manner.

In summary, Score Jacobian Chaining demonstrates that the mathematical structure of diffusion models — specifically, their learning of score functions — can be systematically leveraged for 3D generation through the elegant application of the chain rule. As 2D diffusion models continue to improve in quality and diversity, methods like SJC that bridge the 2D-3D gap without requiring 3D training data will become increasingly valuable for content creation, virtual reality, robotics, and other 3D-centric applications. The theoretical clarity of the score-based perspective may also inspire new connections between generative modeling, differentiable rendering, and inverse problems more broadly.