Comparing the functional behavior of neural network models, whether it is a single network over time or two (or more) networks during or post-training, is an essential step in understanding what they are learning (and what they are not), and for identifying strategies for regularization or efficiency improvements. Motivated by the need for a statistically grounded, robust, and scalable comparison tool, we propose using Partial Distance Correlation (PDC) as a general-purpose tool for comparing representations in deep learning. We describe how it can be used to compare learned representations across networks, identify redundant layers, and guide network compression. We also show that PDC provides a principled approach to disentangling shared and unique information between representations, and demonstrate its effectiveness on a variety of tasks.

Understanding the internal representations learned by deep neural networks is a fundamental challenge in modern machine learning. A variety of tools have been proposed for this purpose, including Centered Kernel Alignment (CKA), Canonical Correlation Analysis (CCA), and projection-weighted CCA. While these methods have proven useful, they each have limitations in terms of statistical interpretability, sensitivity to sample size, or inability to account for confounding factors. In this work, we argue that Partial Distance Correlation provides a unified, statistically principled framework that addresses many of these limitations.

The key idea behind distance correlation is that it measures both linear and nonlinear dependencies between random vectors, unlike classical correlation which only captures linear associations. Partial distance correlation extends this by controlling for the influence of a third variable, enabling more precise comparisons of representations. This property is particularly valuable when comparing layers across different architectures where batch normalization, residual connections, or other structural elements may introduce confounding correlations.

Distance correlation, introduced by Szekely, Rizzo, and Bakirov (2007), is a measure of dependence between random vectors of arbitrary dimensions. Given random vectors X and Y, the distance covariance is computed by taking the Euclidean distances between all pairs of observations, double centering the resulting distance matrices, and computing their inner product. A key property is that distance correlation equals zero if and only if X and Y are statistically independent, a guarantee that classical Pearson correlation cannot provide. The partial distance correlation further conditions on a third variable Z, removing its confounding influence through a projection operation in the space of doubly centered distance matrices.

Previous approaches for comparing neural network representations include CKA, which computes the Hilbert-Schmidt Independence Criterion between kernel matrices of layer activations, and various forms of CCA that project representations into a shared subspace. CKA, while effective and widely adopted, lacks a formal mechanism to account for confounding variables. CCA-based methods require careful selection of the number of canonical components and can be sensitive to the choice of regularization. PDC addresses both issues simultaneously by operating directly on distance matrices and supporting conditional independence testing.

We introduce three primary applications of PDC in deep learning. First, representation similarity analysis: given two networks (or the same network at different training stages), we compute the PDC between corresponding layer representations to quantify their functional similarity while controlling for the input. Second, layer redundancy detection: by computing the PDC between consecutive layers while controlling for the input, we can identify layers that contribute minimal unique information, providing principled guidance for network pruning. Third, disentanglement analysis: PDC naturally decomposes the shared and unique contributions of representations, enabling the study of what information is shared versus unique across network components.

For computational efficiency, we employ an unbiased estimator of distance covariance that operates on the U-centered distance matrices. Given n observations, the computational cost is O(n^2) for computing pairwise distances and O(n^2) for the double centering, which is manageable for typical batch sizes in deep learning. For larger datasets, we propose a mini-batch approximation scheme that computes PDC over random subsets and averages the results, providing a trade-off between statistical accuracy and computational cost that scales to modern architectures with millions of parameters.

We evaluate PDC across three experimental settings. In the representation comparison setting, we analyze ResNet, VGG, and Vision Transformer architectures trained on ImageNet. PDC reveals that early layers across architectures show higher similarity than later layers, consistent with the hypothesis that low-level features are more universal. Critically, when we control for the input using partial distance correlation, the similarity between architecturally different networks decreases substantially, revealing that much of the apparent similarity is driven by shared input statistics rather than learned representations.

In the layer redundancy experiment, we apply PDC to a pre-trained ResNet-50 and compute the unique information contributed by each residual block. Our analysis identifies several blocks in the middle stages that contribute near-zero unique information, suggesting they can be pruned with minimal accuracy loss. When we remove these blocks, the resulting pruned network retains 97.2% of the original accuracy while reducing parameters by 23%. This demonstrates that PDC provides actionable insights for model compression that go beyond existing heuristic pruning criteria.

For the disentanglement analysis, we examine multi-task networks where a shared backbone feeds into task-specific heads. PDC allows us to quantify how much information each head shares with others versus capturing task-unique patterns. On a joint detection and segmentation model, we find that early heads share 60-80% of their information content, while later heads diverge significantly. This analysis provides guidance for determining the optimal branching point in multi-task architectures, an important design decision that is often made by trial and error.

We have presented Partial Distance Correlation as a versatile, statistically grounded tool for analyzing and comparing representations in deep learning. Through three complementary applications — representation comparison, layer redundancy detection, and disentanglement analysis — we have demonstrated that PDC provides insights that are both theoretically principled and practically actionable. The ability to control for confounding variables distinguishes PDC from existing tools and enables more accurate and nuanced analyses of neural network behavior. We believe that PDC will serve as a valuable addition to the toolkit of researchers seeking to understand, compress, and improve deep neural networks.