We present KDSS (Knowledge Distillation meets Self-Supervision), a framework that bridges knowledge distillation and self-supervised learning to produce compact yet powerful student networks. Traditional knowledge distillation transfers knowledge from a large teacher to a small student through output mimicry, which may lose structural information. We propose to use self-supervised auxiliary tasks as a bridge: the teacher and student are encouraged to produce similar representations for self-supervised pretext tasks, enriching the distilled knowledge. This achieves state-of-the-art distillation results on ImageNet and CIFAR-100.

Knowledge distillation compresses large models into smaller ones by training the student to mimic the teacher's outputs. Self-supervised learning learns representations from unlabeled data through pretext tasks. These two paradigms have been studied independently. We identify that self-supervised tasks encode complementary structural knowledge that standard distillation misses. By jointly optimizing distillation and self-supervised objectives, the student learns richer representations that better generalize to downstream tasks.

Prior work in knowledge distillation has explored feature-level matching, attention transfer, and relational distillation, but these still rely on task-specific supervision signals. Self-supervised learning has emerged as a powerful paradigm for learning task-agnostic representations that capture geometric, textural, and structural properties of images. We hypothesize that these properties are precisely what standard distillation fails to transfer effectively.

KDSS consists of three loss components: (1) task-specific distillation loss (KL divergence between teacher and student logits), (2) self-supervised contrastive loss that encourages both teacher and student to produce consistent representations under augmentations, and (3) cross-modal distillation loss that aligns teacher and student self-supervised representations. The total loss is a weighted combination of all three, enabling end-to-end training.

The self-supervised pretext tasks include rotation prediction and contrastive instance discrimination. For rotation prediction, both teacher and student must predict the rotation angle applied to the input image. The key insight is that matching rotation predictions requires understanding spatial structure and object orientation, knowledge that goes beyond class labels. The contrastive task further enriches this with instance-level discrimination ability.

The cross-modal distillation operates in the representation space rather than the output space. We extract intermediate feature maps from both teacher and student during self-supervised tasks and minimize their cosine distance. This forces the student to learn similar internal representations to the teacher for structural tasks, providing a richer learning signal than output-level KL divergence alone.

On ImageNet, KDSS improves ResNet-18 student accuracy from 69.75% (vanilla KD) to 71.96% with a ResNet-34 teacher, a +2.21% improvement. On CIFAR-100, KDSS achieves 76.45% compared to 74.92% for vanilla KD. Ablation studies confirm that both rotation prediction (+0.8%) and contrastive learning (+1.2%) contribute to the improvement, with their combination providing the full gain.

We also evaluate transfer learning performance by fine-tuning the distilled student on downstream tasks. On VOC detection, KDSS-trained ResNet-18 achieves +1.5% mAP improvement over vanilla KD. On COCO instance segmentation, the improvement is +0.9% AP. These results confirm that KDSS produces more transferable representations than standard distillation.

We have shown that knowledge distillation and self-supervised learning are complementary paradigms that can be unified to produce better compact models. KDSS achieves state-of-the-art distillation results by leveraging self-supervised tasks as a bridge for richer knowledge transfer. This work opens up a new direction at the intersection of model compression and self-supervised learning.