Siamese networks have become a common structure in various recent models for unsupervised visual representation learning. These models maximize the similarity between two augmentations of one image, subject to certain conditions for avoiding collapsing solutions. In this paper, the authors report surprising empirical results that simple Siamese networks can learn meaningful representations even without any of the following: (i) negative sample pairs, (ii) large batches, (iii) momentum encoders. The key finding is that a stop-gradient operation plays an essential role in preventing collapsing. The authors hypothesize that SimSiam implicitly involves an expectation-maximization (EM) like algorithm.

Siamese networks are "a natural and effective tool for learning visual representations" by comparing related samples. Recent methods have employed various strategies to prevent representation collapse: contrastive learning uses negative pairs to repel dissimilar representations; BYOL relies on a momentum encoder; SwAV uses online clustering with the Sinkhorn-Knopp algorithm. These increasing complexities raise a fundamental question: what is the minimal set of mechanisms needed to prevent collapse in Siamese representation learning?

The collapsing problem — where all outputs converge to a constant — is the central challenge. Without explicit prevention mechanisms, a Siamese network can trivially minimize the similarity loss by "outputting the same constant vector for all inputs." Existing solutions add complexity: contrastive losses require careful negative sampling and large batch sizes (e.g., 4096 in MoCo); momentum encoders add architectural asymmetry and hyperparameter sensitivity. The authors show that none of these are necessary — a simple stop-gradient operation suffices.

Contrastive learning methods like SimCLR and MoCo learn representations by "pulling positive pairs together and pushing negative pairs apart." SimCLR requires very large batch sizes (up to 8192) for sufficient negative samples. MoCo uses a momentum-updated queue to decouple batch size from the number of negatives. BYOL demonstrated that "negative pairs are not necessary" by using a momentum encoder and a predictor network, but its success was "attributed to the momentum encoder, leaving the role of each component unclear."

SimSiam takes two randomly augmented views x1, x2 of the same image and processes them through a shared encoder f (e.g., ResNet) followed by a projection MLP head h. One branch additionally applies a prediction MLP head p, creating an asymmetry. The loss is a negative cosine similarity: D(p1, z2) = -p1 / ||p1|| * z2 / ||z2||, symmetrized over both views. Critically, the loss does not use negative pairs, momentum encoders, or large batches. A batch size of 256 works well.

The stop-gradient (stopgrad) operation is applied to the branch without the predictor: the loss becomes D(p1, stopgrad(z2)). This means "z2 is treated as a constant in this term" and does not receive gradients from this loss component. The authors demonstrate through controlled experiments that removing stop-gradient immediately leads to collapsing — the training loss reaches its minimum (-1) and the representation becomes a constant. The output standard deviation drops to zero within a few epochs. This simple operation is "critical for preventing collapse — it is the only mechanism needed."

The authors provide a hypothesis based on an expectation-maximization (EM) framework. They consider the loss as involving two sets of variables: the network parameters theta and an implicit set of representation targets eta. The stop-gradient operation can be interpreted as "alternating between optimizing theta with eta fixed (one gradient step) and updating eta given theta (the stop-gradient target)." This resembles the E-step and M-step of EM algorithms. While not a formal proof, this framework "provides a plausible explanation for why SimSiam avoids collapse" — the alternating optimization naturally prevents trivial solutions.

SimSiam is evaluated on ImageNet linear evaluation and various transfer learning benchmarks. With a ResNet-50 backbone and 100-epoch pre-training, SimSiam achieves 68.1% top-1 accuracy under linear evaluation, competitive with SimCLR (66.5%) and MoCo v2 (67.4%), and close to BYOL (68.8%). With 200-epoch training, SimSiam reaches 70.0%. On COCO object detection and instance segmentation, SimSiam-pretrained models achieve results on par with supervised pre-training. Ablation studies show the predictor MLP and stop-gradient are both essential; removing either causes complete collapse. Batch size can be as small as 64 with minimal degradation, a significant advantage over SimCLR's requirement of 4096+.

This work shows that simple Siamese representation learning, with neither negative pairs nor momentum encoders, can achieve competitive results. The stop-gradient operation is identified as the key to avoiding collapse. An EM-like hypothesis provides a plausible theoretical framework. The simplicity of SimSiam makes it "a useful baseline for understanding self-supervised representation learning" and suggests that Siamese networks are a natural and effective tool that deserves further investigation.