Batch Normalization (BN) is a milestone technique in the development of deep learning, enabling various networks to train. However, normalizing along the batch dimension introduces problems — BN's error increases rapidly when the batch size becomes smaller, caused by inaccurate batch statistics estimation. This limits BN's usage for training larger models and transferring features to computer vision tasks including detection, segmentation, and video, which require small batches constrained by memory consumption. In this paper, we present Group Normalization (GN) as a simple alternative to BN. GN divides the channels into groups and computes within each group the mean and variance for normalization. GN's computation is independent of batch sizes, and its accuracy is stable in a wide range of batch sizes.

Batch Normalization has been established as a very effective component in deep learning. BN normalizes the features by the mean and variance computed within a (mini-)batch. This has been shown by many practices to ease optimization and enable very deep networks to converge. BN has been a foundation of many competition-winning entries and important methodologies developed since 2015.

Despite its great success, BN exhibits drawbacks that are also caused by its distinct behavior of normalizing along the batch dimension. In particular, BN requires sufficiently large batch sizes to work effectively (e.g., 32 per worker). A small batch leads to inaccurate estimation of the batch statistics, and reducing BN's batch size increases the model error dramatically. This limits many visual recognition tasks that require large memory, such as object detection, semantic segmentation, and video recognition.

Prior normalization methods include Layer Normalization (LN), Instance Normalization (IN), and Weight Normalization. LN operates along the channel dimension for each sample, and has been adopted in recurrent networks and Transformers. IN was originally designed for style transfer. However, LN and IN are less effective than BN for visual recognition tasks, because they either normalize over too many or too few channels. GN can be viewed as a natural interpolation between LN and IN.

The central idea of Group Normalization is simple: we divide channels into groups and normalize within each group. The pixels in the same group are normalized together by the shared mean and variance. Formally, GN computes mean and variance for the set of pixels S_i defined by: S_i = {k | floor(k_C/G) = floor(i_C/G)}, where G is the number of groups, C is the number of channels, and k_C and i_C index the channel dimension. When G = 1, GN becomes equivalent to LN; when G = C, GN becomes equivalent to IN.

The motivation behind grouping channels can be understood from classical feature engineering. Features like SIFT, HOG, and GIST are group-wise representations by design, where each group of channels is constructed by some kind of histogram. These features are often processed by normalization within each histogram or each orientation. GN inherits this group-wise normalization philosophy and applies it to learned deep features, where the grouping is not manually defined but implicitly learned by the network.

On ImageNet classification with ResNet-50, GN achieves comparable results to BN (only 0.5% lower top-1 error) when using a regular batch size of 32. Critically, when the batch size decreases, BN's error increases by 10.6% when using a batch size of 2, while GN's error increases by only 1.0%. This validates the core thesis that GN is robust to batch size variations.

For object detection and segmentation on COCO using Mask R-CNN, GN outperforms BN when the batch size is small (as is typical in these tasks). Specifically, GN achieves box AP of 40.3 and mask AP of 36.4, compared to BN's 39.2 and 35.4 respectively when using a batch of 2 images per GPU. GN also shows strong results in video classification on Kinetics, demonstrating its broad applicability across visual recognition tasks.

We have presented Group Normalization (GN) as a simple yet effective normalization method that is independent of batch sizes. GN can naturally transfer from pre-training to fine-tuning. We hope GN will become a powerful alternative to BN, especially for tasks that are constrained by small batch sizes. Given its simplicity and effectiveness, GN has the potential to become a fundamental building block in deep learning.