Group Normalization

The problem

The paper starts from the role of batch normalization, which it describes as a milestone technique that enabled many networks to train ^[1]. Batch normalization works by normalizing along the batch dimension ^[1]. That design has a cost: the statistics used for normalization are estimated from the batch, and when the batch is small those estimates are inaccurate ^[1]. The paper states that batch normalization's error increases rapidly as batch size becomes small ^[1].

This is not only a theoretical concern. The paper links the small-batch problem to two practical settings: training larger models, and transferring features to detection, segmentation and video tasks ^[1]. Those tasks need small batches because of memory, so a normalization method that depends on batch statistics is at a disadvantage ^[1]. The problem the paper addresses is therefore how to normalize a network without making the result depend on the batch dimension.

The method

Group Normalization is presented as a simple alternative to batch normalization ^[1]. Instead of computing statistics along the batch dimension, it divides channels into groups and computes the mean and variance within each group ^[1]. Those group statistics are then used in the normalization step. The computation is independent of batch size ^[1].

In practical terms, the method changes where the normalization statistics come from. A batch-normalized layer pools information across examples in a mini-batch; a group-normalized layer pools information across channels inside each group. Because the procedure is defined in terms of channel groups rather than batch size, changing the batch size does not change the normalization computation ^[1]. The paper presents this as a simple design: group the channels, compute a mean and variance for each group, and normalize with them ^[1].

Why it matters

The paper's motivation is closely tied to the conditions under which batch normalization becomes unreliable ^[1]. If a model is trained with small batches, batch statistics are estimated inaccurately, and the error of batch normalization increases rapidly ^[1]. Group Normalization avoids that particular dependence because its computation is independent of batch size ^[1]. This makes it relevant to settings where small batches are forced by memory, including training larger models and transferring features to detection, segmentation and video tasks ^[1].

The paper does not present Group Normalization as a complicated new architecture. It is a normalization layer: it changes how activations are normalized, not the overall network. The paper presents its advantage as simplicity plus independence from batch size ^[1]. For a reader, the key idea is that normalization can be organized around channel groups rather than around the batch, so the same layer can be used without relying on batch statistics.

Paper details

Group Normalization was first submitted to arXiv on 22 March 2018 as 1803.08494 ^[1]. The first author is Yuxin Wu, and Kaiming He is a co-author ^[1].