Batch normalization is a widely used technique for making neural-network training faster and more stable. It normalizes intermediate activations and then applies learned scale and shift parameters, allowing the model to keep the representation it needs.
Normalization inside the network
During training, batch normalization calculates a mean and variance from the current mini-batch. Activations are standardized using these values. Two learnable parameters then let the layer rescale and reposition the normalized output instead of forcing it to remain at a fixed distribution.
Why training often improves
Normalized activations make optimization less sensitive to parameter scale and can support larger learning rates. Gradients often behave more predictably, which helps deeper networks converge. The small variation introduced by different mini-batches can also provide a mild regularization effect.
Training and inference are different
At inference time, a model may receive one example rather than a full batch. Batch normalization therefore uses moving estimates of the mean and variance collected during training. Forgetting to switch the model to evaluation mode can produce unstable or incorrect predictions.
Small batches can be difficult
Very small batches provide noisy statistics. This is common in medical imaging or segmentation, where large images limit memory. Group normalization or layer normalization may be more reliable because they do not depend on statistics from many examples in the batch.
Use it as one design choice
Batch normalization is not automatically necessary in every architecture. Transformers commonly use layer normalization, and some modern convolutional networks use alternative normalization strategies. The right choice depends on batch size, architecture, hardware, and validation results. Understanding its training and inference behavior is more useful than treating it as a layer that should always be added.