Rectified activation units (rectifiers) are essential for recent advances in deep neural networks. In this work, we study rectifier neural networks from two aspects. First, we propose Parametric Rectified Linear Unit (PReLU), which generalizes the traditional rectified unit and improves model fitting with nearly zero extra computational cost and little overfitting risk. Second, we derive a robust initialization method that particularly considers the rectifier nonlinearities, enabling us to train extremely deep rectified models directly from scratch. Based on these two techniques, we achieve 4.94% top-5 test error on the ImageNet 2012 classification dataset, which surpasses the reported human-level performance (5.1%) for the first time.

Deep neural networks have recently achieved remarkable progress in visual recognition. A key factor behind this success is the use of Rectified Linear Units (ReLU): f(x) = max(0, x). While ReLU has enabled training of deeper networks compared to sigmoid and tanh, its behavior on the negative side — hard zero activation — leads to "dying neurons" that never activate again once pushed into the negative regime. Moreover, standard initialization methods like Xavier assume linear activations and are not designed for networks with rectifier nonlinearities, causing training difficulties for very deep models.

We propose PReLU, which generalizes ReLU by allowing a learnable slope for negative inputs: f(y_i) = y_i if y_i > 0; a_i · y_i if y_i ≤ 0. The coefficient a_i is learned jointly with the other model parameters through back-propagation, requiring no manual tuning. When a_i = 0, PReLU reduces to ReLU; when a_i is a fixed small value, it becomes Leaky ReLU. We investigate both channel-wise (each channel has its own a_i) and channel-shared (all channels share one a) variants. The channel-wise version on a 14-layer model shows a 1.2% gain over the ReLU baseline. The additional parameters are negligible: only one per channel for channel-wise, adding virtually zero computational cost.

We derive a new initialization method specifically designed for rectifier networks. The key insight is that the variance of responses in each layer must be maintained during both forward and backward passes. For ReLU networks, the proper standard deviation for weight initialization should be √(2/n_l), where n_l is the number of input connections, rather than Xavier's √(1/n_l) which assumes linear activations. The factor of 2 accounts for the fact that ReLU zeros out roughly half of the activations. This seemingly simple correction has dramatic practical impact: it enables training extremely deep rectified models (30+ layers) directly from scratch without the need for pre-training or other tricks, whereas Xavier initialization causes such models to fail to converge.

We train a series of deep models on ImageNet 2012 using PReLU and our initialization. Model A (19-layer) achieves 6.28% top-5 error with PReLU; Model B (22-layer) achieves 6.27%; Model C (wider variant) achieves 5.71% single-model top-5 error. The multi-model ensemble reaches 4.94% top-5 test error, representing a 26% relative improvement over ILSVRC 2014 winner GoogLeNet (6.66%). This is the first result to surpass the reported human-level performance of 5.1%. Our models use spatial pyramid pooling, aggressive data augmentation with scale jittering in range [256, 512], and multi-GPU training. The paper emphasizes that width increases (more filters per layer) contribute more than depth alone.

We have investigated rectifier neural networks from the perspectives of activation functions and initialization. PReLU provides a principled generalization of ReLU with negligible extra cost, while our initialization method enables training of very deep models that were previously intractable. Together, these contributions achieve 4.94% top-5 error on ImageNet, surpassing human-level performance for the first time. These results demonstrate that careful attention to seemingly minor design choices — activation functions and initialization — can yield substantial practical improvements in deep learning.