We develop a new edge detection algorithm that addresses two important issues in this long-standing vision problem: (1) holistic image training and prediction; and (2) multi-scale and multi-level feature learning. Our proposed method, holistically-nested edge detection (HED), performs image-to-image prediction by means of a deep learning model that leverages fully convolutional neural networks and deeply-supervised nets. HED automatically learns rich hierarchical representations that are important in order to approach the human ability of resolving the challenging ambiguity in edge and object boundary detection. We significantly advance the state-of-the-art on the BSD500 dataset (ODS F-score of .782) and the NYU Depth dataset (.746), with substantially faster speed (0.4 seconds per image on GPU).

Edge and boundary detection is fundamental to computer vision, serving applications from visual saliency, segmentation, object detection/recognition, tracking, and motion analysis to modern tasks like autonomous driving and 3D reconstruction. Human consistency studies established an F-score of 0.80 for edge placement, setting a practical upper bound. Prior work falls into three categories: early gradient-based methods (Sobel, Canny), information-theory approaches (Pb, gPb), and learning-based methods with hand-crafted features. Recent CNN approaches showed promise but suffered from slow inference speeds and modest improvements over Structured Edges.

We formalize existing multi-scale deep learning approaches into four categories: multi-stream learning, skip-layer networks, single models on multiple inputs, and independent networks. Our holistically-nested approach differs fundamentally: it uses a single-stream deep network with multiple side outputs, drawing on deeply-supervised net methodology. The training objective combines side-output losses across M layers: L_side(W,w) = Σ_m α_m l_side(W,w^(m)). To handle the severe class imbalance (roughly 90% non-edge pixels), we employ class-balanced cross-entropy loss with weight β = |Y-|/|Y|. A weighted-fusion layer learns to combine multi-scale outputs during training.

The architecture adopts VGGNet but removes the 5th pooling layer and all fully connected layers to improve efficiency and produce meaningful multi-scale outputs. Side outputs connect to layers conv1_2, conv2_2, conv3_3, conv4_3, and conv5_3, with receptive field sizes ranging from 5 to 196 and strides from 1 to 16. Comparison with FCN variants showed HED outperformed FCN-8s (.697 ODS) and FCN-2s (.738 ODS). Deep supervision proved critical: "training with only the weighted-fusion output loss gives edge predictions that lack discernible order" with many critical edges absent at higher layers.

Training uses Caffe with VGG-16 pre-training, mini-batch size 10, learning rate 1e-6, momentum 0.9, and 10,000 iterations. Consensus sampling uses only pixels labeled positive by at least 3 annotators. Data augmentation rotates images to 16 angles with flipping, creating 32x dataset expansion. On BSDS500, HED achieves ODS F-score of .782, OIS of .804, and AP of .833. Individual side outputs contribute variably: layer 1 reaches .595 ODS, layer 2 achieves .697, layer 3 reaches .738, while layer 5 achieves only .606. Processing takes 0.4 seconds per 320x480 image on GPU, compared to seconds or hours for competing methods. On NYUDv2, the combined RGB-HHA model achieves ODS of .746.

We present an end-to-end edge detection system that achieves state-of-the-art performance on natural images at a speed of practical relevance (0.4 seconds using GPU and 12 seconds using CPU). The method combines fully convolutional networks with deep supervision on a trimmed VGG architecture. The holistically-nested design enables automatic learning of multi-scale, multi-level edge representations, addressing a fundamental challenge in this classic vision task. Code and pre-trained models are publicly released.