We introduce ESPNet, a fast and efficient convolutional neural network for semantic segmentation of high resolution images under resource constraints. ESPNet is based on a new convolutional module, efficient spatial pyramid (ESP), which is efficient in terms of computation, memory, and power. The ESP module decomposes a standard convolution into a point-wise convolution and a spatial pyramid of dilated convolutions, which dramatically reduces the number of parameters and operations while maintaining a large effective receptive field. Our ESPNet model is 22 times faster than the state-of-the-art semantic segmentation network PSPNet, while being 180 times smaller. ESPNet can process high resolution images at 112 fps on a standard GPU and 9 fps on an edge device, making it suitable for real-time applications on resource-constrained devices.

Semantic segmentation requires assigning a class label to each pixel in an image. While significant progress has been made using deep convolutional neural networks, most state-of-the-art models are computationally expensive and require large amounts of memory, making them unsuitable for deployment on edge devices such as mobile phones, drones, and autonomous vehicles. For instance, PSPNet has 65.7 million parameters and runs at about 1 FPS while discharging the battery of a standard laptop at a rate of 77 Watts. The key challenge is to design a network that is both accurate and efficient enough for real-time inference on resource-constrained platforms.

Existing efficient segmentation networks such as ENet and ICNet either sacrifice too much accuracy or still require significant computational resources. Our approach is to design a new convolutional module that is inherently efficient by construction, rather than applying post-hoc compression techniques like pruning or quantization. The ESP module achieves this by factorizing convolutions into point-wise and dilated components, with a hierarchical feature fusion (HFF) strategy to address the gridding artifact caused by dilated convolutions.

Beyond traditional accuracy metrics, we introduce system-level metrics for comprehensively evaluating network performance on edge devices, including GPU frequency sensitivity, warp execution efficiency, memory efficiency, and power consumption. Our ESPNet processes high-resolution RGB images at 112 FPS on a high-end GPU, 21 FPS on a laptop, and 9 FPS on an edge device, while consuming only 1 Watt of average power on the Jetson TX2 at 824 MHz.

The ESP module is based on a reduce-split-transform-merge strategy. First, a 1x1 point-wise convolution reduces the input feature maps from M channels to N/K channels (Reduce). The reduced feature maps are then split into K parallel branches (Split), where each branch applies a 3x3 dilated convolution with dilation rate 2^(k-1) for k = 1, 2, ..., K (Transform). Finally, the outputs of all K branches are concatenated to produce an N-dimensional output (Merge). This factorization reduces parameters by a factor of n^2 MK / (M + n^2 N) while increasing the effective receptive field by approximately [2^(K-1)]^2.

The Hierarchical Feature Fusion (HFF) is a key design choice to address the gridding artifact. In standard spatial pyramid modules, dilated convolutions with large dilation rates produce checkerboard-like patterns due to the sparse sampling of the input. Our HFF strategy hierarchically adds the output of each branch to the output of the previous branch before concatenation, effectively filling the gaps in the receptive field. This results in smooth and complete receptive field coverage without the gridding artifact, and does not increase the complexity of the ESP module.

The network architecture progresses through four variants. ESPNet-A is the baseline that learns representations at multiple spatial levels. ESPNet-B adds long-range connections by concatenating feature maps from strided ESP modules. ESPNet-C introduces input reinforcement by downsampling the original image and concatenating it at each spatial level, compensating for spatial information loss. The full ESPNet attaches a lightweight decoder using a reduce-upsample-merge (RUM) strategy, producing segmentation masks at the original resolution. A depth multiplier alpha controls network depth without changing the topology.

We evaluate ESPNet on the Cityscapes dataset for semantic segmentation. Our model achieves 60.3% class mIoU and 82.2% category mIoU on the test set with only 0.4M parameters. Compared to ENet which achieves 58.3% mIoU with 0.36M parameters, ESPNet achieves 2 percentage points higher accuracy while running 1.27x and 1.16x faster on desktop and laptop, respectively. Compared to PSPNet which achieves 78.4% mIoU with 65.7M parameters, ESPNet has only 8% lower category-wise mIoU while learning 180x fewer parameters.

We further evaluate the real-time capability on edge devices. On the NVIDIA Jetson TX2, ESPNet processes high-resolution Cityscapes images at 9 fps. On a standard NVIDIA TitanX GPU, ESPNet runs at 112 fps. The model size is only 0.7 MB. Comparing convolutional modules, ESP outperforms MobileNet by 7% and ShuffleNet by 12% while learning a similar number of parameters and having comparable network size and inference speed.

Cross-dataset generalization tests on the unseen Mapillary dataset (trained only on Cityscapes) show ESPNet achieving 0.40 mIoU with 0.364M parameters, outperforming ENet (0.33) and ERFNet (0.25). On PASCAL VOC 2012, ESPNet achieves 63.01% mIoU with 0.364M parameters, compared to SegNet's 59.10% with 29.5M parameters (81x fewer). Power consumption on the Jetson TX2 is only 1 W at 824 MHz GPU frequency, compared to ENet's 1.5 W and ERFNet's 2.9 W, with warp execution efficiency about 9% higher than ENet and 14% higher than ERFNet.

We have presented ESPNet, an efficient convolutional neural network for real-time semantic segmentation on resource-constrained devices. The core contribution is the ESP module, which uses a factorized convolution approach with hierarchical feature fusion to achieve a large effective receptive field with minimal computational cost. Beyond traditional accuracy metrics, we introduce system-level metrics for comprehensively evaluating network performance on edge devices. Empirical analysis demonstrates ESPNet achieves compelling trade-offs between accuracy and efficiency while learning generalizable representations across diverse datasets and conditions.