Saliency detection aims to identify the most visually prominent regions in an image. A fundamental challenge arises when salient foreground or background contains small-scale high-contrast patterns, which can adversely affect detection accuracy. Existing methods that use varying patch sizes or image downsampling to handle scale issues often introduce artifacts or lose fine details. We propose a hierarchical approach that analyzes saliency through a scale-based tree model rather than varying patch sizes or downsampling. Our method constructs a hierarchical segmentation tree and computes saliency at multiple layers, from fine to coarse, then integrates these multi-scale results. This approach improves saliency detection on many images that cannot be handled well by traditional single-scale methods. We also present a newly constructed large-scale benchmark dataset for evaluation.

Visual saliency plays a crucial role in image retargeting, object recognition, image quality assessment, and content-aware editing. The goal is to produce a saliency map where pixel values indicate the probability of belonging to a salient object. Most existing methods compute saliency at a single scale determined by the superpixel or patch size. However, real-world salient objects exhibit complex multi-scale structures: a person wearing a striped shirt has a uniform silhouette at coarse scale but high-contrast internal patterns at fine scale. Single-scale methods may incorrectly mark the stripes as salient boundaries or fail to group them as part of the salient object.

We propose a hierarchical saliency detection framework that operates on a tree-structured image representation. Instead of processing saliency at a single fixed scale, we compute saliency cues at multiple layers of a hierarchical segmentation, from individual superpixels to large merged regions. At each layer, saliency is computed based on region-level contrast and spatial distribution features. The multi-layer results are then integrated through a principled combination scheme that preserves fine details from lower layers while maintaining global coherence from upper layers. Additionally, we introduce a new large-scale dataset with 5,000 images and pixel-accurate ground truth for benchmarking.

Saliency detection methods can be broadly categorized into bottom-up and top-down approaches. Bottom-up methods compute saliency based on low-level visual contrast, including frequency domain analysis, region-based contrast, and graph-based manifold ranking. The influential RC (Region Contrast) method computes saliency as the weighted sum of color distances to all other regions. GC (Global Contrast) uses color histogram distances. However, these methods operate at a fixed segmentation granularity and cannot adapt to the intrinsic scale of salient objects. Some recent works use multi-scale superpixels but simply average or concatenate features across scales without principled integration.

We construct a hierarchical segmentation tree from the input image. The tree is built using an agglomerative clustering process: starting from an initial fine-grained superpixel segmentation (leaf nodes), we iteratively merge adjacent regions based on color similarity and boundary strength, creating parent nodes at each merge step. The resulting tree has L layers (typically 3-5), where layer 0 contains the finest superpixels and layer L-1 contains the coarsest regions. Each node in the tree represents a region at a specific scale, and the parent-child relationships encode how fine-grained regions compose into coarser ones. This tree provides a natural multi-scale representation that respects image boundaries at every level.

At each layer l of the tree, we compute a saliency value for each region based on two complementary cues. The global contrast cue measures the color distinctiveness of a region relative to all other regions at the same layer, weighted by spatial distance to emphasize nearby contrast. The spatial distribution cue measures how compactly a region's color is distributed in the image — salient objects tend to be spatially compact, while background colors are widely scattered. These two cues are combined as S_l(r) = C_l(r) * D_l(r), where C is contrast and D is spatial compactness. Different layers produce saliency maps at different granularities: fine layers capture details but are noisy, while coarse layers provide holistic assessment but lose boundaries.

The final saliency map is obtained by integrating saliency values across all tree layers. For each pixel p, its saliency is computed as a weighted combination of the saliency values from the regions containing p at each layer. The weights are determined by the reliability of each layer, estimated from the distribution of saliency values at that layer — a layer with clear bimodal distribution (salient vs. non-salient) receives higher weight. Additionally, we apply a hierarchical refinement step: saliency from coarse layers provides spatial priors that constrain the saliency computation at finer layers, ensuring that fine-scale details are consistent with the global saliency structure. This bidirectional information flow between scales is the core advantage of our tree-based approach.

We evaluate on four benchmark datasets: the MSRA-B (5,000 images), ECSSD (1,000 images with complex structures), DUT-OMRON (5,168 images), and our newly constructed HKU-IS dataset (4,447 images). We compare against 14 state-of-the-art methods including RC, GC, SF, MC, and DSR. Using the standard precision-recall curve and F-measure, our hierarchical approach achieves the highest F-measure on all four datasets. Notably, on ECSSD, which specifically contains images with complex structures, we improve F-measure by 4.2% over the next best method. Ablation analysis confirms that multi-layer integration outperforms any single layer, and that the hierarchical refinement provides an additional 2-3% improvement over simple averaging.

We have presented a hierarchical saliency detection method that addresses the fundamental scale challenge in saliency computation. By constructing a segmentation tree and computing saliency at multiple levels with principled integration, our approach captures both fine details and global structure simultaneously. The consistent improvements across four datasets and strong performance on complex-structure images validate the effectiveness of hierarchical processing. Our new HKU-IS dataset also provides a valuable resource for future research. Potential extensions include integrating deep features into the hierarchical framework and applying it to video saliency detection.