Scene graph generation (SGG) aims to parse an image into a structured representation that captures objects and their pairwise relationships. However, existing SGG methods rely on bounding box-based object representations, which lack precise localization and cannot distinguish between stuff and thing categories. In this paper, we introduce Panoptic Scene Graph Generation (PSG), a new task that bridges panoptic segmentation and scene graph generation. PSG requires generating a scene graph grounded on panoptic segmentation masks rather than bounding boxes, providing a more comprehensive and precise structured understanding of visual scenes. We construct a large-scale PSG dataset with over 48,000 images and rich relation annotations, and propose baseline methods that demonstrate the challenges and opportunities of this new task.

Scene graph generation has emerged as a powerful paradigm for structured visual understanding, with applications ranging from visual question answering and image captioning to image retrieval and robotic manipulation. A scene graph represents an image as a directed graph where nodes correspond to objects and edges represent relationships between them. Despite significant progress, current SGG methods are fundamentally limited by their reliance on bounding boxes as the underlying object representation. Bounding boxes are inherently imprecise, especially for irregularly shaped objects or overlapping entities, and they cannot represent amorphous "stuff" categories such as sky, grass, or water.

Panoptic segmentation, which unifies semantic segmentation for stuff regions and instance segmentation for thing objects, provides a natural and more complete visual representation for scene graph grounding. By replacing bounding boxes with panoptic masks, we can represent both things and stuff as graph nodes, enabling richer relationship modeling such as "sky above building" or "road under car" that were previously impossible in traditional SGG. This motivates our proposal of Panoptic Scene Graph Generation (PSG), which we formalize as: given an input image, produce a panoptic segmentation and a set of relation triplets (subject mask, predicate, object mask) that describe the interactions in the scene.

Traditional scene graph generation methods follow a two-stage paradigm: first detecting objects with bounding boxes, then classifying pairwise relationships. Representative works include Neural Motifs, which leverages frequency baselines and LSTM-based context modeling, VCTree that constructs dynamic tree structures for context propagation, and Transformer-based methods such as RelTR that formulate SGG as a set prediction problem. A key limitation shared by all these methods is their inability to handle stuff categories, which constitute a significant portion of visual scenes but are excluded from bounding box detection. Moreover, the coarse localization of bounding boxes introduces noise in relationship classification, as the visual features extracted from boxes may include irrelevant background regions.

Panoptic segmentation has seen rapid progress with methods like Panoptic FPN, MaskFormer, and Mask2Former, which produce unified segmentation masks for both things and stuff. However, these methods focus solely on pixel-level labeling and do not capture the relational structure between segmented regions. Our PSG task extends panoptic segmentation by additionally requiring the prediction of inter-object relationships, thereby providing a richer and more actionable scene representation than either SGG or panoptic segmentation alone.

We construct the PSG dataset by augmenting the COCO panoptic annotations with relation labels. Starting from the COCO 2017 panoptic segmentation dataset, we define a relation taxonomy of 56 predicates covering spatial relations (e.g., above, below, in front of), action relations (e.g., riding, eating, holding), and descriptive relations (e.g., made of, part of, belonging to). Each image is annotated with a set of relation triplets (subject segment, predicate, object segment) where subjects and objects are panoptic segments. The final dataset contains 48,749 images with an average of 5.3 relation triplets per image, providing a comprehensive benchmark for the PSG task.

We propose two baseline frameworks for PSG. The two-stage approach (PSGFormer) first generates panoptic segmentation using Mask2Former, then extracts features for each segment and classifies pairwise relations using a transformer-based relation decoder. The relation decoder takes pairs of segment features as input, augmented with spatial encoding that captures the relative position and overlap between segments. The one-stage approach (PSGTR) extends DETR by jointly predicting segment masks and relations in a unified decoder, formulating PSG as an end-to-end set prediction problem. Both approaches use focal loss for the relation classification to address the severe class imbalance among predicates.

We evaluate the proposed baselines using standard SGG metrics adapted for PSG: Recall@K (R@K) and mean Recall@K (mR@K). PSGFormer achieves R@20 of 18.3 and mR@20 of 14.1, while PSGTR achieves R@20 of 16.6 and mR@20 of 12.8, indicating that the two-stage approach benefits from the stronger panoptic segmentation backbone. Compared to adapting traditional SGG methods (Neural Motifs, VCTree) to use panoptic segments instead of bounding boxes, both PSG-specific baselines show significant improvements, with PSGFormer outperforming adapted Neural Motifs by 4.2 mR@20. Analysis of per-predicate performance reveals that spatial relations (above, below) achieve the highest accuracy, while fine-grained action relations (playing, cooking) remain challenging.

We conduct ablation studies on key design choices. Adding spatial encoding to the relation decoder improves mR@20 by 2.1 points, confirming that relative spatial information between segments is crucial for relation prediction. Using panoptic masks instead of bounding boxes for feature extraction improves mR@20 by 1.8 points, validating the advantage of precise segment-level features. We also find that stuff-related relations account for 23% of all annotated triplets, underscoring the importance of including stuff categories in scene graph generation and the limitation of traditional box-based SGG approaches.

We have introduced Panoptic Scene Graph Generation (PSG), a new task that combines panoptic segmentation with scene graph generation to produce a more complete and precise structured understanding of visual scenes. By grounding scene graphs on panoptic masks rather than bounding boxes, PSG enables the representation of both things and stuff in the relational graph, capturing 23% more relations that were previously inaccessible. We have provided a large-scale dataset with 48K images and 56 relation categories, along with two baseline methods that establish initial benchmarks for the community. The significant gap between baseline performance and human-level understanding indicates that PSG represents a challenging and impactful research direction for comprehensive visual scene understanding.