COCO-Stuff: Thing and Stuff Classes in Context

Abstract: Semantic classes can be either things (objects with a well-defined shape, e.g. car, person) or stuff (amorphous background regions, e.g. grass, sky). While lots of classification and detection works focus on thing classes, less attention has been given to stuff classes. Nonetheless, stuff classes are important as they allow to explain important aspects of an image, including (1) scene type; (2) which thing classes are likely to be present and their location (through contextual reasoning); (3) physical attributes, material types and geometric properties of the scene. To understand stuff and things in context we introduce COCO-Stuff, which augments all 164K images of the COCO 2017 dataset with pixel-wise annotations for 91 stuff classes. We introduce an efficient stuff annotation protocol based on superpixels, which leverages the original thing annotations. We quantify the speed versus quality trade-off of our protocol and explore the relation between annotation time and boundary complexity. Furthermore, we use COCO-Stuff to analyze: (a) the importance of stuff and thing classes in terms of their surface cover and how frequently they are mentioned in image captions; (b) the spatial relations between stuff and things, highlighting the rich contextual relations that make our dataset unique; (c) the performance of a modern semantic segmentation method on stuff and thing classes, and whether stuff is easier to segment than things.

• The paper introduces the COCO-Stuff dataset, adding pixel-level annotations for 91 stuff categories to complement COCO’s 80 thing classes.
• The paper proposes an efficient superpixel-based annotation protocol that reduces labeling time while preserving high accuracy.
• The paper evaluates segmentation with DeepLab V2, revealing that fine-grained stuff segmentation can be as challenging as detecting things in complex scenes.

COCO-Stuff: Thing and Stuff Classes in Context

The paper "COCO-Stuff: Thing and Stuff Classes in Context," authored by Holger Caesar, Jasper Uijlings, and Vittorio Ferrari, presents a comprehensive augmentation of the COCO dataset with pixel-level annotations for 91 'stuff' categories. This is in addition to the 'thing' annotations already present in COCO. The work aims to bridge the gap between the extensive focus on 'things' (e.g., cars, people) and the relatively neglected 'stuff' (e.g., grass, sky) in the field of object detection and semantic segmentation.

The paper underscores several vital contributions:

1. COCO-Stuff Dataset Introduction: COCO-Stuff expands the COCO dataset by providing pixel-wise annotations for 91 stuff categories across all 164K images. This enables a richer analysis of scenes by augmenting the existing 80 thing categories with a comprehensive set of stuff annotations.
2. Efficient Annotation Protocol: The authors propose an annotation protocol based on superpixels and existing thing annotations. This approach is designed to balance quality and annotation speed, resulting in a significant reduction in time spent per image without compromising annotation accuracy.
3. Analysis of Stuff's Importance: The paper explores the role of stuff in image captions, spatial relations between stuff and things, and performance differentials in semantic segmentation tasks. Key findings included the fact that stuff covers a majority of the image surface and is frequently mentioned in descriptive captions.
4. Semantic Segmentation Benchmark: By evaluating a modern semantic segmentation method, DeepLab V2, on COCO-Stuff, the study establishes important baselines and reveals that segmenting stuff is not inherently easier than segmenting things when the dataset includes a rich variety of both categories.

Annotation Protocol and Dataset Expansion

The proposed annotation protocol leverages superpixels to efficiently label stuff regions, capitalizing on pre-existing, detailed thing annotations in COCO. This hybrid approach achieves a high annotation accuracy with reduced effort. The annotated dataset spans 164K images, comprising a diverse set of 91 stuff classes that reflect common visual elements in both indoor and outdoor scenes. The authors justify the decision to predefine stuff categories, as opposed to allowing free-form labels, to avoid inconsistencies and ensure that labels are mutually exclusive.

Analysis of Contextual Relations and Importance

One major insight from this work is the spatial contextual relationship between stuff and things. The study quantifies spatial context by analyzing the relative positions of stuff and things within an image. This analysis uncovers robust patterns, such as cars being frequently found above roads, emphasizing the importance of contextual understanding for scene interpretation.

Quantitatively, stuff constitutes about 69% of annotated pixels and 69% of labeled regions, highlighting its substantial presence in images. Moreover, stuff categories account for approximately 38% of nouns in human-generated captions, underscoring their descriptive importance.

Segmentation Performance and Dataset Impact

The authors provide a detailed evaluation of segmentation performance using DeepLab V2. They observe that, contrary to previous studies which predominantly dealt with coarser-grained stuff categories, the COCO-Stuff dataset's fine-grained and diverse labels lead to stuff being more challenging to segment than things. This challenges the prevailing notion that stuff is generally easier to segment.

Furthermore, the study demonstrates the benefits of large datasets for deep learning models. The performance of semantic segmentation models sees consistent improvement with the increase in training data size, underscoring the importance of expansive datasets like COCO-Stuff for advancing the field.

Implications and Future Directions

The COCO-Stuff dataset sets a new standard for large-scale scene understanding by presenting a balanced and richly annotated resource that includes both stuff and things in diverse contexts. This comprehensive dataset facilitates deeper investigations into the roles of various semantic categories in scene interpretation, enabling more informed development of models that can understand and interact with complex environments.

Future research could build upon COCO-Stuff to further explore multi-modal scene understanding, integrate 3D scene geometry with stuff-thing interactions, and develop models that better leverage contextual information for improved semantic segmentation and object detection. The dataset and the presented findings highlight the significance of both stuff and thing categories in advancing computational scene understanding, fostering more holistic AI systems.

By detailing these contributions and implications, the paper represents a substantial step forward in the field of computer vision and semantic segmentation, providing both a valuable dataset and critical insights that will drive further research and development.