Tubegraph

Comparison of segmentation data sets

4 min read

From: hu-po
The Segment Anything Model (SAM) introduces the SA-1B dataset, marking a significant advancement in image segmentation. This dataset is compared against existing ones, highlighting differences in scale, annotation methods, and diversity.

SA-1B Dataset Overview

The SA-1B dataset, released by Meta AI Research, is a large-scale collection specifically designed for image segmentation tasks [00:04:55]. It is touted as the largest segmentation dataset to date [00:05:06].

Key features of SA-1B:

  • Scale: Contains over 1 billion masks on 11 million licensed, privacy-respecting images [00:05:10].
  • Mask Quantity: It has 400 times more masks than any existing segmentation dataset [00:19:16].
  • Annotation Process: The data engine for SA-1B involved three stages: assisted manual, semi-automatic, and fully automatic [00:16:37].
    • Initially, human annotators, primarily from Kenya, assisted in creating masks [02:04:07], decreasing annotation time as the model improved [00:53:17].
    • 99% of the masks in SA-1B are generated fully automatically [01:00:29].
    • Masks are verified to be high quality; 94% of automatically predicted masks had an Intersection Over Union (IOU) greater than 90% when compared to professional annotations [01:01:16].
  • Image Characteristics: The images in SA-1B are high-resolution, often 2000x3000 pixels or larger [00:59:56].

Comparison with Existing Datasets

SA-1B significantly differs from traditional segmentation datasets like COCO and ImageNet in several aspects:

Scale and Density of Annotations

  • COCO: This dataset, often used for training segmentation models, has 1.2 million images [02:08:12] with polygon-based masks, not pixel-level [01:37:15]. It has a high number of images with fewer masks and very few images with many masks [01:05:01].
  • ImageNet: An older dataset used for pre-trained models, ImageNet 1K contains 1.2 million images [02:08:12].
  • SA-1B vs. Others: SA-1B’s 11 million images are roughly 10 times more than ImageNet’s 1.2 million [00:24:14], and it boasts over 1 billion masks, a magnitude higher than previous datasets [00:24:18]. SA-1B tends to have more masks per image, with a median of 51 to 200 masks per image, compared to older datasets with fewer masks [01:05:13].

Annotation Methodology

Existing datasets often relied heavily on manual annotation, which is time-consuming and expensive [01:38:16]. SA-1B, in contrast, leveraged the Segment Anything Model itself in a data engine loop to automate mask generation [00:10:57]. This process of using a model to generate data to train itself is a form of self-supervised learning [01:09:54].

Object Size and Detail

Older datasets like COCO had a bias towards larger objects and were often limited to polygon-level masks, which are less precise than pixel-level masks [01:37:15]. SA-1B, with its automatically generated masks, captures smaller and medium-sized objects more frequently, resulting in a richer diversity of mask sizes [01:03:40], [01:04:12]. The generated masks are highly detailed, allowing for pixel-level precision [01:07:09].

Geographic and Demographic Diversity

While all datasets exhibit common photographer biases (e.g., objects centered in images) [01:03:03], SA-1B aims for greater geographic diversity. It includes a higher percentage of images from Europe and Asia, aiming to reduce biases in image distribution, unlike datasets primarily sourced from a single region [01:08:46]. The dataset’s perceived gender, skin tone, and age group representation are also analyzed to ensure fairness in segmenting people [01:09:05].

Applications and Performance

SA-1B is designed to foster research in Foundation models for segmentation [00:06:30]. The SAM, trained on SA-1B, shows strong zero-shot transfer performance across various tasks including:

SAM’s capabilities were evaluated on a diverse suite of 23 segmentation datasets, including novel image distributions like underwater or egocentric images, X-ray images, and even oil paintings [01:15:24]. SAM significantly outperforms prior interactive segmentors like SimpleClick, especially with a single prompt point [01:23:51]. However, it may underperform on highly specialized or low-level tasks, such as specific biological or plant images [01:17:59].

Overall, the SA-1B dataset, combined with the powerful SAM, aims to set a new standard for large-scale image segmentation, enabling more robust and generalized models for various computer vision applications [01:50:50].

Graph View