OneFormer · Hugging Face (original) (raw)

This model was released on 2022-11-10 and added to Hugging Face Transformers on 2023-01-19.

Overview

The OneFormer model was proposed in OneFormer: One Transformer to Rule Universal Image Segmentation by Jitesh Jain, Jiachen Li, MangTik Chiu, Ali Hassani, Nikita Orlov, Humphrey Shi. OneFormer is a universal image segmentation framework that can be trained on a single panoptic dataset to perform semantic, instance, and panoptic segmentation tasks. OneFormer uses a task token to condition the model on the task in focus, making the architecture task-guided for training, and task-dynamic for inference.

The abstract from the paper is the following:

Universal Image Segmentation is not a new concept. Past attempts to unify image segmentation in the last decades include scene parsing, panoptic segmentation, and, more recently, new panoptic architectures. However, such panoptic architectures do not truly unify image segmentation because they need to be trained individually on the semantic, instance, or panoptic segmentation to achieve the best performance. Ideally, a truly universal framework should be trained only once and achieve SOTA performance across all three image segmentation tasks. To that end, we propose OneFormer, a universal image segmentation framework that unifies segmentation with a multi-task train-once design. We first propose a task-conditioned joint training strategy that enables training on ground truths of each domain (semantic, instance, and panoptic segmentation) within a single multi-task training process. Secondly, we introduce a task token to condition our model on the task at hand, making our model task-dynamic to support multi-task training and inference. Thirdly, we propose using a query-text contrastive loss during training to establish better inter-task and inter-class distinctions. Notably, our single OneFormer model outperforms specialized Mask2Former models across all three segmentation tasks on ADE20k, CityScapes, and COCO, despite the latter being trained on each of the three tasks individually with three times the resources. With new ConvNeXt and DiNAT backbones, we observe even more performance improvement. We believe OneFormer is a significant step towards making image segmentation more universal and accessible.

The figure below illustrates the architecture of OneFormer. Taken from the original paper.

This model was contributed by Jitesh Jain. The original code can be found here.

Usage tips

• OneFormer requires two inputs during inference: image and task token.
• During training, OneFormer only uses panoptic annotations.
• If you want to train the model in a distributed environment across multiple nodes, then one should update theget_num_masks function inside in the OneFormerLoss class of modeling_oneformer.py. When training on multiple nodes, this should be set to the average number of target masks across all nodes, as can be seen in the original implementation here.
• One can use OneFormerProcessor to prepare input images and task inputs for the model and optional targets for the model. OneFormerProcessor wraps OneFormerImageProcessor and CLIPTokenizer into a single instance to both prepare the images and encode the task inputs.
• To get the final segmentation, depending on the task, you can call post_process_semantic_segmentation() or post_process_instance_segmentation() or post_process_panoptic_segmentation(). All three tasks can be solved using OneFormerForUniversalSegmentation output, panoptic segmentation accepts an optional label_ids_to_fuse argument to fuse instances of the target object/s (e.g. sky) together.

Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with OneFormer.

• Demo notebooks regarding inference + fine-tuning on custom data can be found here.

If you’re interested in submitting a resource to be included here, please feel free to open a Pull Request and we will review it. The resource should ideally demonstrate something new instead of duplicating an existing resource.

OneFormer specific outputs

class transformers.models.oneformer.modeling_oneformer.OneFormerModelOutput

< source >

( encoder_hidden_states: tuple[torch.FloatTensor] | None = None pixel_decoder_hidden_states: tuple[torch.FloatTensor] | None = None transformer_decoder_hidden_states: torch.FloatTensor | None = None transformer_decoder_object_queries: torch.FloatTensor | None = None transformer_decoder_contrastive_queries: torch.FloatTensor | None = None transformer_decoder_mask_predictions: torch.FloatTensor | None = None transformer_decoder_class_predictions: torch.FloatTensor | None = None transformer_decoder_auxiliary_predictions: tuple[dict[str, torch.FloatTensor]] | None = None text_queries: torch.FloatTensor | None = None task_token: torch.FloatTensor | None = None attentions: tuple[torch.FloatTensor] | None = None )

Parameters

• encoder_hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings + one for the output of each stage) of shape (batch_size, num_channels, height, width). Hidden-states (also called feature maps) of the encoder model at the output of each stage.
• pixel_decoder_hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings + one for the output of each stage) of shape (batch_size, num_channels, height, width). Hidden-states (also called feature maps) of the pixel decoder model at the output of each stage.
• transformer_decoder_hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings + one for the output of each stage) of shape (batch_size, sequence_length, hidden_size). Hidden-states (also called feature maps) of the transformer decoder at the output of each stage.
• transformer_decoder_object_queries (torch.FloatTensor of shape (batch_size, num_queries, hidden_dim)) — Output object queries from the last layer in the transformer decoder.
• transformer_decoder_contrastive_queries (torch.FloatTensor of shape (batch_size, num_queries, hidden_dim)) — Contrastive queries from the transformer decoder.
• transformer_decoder_mask_predictions (torch.FloatTensor of shape (batch_size, num_queries, height, width)) — Mask Predictions from the last layer in the transformer decoder.
• transformer_decoder_class_predictions (torch.FloatTensor of shape (batch_size, num_queries, num_classes+1)) — Class Predictions from the last layer in the transformer decoder.
• transformer_decoder_auxiliary_predictions (Tuple of Dict of str, torch.FloatTensor, optional) — Tuple of class and mask predictions from each layer of the transformer decoder.
• text_queries (torch.FloatTensor, optional of shape (batch_size, num_queries, hidden_dim)) — Text queries derived from the input text list used for calculating contrastive loss during training.
• task_token (torch.FloatTensor of shape (batch_size, hidden_dim)) — 1D task token to condition the queries.
• attentions (tuple(tuple(torch.FloatTensor)), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of tuple(torch.FloatTensor) (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length). Self and Cross Attentions weights from transformer decoder.

Class for outputs of OneFormerModel. This class returns all the needed hidden states to compute the logits.

class transformers.models.oneformer.modeling_oneformer.OneFormerForUniversalSegmentationOutput

< source >

( loss: torch.FloatTensor | None = None class_queries_logits: torch.FloatTensor | None = None masks_queries_logits: torch.FloatTensor | None = None auxiliary_predictions: list = None encoder_hidden_states: tuple[torch.FloatTensor] | None = None pixel_decoder_hidden_states: list[torch.FloatTensor] | None = None transformer_decoder_hidden_states: torch.FloatTensor | None = None transformer_decoder_object_queries: torch.FloatTensor | None = None transformer_decoder_contrastive_queries: torch.FloatTensor | None = None transformer_decoder_mask_predictions: torch.FloatTensor | None = None transformer_decoder_class_predictions: torch.FloatTensor | None = None transformer_decoder_auxiliary_predictions: list[dict[str, torch.FloatTensor]] | None = None text_queries: torch.FloatTensor | None = None task_token: torch.FloatTensor | None = None attentions: tuple[tuple[torch.FloatTensor]] | None = None )

Parameters

• loss (torch.Tensor, optional) — The computed loss, returned when labels are present.
• class_queries_logits (torch.FloatTensor, optional, defaults to None) — A tensor of shape (batch_size, num_queries, num_labels + 1) representing the proposed classes for each query. Note the + 1 is needed because we incorporate the null class.
• masks_queries_logits (torch.FloatTensor, optional, defaults to None) — A tensor of shape (batch_size, num_queries, height, width) representing the proposed masks for each query.
• auxiliary_predictions (List of Dict of str, torch.FloatTensor, optional) — List of class and mask predictions from each layer of the transformer decoder.
• encoder_hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings + one for the output of each stage) of shape (batch_size, num_channels, height, width). Hidden-states (also called feature maps) of the encoder model at the output of each stage.
• pixel_decoder_hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings + one for the output of each stage) of shape (batch_size, num_channels, height, width). Hidden-states (also called feature maps) of the pixel decoder model at the output of each stage.
• transformer_decoder_hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings + one for the output of each stage) of shape (batch_size, sequence_length, hidden_size). Hidden-states (also called feature maps) of the transformer decoder at the output of each stage.
• transformer_decoder_object_queries (torch.FloatTensor of shape (batch_size, num_queries, hidden_dim)) — Output object queries from the last layer in the transformer decoder.
• transformer_decoder_contrastive_queries (torch.FloatTensor of shape (batch_size, num_queries, hidden_dim)) — Contrastive queries from the transformer decoder.
• transformer_decoder_mask_predictions (torch.FloatTensor of shape (batch_size, num_queries, height, width)) — Mask Predictions from the last layer in the transformer decoder.
• transformer_decoder_class_predictions (torch.FloatTensor of shape (batch_size, num_queries, num_classes+1)) — Class Predictions from the last layer in the transformer decoder.
• transformer_decoder_auxiliary_predictions (List of Dict of str, torch.FloatTensor, optional) — List of class and mask predictions from each layer of the transformer decoder.
• text_queries (torch.FloatTensor, optional of shape (batch_size, num_queries, hidden_dim)) — Text queries derived from the input text list used for calculating contrastive loss during training.
• task_token (torch.FloatTensor of shape (batch_size, hidden_dim)) — 1D task token to condition the queries.
• attentions (tuple(tuple(torch.FloatTensor)), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of tuple(torch.FloatTensor) (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length). Self and Cross Attentions weights from transformer decoder.

Class for outputs of OneFormerForUniversalSegmentationOutput.

This output can be directly passed to post_process_semantic_segmentation() orpost_process_instance_segmentation() orpost_process_panoptic_segmentation() depending on the task. Please, see [`~OneFormerImageProcessor] for details regarding usage.

OneFormerConfig

class transformers.OneFormerConfig

< source >

( transformers_version: str | None = None architectures: list[str] | None = None output_hidden_states: bool | None = False return_dict: bool | None = True dtype: typing.Union[str, ForwardRef('torch.dtype'), NoneType] = None chunk_size_feed_forward: int = 0 is_encoder_decoder: bool = False id2label: dict[int, str] | dict[str, str] | None = None label2id: dict[str, int] | dict[str, str] | None = None problem_type: typing.Optional[typing.Literal['regression', 'single_label_classification', 'multi_label_classification']] = None backbone_config: dict | transformers.configuration_utils.PreTrainedConfig | None = None ignore_value: int = 255 num_queries: int = 150 no_object_weight: float = 0.1 class_weight: float = 2.0 mask_weight: float = 5.0 dice_weight: float = 5.0 contrastive_weight: float = 0.5 contrastive_temperature: float = 0.07 train_num_points: int = 12544 oversample_ratio: float = 3.0 importance_sample_ratio: float = 0.75 init_std: float = 0.02 init_xavier_std: float = 1.0 layer_norm_eps: float = 1e-05 is_training: bool = False use_auxiliary_loss: bool = True output_auxiliary_logits: bool = True strides: list[int] | tuple[int, ...] = (4, 8, 16, 32) task_seq_len: int = 77 text_encoder_width: int = 256 text_encoder_context_length: int = 77 text_encoder_num_layers: int = 6 text_encoder_vocab_size: int = 49408 text_encoder_proj_layers: int = 2 text_encoder_n_ctx: int = 16 conv_dim: int = 256 mask_dim: int = 256 hidden_dim: int = 256 encoder_feedforward_dim: int = 1024 norm: str = 'GN' encoder_layers: int = 6 decoder_layers: int = 10 use_task_norm: bool = True num_attention_heads: int = 8 dropout: float | int = 0.1 dim_feedforward: int = 2048 pre_norm: bool = False enforce_input_proj: bool = False query_dec_layers: int = 2 common_stride: int = 4 )

Parameters

• backbone_config (Union[dict, ~configuration_utils.PreTrainedConfig], optional) — The configuration of the backbone model.
• ignore_value (int, optional, defaults to 255) — Values to be ignored in GT label while calculating loss.
• num_queries (int, optional, defaults to 150) — Number of object queries.
• no_object_weight (float, optional, defaults to 0.1) — Relative classification weight of the no-object class in the object detection loss.
• class_weight (float, optional, defaults to 2.0) — Relative weight of the classification error in the Hungarian matching cost.
• mask_weight (float, optional, defaults to 5.0) — Relative weight of the focal loss in the panoptic segmentation loss.
• dice_weight (float, optional, defaults to 5.0) — Relative weight of the dice loss in the panoptic segmentation loss.
• contrastive_weight (float, optional, defaults to 0.5) — Weight for contrastive loss.
• contrastive_temperature (float, optional, defaults to 0.07) — Initial value for scaling the contrastive logits.
• train_num_points (int, optional, defaults to 12544) — Number of points to sample while calculating losses on mask predictions.
• oversample_ratio (float, optional, defaults to 3.0) — Ratio to decide how many points to oversample.
• importance_sample_ratio (float, optional, defaults to 0.75) — Ratio of points that are sampled via importance sampling..
• init_std (float, optional, defaults to 0.02) — The standard deviation of the truncated_normal_initializer for initializing all weight matrices.
• init_xavier_std (float, optional, defaults to 1.0) — The scaling factor used for the Xavier initialization of the cross-attention weights.
• layer_norm_eps (float, optional, defaults to 1e-05) — The epsilon used by the layer normalization layers.
• is_training (bool, optional, defaults to False) — Whether to run in training or inference mode.
• use_auxiliary_loss (bool, optional, defaults to True) — Whether to calculate loss using intermediate predictions from transformer decoder.
• output_auxiliary_logits (bool, optional, defaults to True) — Whether to return intermediate predictions from transformer decoder.
• strides (list, optional, defaults to [4, 8, 16, 32]) — List containing the strides for feature maps in the encoder.
• task_seq_len (int, optional, defaults to 77) — Sequence length for tokenizing text list input.
• text_encoder_width (int, optional, defaults to 256) — Hidden size for text encoder.
• text_encoder_context_length (int, optional, defaults to 77) — Input sequence length for text encoder.
• text_encoder_num_layers (int, optional, defaults to 6) — Number of layers for transformer in text encoder.
• text_encoder_vocab_size (int, optional, defaults to 49408) — Vocabulary size for tokenizer.
• text_encoder_proj_layers (int, optional, defaults to 2) — Number of layers in MLP for project text queries.
• text_encoder_n_ctx (int, optional, defaults to 16) — Number of learnable text context queries.
• conv_dim (int, optional, defaults to 256) — Feature map dimension to map outputs from the backbone.
• mask_dim (int, optional, defaults to 256) — Dimension for feature maps in pixel decoder.
• hidden_dim (int, optional, defaults to 256) — Dimension for hidden states in transformer decoder.
• encoder_feedforward_dim (int, optional, defaults to 1024) — Dimension for FFN layer in pixel decoder.
• norm (str, optional, defaults to "GN") — Type of normalization.
• encoder_layers (int, optional, defaults to 6) — Number of hidden layers in the Transformer encoder. Will use the same value as num_layers if not set.
• decoder_layers (int, optional, defaults to 10) — Number of hidden layers in the Transformer decoder. Will use the same value as num_layers if not set.
• use_task_norm (bool, optional, defaults to True) — Whether to normalize the task token.
• num_attention_heads (int, optional, defaults to 8) — Number of attention heads for each attention layer in the Transformer decoder.
• dropout (Union[float, int], optional, defaults to 0.1) — The ratio for all dropout layers.
• dim_feedforward (int, optional, defaults to 2048) — Dimension for FFN layer in transformer decoder.
• pre_norm (bool, optional, defaults to False) — Whether to normalize hidden states before attention layers in transformer decoder.
• enforce_input_proj (bool, optional, defaults to False) — Whether to project hidden states in transformer decoder.
• query_dec_layers (int, optional, defaults to 2) — Number of layers in query transformer.
• common_stride (int, optional, defaults to 4) — Common stride used for features in pixel decoder.

This is the configuration class to store the configuration of a OneFormerModel. It is used to instantiate a Oneformer model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the shi-labs/oneformer_ade20k_swin_tiny

Configuration objects inherit from PreTrainedConfig and can be used to control the model outputs. Read the documentation from PreTrainedConfig for more information.

Examples:

from transformers import OneFormerConfig, OneFormerModel

configuration = OneFormerConfig()

model = OneFormerModel(configuration)

configuration = model.config

OneFormerImageProcessor

class transformers.OneFormerImageProcessor

< source >

( **kwargs: typing_extensions.Unpack[transformers.models.oneformer.image_processing_oneformer.OneFormerImageProcessorKwargs] )

Parameters

• repo_path (str, kwargs, optional, defaults to shi-labs/oneformer_demo) — Path to a local directory or HuggingFace Hub repository containing model metadata.
• class_info_file (str, kwargs, optional) — Path to the JSON file within the repository that contains class metadata.
• num_text (int, kwargs, optional) — Number of text queries for the text encoder, used as task-guiding prompts.
• num_labels (int, kwargs, optional) — Number of semantic classes for segmentation, determining the output layer’s size.
• ignore_index (int, kwargs, optional) — Label to ignore in segmentation maps, often used for padding.
• do_reduce_labels (bool, kwargs, optional, defaults to False) — Whether to decrement all label values by 1, mapping the background class to ignore_index.
• * *kwargs (ImagesKwargs, optional) — Additional image preprocessing options. Model-specific kwargs are listed above; see the TypedDict class for the complete list of supported arguments.

Constructs a OneFormerImageProcessor image processor.

preprocess

< source >

( images: typing.Union[ForwardRef('PIL.Image.Image'), numpy.ndarray, ForwardRef('torch.Tensor'), list['PIL.Image.Image'], list[numpy.ndarray], list['torch.Tensor']] task_inputs: list[str] | None = None segmentation_maps: typing.Union[ForwardRef('PIL.Image.Image'), numpy.ndarray, ForwardRef('torch.Tensor'), list['PIL.Image.Image'], list[numpy.ndarray], list['torch.Tensor'], NoneType] = None instance_id_to_semantic_id: list[dict[int, int]] | dict[int, int] | None = None **kwargs: typing_extensions.Unpack[transformers.models.oneformer.image_processing_oneformer.OneFormerImageProcessorKwargs] ) → ~image_processing_base.BatchFeature

Parameters

• images (Union[PIL.Image.Image, numpy.ndarray, torch.Tensor, list[PIL.Image.Image], list[numpy.ndarray], list[torch.Tensor]]) — Image to preprocess. Expects a single or batch of images with pixel values ranging from 0 to 255. If passing in images with pixel values between 0 and 1, set do_rescale=False.
• task_inputs (list[str], optional) — List of tasks ("panoptic", "instance", "semantic") for each image in the batch.
• segmentation_maps (ImageInput, optional) — The segmentation maps.
• instance_id_to_semantic_id (Union[list[dict[int, int]], dict[int, int]], optional) — A mapping from instance IDs to semantic IDs.
• repo_path (str, kwargs, optional, defaults to shi-labs/oneformer_demo) — Path to a local directory or HuggingFace Hub repository containing model metadata.
• class_info_file (str, kwargs, optional) — Path to the JSON file within the repository that contains class metadata.
• num_text (int, kwargs, optional) — Number of text queries for the text encoder, used as task-guiding prompts.
• num_labels (int, kwargs, optional) — Number of semantic classes for segmentation, determining the output layer’s size.
• ignore_index (int, kwargs, optional) — Label to ignore in segmentation maps, often used for padding.
• do_reduce_labels (bool, kwargs, optional, defaults to False) — Whether to decrement all label values by 1, mapping the background class to ignore_index.
• return_tensors (str or TensorType, optional) — Returns stacked tensors if set to 'pt', otherwise returns a list of tensors.
• * *kwargs (ImagesKwargs, optional) — Additional image preprocessing options. Model-specific kwargs are listed above; see the TypedDict class for the complete list of supported arguments.

Returns

~image_processing_base.BatchFeature

• data (dict) — Dictionary of lists/arrays/tensors returned by the call method (‘pixel_values’, etc.).
• tensor_type (Union[None, str, TensorType], optional) — You can give a tensor_type here to convert the lists of integers in PyTorch/Numpy Tensors at initialization.

post_process_semantic_segmentation

< source >

( outputs target_sizes: list[tuple[int, int]] | None = None ) → List[torch.Tensor]

Parameters

• outputs (MaskFormerForInstanceSegmentation) — Raw outputs of the model.
• target_sizes (List[Tuple[int, int]], optional) — List of length (batch_size), where each list item (Tuple[int, int]]) corresponds to the requested final size (height, width) of each prediction. If left to None, predictions will not be resized.

Returns

List[torch.Tensor]

A list of length batch_size, where each item is a semantic segmentation map of shape (height, width) corresponding to the target_sizes entry (if target_sizes is specified). Each entry of eachtorch.Tensor correspond to a semantic class id.

Converts the output of MaskFormerForInstanceSegmentation into semantic segmentation maps. Only supports PyTorch.

post_process_instance_segmentation

< source >

( outputs task_type: str = 'instance' is_demo: bool = True threshold: float = 0.5 mask_threshold: float = 0.5 overlap_mask_area_threshold: float = 0.8 target_sizes: list[tuple[int, int]] | None = None return_coco_annotation: bool | None = False ) → List[Dict]

Parameters

• outputs (OneFormerForUniversalSegmentationOutput) — The outputs from OneFormerForUniversalSegmentationOutput.
• task_type (str, optional, defaults to “instance”) — The post processing depends on the task token input. If the task_type is “panoptic”, we need to ignore the stuff predictions.
• is_demo (bool, optional), defaults to True) — Whether the model is in demo mode. If true, use threshold to predict final masks.
• threshold (float, optional, defaults to 0.5) — The probability score threshold to keep predicted instance masks.
• mask_threshold (float, optional, defaults to 0.5) — Threshold to use when turning the predicted masks into binary values.
• overlap_mask_area_threshold (float, optional, defaults to 0.8) — The overlap mask area threshold to merge or discard small disconnected parts within each binary instance mask.
• target_sizes (List[Tuple], optional) — List of length (batch_size), where each list item (Tuple[int, int]]) corresponds to the requested final size (height, width) of each prediction in batch. If left to None, predictions will not be resized.
• return_coco_annotation (bool, optional), defaults to False) — Whether to return predictions in COCO format.

A list of dictionaries, one per image, each dictionary containing two keys:

• segmentation — a tensor of shape (height, width) where each pixel represents a segment_id, set to None if no mask if found above threshold. If target_sizes is specified, segmentation is resized to the corresponding target_sizes entry.
• segments_info — A dictionary that contains additional information on each segment.
  • id — an integer representing the segment_id.
  • label_id — An integer representing the label / semantic class id corresponding to segment_id.
  • was_fused — a boolean, True if label_id was in label_ids_to_fuse, False otherwise. Multiple instances of the same class / label were fused and assigned a single segment_id.
  • score — Prediction score of segment with segment_id.

Converts the output of OneFormerForUniversalSegmentationOutput into image instance segmentation predictions. Only supports PyTorch.

post_process_panoptic_segmentation

< source >

( outputs threshold: float = 0.5 mask_threshold: float = 0.5 overlap_mask_area_threshold: float = 0.8 label_ids_to_fuse: set[int] | None = None target_sizes: list[tuple[int, int]] | None = None ) → list[Dict]

Parameters

• outputs (MaskFormerForInstanceSegmentationOutput) — The outputs from MaskFormerForInstanceSegmentation.
• threshold (float, optional, defaults to 0.5) — The probability score threshold to keep predicted instance masks.
• mask_threshold (float, optional, defaults to 0.5) — Threshold to use when turning the predicted masks into binary values.
• overlap_mask_area_threshold (float, optional, defaults to 0.8) — The overlap mask area threshold to merge or discard small disconnected parts within each binary instance mask.
• label_ids_to_fuse (Set[int], optional) — The labels in this state will have all their instances be fused together. For instance we could say there can only be one sky in an image, but several persons, so the label ID for sky would be in that set, but not the one for person.
• target_sizes (list[Tuple], optional) — List of length (batch_size), where each list item (tuple[int, int]]) corresponds to the requested final size (height, width) of each prediction in batch. If left to None, predictions will not be resized.

A list of dictionaries, one per image, each dictionary containing two keys:

• segmentation — a tensor of shape (height, width) where each pixel represents a segment_id, set to None if no mask if found above threshold. If target_sizes is specified, segmentation is resized to the corresponding target_sizes entry.
• segments_info — A dictionary that contains additional information on each segment.
  • id — an integer representing the segment_id.
  • label_id — An integer representing the label / semantic class id corresponding to segment_id.
  • was_fused — a boolean, True if label_id was in label_ids_to_fuse, False otherwise. Multiple instances of the same class / label were fused and assigned a single segment_id.
  • score — Prediction score of segment with segment_id.

Converts the output of MaskFormerForInstanceSegmentationOutput into image panoptic segmentation predictions. Only supports PyTorch.

OneFormerImageProcessorPil

class transformers.OneFormerImageProcessorPil

< source >

( **kwargs: typing_extensions.Unpack[transformers.models.oneformer.image_processing_pil_oneformer.OneFormerImageProcessorKwargs] )

Parameters

• repo_path (str, kwargs, optional, defaults to shi-labs/oneformer_demo) — Path to a local directory or HuggingFace Hub repository containing model metadata.
• class_info_file (str, kwargs, optional) — Path to the JSON file within the repository that contains class metadata.
• num_text (int, kwargs, optional) — Number of text queries for the text encoder, used as task-guiding prompts.
• num_labels (int, kwargs, optional) — Number of semantic classes for segmentation, determining the output layer’s size.
• ignore_index (int, kwargs, optional) — Label to ignore in segmentation maps, often used for padding.
• do_reduce_labels (bool, kwargs, optional, defaults to False) — Whether to decrement all label values by 1, mapping the background class to ignore_index.
• * *kwargs (ImagesKwargs, optional) — Additional image preprocessing options. Model-specific kwargs are listed above; see the TypedDict class for the complete list of supported arguments.

Constructs a OneFormerImageProcessor image processor.

preprocess

< source >

( images: typing.Union[ForwardRef('PIL.Image.Image'), numpy.ndarray, ForwardRef('torch.Tensor'), list['PIL.Image.Image'], list[numpy.ndarray], list['torch.Tensor']] task_inputs: list[str] | None = None segmentation_maps: typing.Union[ForwardRef('PIL.Image.Image'), numpy.ndarray, ForwardRef('torch.Tensor'), list['PIL.Image.Image'], list[numpy.ndarray], list['torch.Tensor'], NoneType] = None instance_id_to_semantic_id: list[dict[int, int]] | dict[int, int] | None = None **kwargs: typing_extensions.Unpack[transformers.models.oneformer.image_processing_pil_oneformer.OneFormerImageProcessorKwargs] ) → ~image_processing_base.BatchFeature

Parameters

• images (Union[PIL.Image.Image, numpy.ndarray, torch.Tensor, list[PIL.Image.Image], list[numpy.ndarray], list[torch.Tensor]]) — Image to preprocess. Expects a single or batch of images with pixel values ranging from 0 to 255. If passing in images with pixel values between 0 and 1, set do_rescale=False.
• task_inputs (list[str], optional) — List of tasks ("panoptic", "instance", "semantic") for each image in the batch.
• segmentation_maps (ImageInput, optional) — The segmentation maps.
• instance_id_to_semantic_id (Union[list[dict[int, int]], dict[int, int]], optional) — A mapping from instance IDs to semantic IDs.
• repo_path (str, kwargs, optional, defaults to shi-labs/oneformer_demo) — Path to a local directory or HuggingFace Hub repository containing model metadata.
• class_info_file (str, kwargs, optional) — Path to the JSON file within the repository that contains class metadata.
• num_text (int, kwargs, optional) — Number of text queries for the text encoder, used as task-guiding prompts.
• num_labels (int, kwargs, optional) — Number of semantic classes for segmentation, determining the output layer’s size.
• ignore_index (int, kwargs, optional) — Label to ignore in segmentation maps, often used for padding.
• do_reduce_labels (bool, kwargs, optional, defaults to False) — Whether to decrement all label values by 1, mapping the background class to ignore_index.
• return_tensors (str or TensorType, optional) — Returns stacked tensors if set to 'pt', otherwise returns a list of tensors.
• * *kwargs (ImagesKwargs, optional) — Additional image preprocessing options. Model-specific kwargs are listed above; see the TypedDict class for the complete list of supported arguments.

Returns

~image_processing_base.BatchFeature

• data (dict) — Dictionary of lists/arrays/tensors returned by the call method (‘pixel_values’, etc.).
• tensor_type (Union[None, str, TensorType], optional) — You can give a tensor_type here to convert the lists of integers in PyTorch/Numpy Tensors at initialization.

post_process_semantic_segmentation

< source >

( outputs target_sizes: list[tuple[int, int]] | None = None ) → List[torch.Tensor]

Parameters

• outputs (MaskFormerForInstanceSegmentation) — Raw outputs of the model.
• target_sizes (List[Tuple[int, int]], optional) — List of length (batch_size), where each list item (Tuple[int, int]]) corresponds to the requested final size (height, width) of each prediction. If left to None, predictions will not be resized.

Returns

List[torch.Tensor]

A list of length batch_size, where each item is a semantic segmentation map of shape (height, width) corresponding to the target_sizes entry (if target_sizes is specified). Each entry of eachtorch.Tensor correspond to a semantic class id.

Converts the output of MaskFormerForInstanceSegmentation into semantic segmentation maps. Only supports PyTorch.

post_process_instance_segmentation

< source >

( outputs task_type: str = 'instance' is_demo: bool = True threshold: float = 0.5 mask_threshold: float = 0.5 overlap_mask_area_threshold: float = 0.8 target_sizes: list[tuple[int, int]] | None = None return_coco_annotation: bool | None = False ) → List[Dict]

Parameters

• outputs (OneFormerForUniversalSegmentationOutput) — The outputs from OneFormerForUniversalSegmentationOutput.
• task_type (str, optional, defaults to “instance”) — The post processing depends on the task token input. If the task_type is “panoptic”, we need to ignore the stuff predictions.
• is_demo (bool, optional), defaults to True) — Whether the model is in demo mode. If true, use threshold to predict final masks.
• threshold (float, optional, defaults to 0.5) — The probability score threshold to keep predicted instance masks.
• mask_threshold (float, optional, defaults to 0.5) — Threshold to use when turning the predicted masks into binary values.
• overlap_mask_area_threshold (float, optional, defaults to 0.8) — The overlap mask area threshold to merge or discard small disconnected parts within each binary instance mask.
• target_sizes (List[Tuple], optional) — List of length (batch_size), where each list item (Tuple[int, int]]) corresponds to the requested final size (height, width) of each prediction in batch. If left to None, predictions will not be resized.
• return_coco_annotation (bool, optional), defaults to False) — Whether to return predictions in COCO format.

A list of dictionaries, one per image, each dictionary containing two keys:

• segmentation — a tensor of shape (height, width) where each pixel represents a segment_id, set to None if no mask if found above threshold. If target_sizes is specified, segmentation is resized to the corresponding target_sizes entry.
• segments_info — A dictionary that contains additional information on each segment.
  • id — an integer representing the segment_id.
  • label_id — An integer representing the label / semantic class id corresponding to segment_id.
  • was_fused — a boolean, True if label_id was in label_ids_to_fuse, False otherwise. Multiple instances of the same class / label were fused and assigned a single segment_id.
  • score — Prediction score of segment with segment_id.

Converts the output of OneFormerForUniversalSegmentationOutput into image instance segmentation predictions. Only supports PyTorch.

post_process_panoptic_segmentation

< source >

( outputs threshold: float = 0.5 mask_threshold: float = 0.5 overlap_mask_area_threshold: float = 0.8 label_ids_to_fuse: set[int] | None = None target_sizes: list[tuple[int, int]] | None = None ) → list[Dict]

Parameters

• outputs (MaskFormerForInstanceSegmentationOutput) — The outputs from MaskFormerForInstanceSegmentation.
• threshold (float, optional, defaults to 0.5) — The probability score threshold to keep predicted instance masks.
• mask_threshold (float, optional, defaults to 0.5) — Threshold to use when turning the predicted masks into binary values.
• overlap_mask_area_threshold (float, optional, defaults to 0.8) — The overlap mask area threshold to merge or discard small disconnected parts within each binary instance mask.
• label_ids_to_fuse (Set[int], optional) — The labels in this state will have all their instances be fused together. For instance we could say there can only be one sky in an image, but several persons, so the label ID for sky would be in that set, but not the one for person.
• target_sizes (list[Tuple], optional) — List of length (batch_size), where each list item (tuple[int, int]]) corresponds to the requested final size (height, width) of each prediction in batch. If left to None, predictions will not be resized.

A list of dictionaries, one per image, each dictionary containing two keys:

• segmentation — a tensor of shape (height, width) where each pixel represents a segment_id, set to None if no mask if found above threshold. If target_sizes is specified, segmentation is resized to the corresponding target_sizes entry.
• segments_info — A dictionary that contains additional information on each segment.
  • id — an integer representing the segment_id.
  • label_id — An integer representing the label / semantic class id corresponding to segment_id.
  • was_fused — a boolean, True if label_id was in label_ids_to_fuse, False otherwise. Multiple instances of the same class / label were fused and assigned a single segment_id.
  • score — Prediction score of segment with segment_id.

Converts the output of MaskFormerForInstanceSegmentationOutput into image panoptic segmentation predictions. Only supports PyTorch.

OneFormerProcessor

class transformers.OneFormerProcessor

< source >

( image_processor = None tokenizer = None max_seq_length: int = 77 task_seq_length: int = 77 **kwargs )

Parameters

• image_processor (OneFormerImageProcessor) — The image processor is a required input.
• tokenizer (CLIPTokenizer) — The tokenizer is a required input.
• max_seq_length (int, optional, defaults to 77) — Maximum sequence length for encoding class names and text inputs. This parameter controls the maximum number of tokens used when tokenizing class names and other text inputs for the model.
• task_seq_length (int, optional, defaults to 77) — Maximum sequence length specifically for encoding task descriptions. Task descriptions (e.g., “the task is semantic”) are tokenized with this length limit, which may differ from the general text sequence length.

Constructs a OneFormerProcessor which wraps a image processor and a tokenizer into a single processor.

OneFormerProcessor offers all the functionalities of OneFormerImageProcessor and CLIPTokenizer. See the~OneFormerImageProcessor and ~CLIPTokenizer for more information.

__call__

< source >

( images = None task_inputs = None segmentation_maps = None **kwargs ) → BatchFeature

Parameters

• images (` ) -- Image to preprocess. Expects a single or batch of images with pixel values ranging from 0 to 255. If passing in images with pixel values between 0 and 1, set do_rescale=False`.
• task_inputs (str or list[str], required) — The task type(s) for segmentation. Must be one or more of "semantic", "instance", or "panoptic". Can be a single string for a single image, or a list of strings (one per image) for batch processing. The task type determines which type of segmentation the model will perform on the input images.
• segmentation_maps (ImageInput, optional) — The corresponding semantic segmentation maps with the pixel-wise annotations.
  (bool, optional, defaults to True): Whether or not to pad images up to the largest image in a batch and create a pixel mask.
  If left to the default, will return a pixel mask that is:
  • 1 for pixels that are real (i.e. not masked),
  • 0 for pixels that are padding (i.e. masked).
• return_tensors (str or TensorType, optional) — If set, will return tensors of a particular framework. Acceptable values are:
  • 'pt': Return PyTorch torch.Tensor objects.
  • 'np': Return NumPy np.ndarray objects.

A BatchFeature with the following fields:

• task_inputs — List of token ids to be fed to a model. Returned when text is not None.
• pixel_values — Pixel values to be fed to a model. Returned when images is not None.

OneFormerModel

class transformers.OneFormerModel

< source >

( config: OneFormerConfig )

Parameters

• config (OneFormerConfig) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out thefrom_pretrained() method to load the model weights.

The bare Oneformer Model outputting raw hidden-states without any specific head on top.

This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.)

This model is also a PyTorch torch.nn.Module subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.

forward

< source >

( pixel_values: Tensor task_inputs: Tensor text_inputs: torch.Tensor | None = None pixel_mask: torch.Tensor | None = None output_hidden_states: bool | None = None output_attentions: bool | None = None return_dict: bool | None = None **kwargs ) → OneFormerModelOutput or tuple(torch.FloatTensor)

Parameters

• pixel_values (torch.Tensor of shape (batch_size, num_channels, image_size, image_size)) — The tensors corresponding to the input images. Pixel values can be obtained usingOneFormerImageProcessor. See OneFormerImageProcessor.__call__() for details (OneFormerProcessor usesOneFormerImageProcessor for processing images).
• task_inputs (torch.FloatTensor of shape (batch_size, sequence_length)) — Task inputs. Task inputs can be obtained using AutoImageProcessor. See OneFormerProcessor.call()for details.
• text_inputs (list[torch.Tensor], optional) — Tensor of shape (num_queries, sequence_length) to be fed to a model
• pixel_mask (torch.Tensor of shape (batch_size, height, width), optional) — Mask to avoid performing attention on padding pixel values. Mask values selected in [0, 1]:
  • 1 for pixels that are real (i.e. not masked),
  • 0 for pixels that are padding (i.e. masked).
    What are attention masks?
• output_hidden_states (bool, optional) — Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for more detail.
• output_attentions (bool, optional) — Whether or not to return the attentions tensors of all attention layers. See attentions under returned tensors for more detail.
• return_dict (bool, optional) — Whether or not to return a ModelOutput instead of a plain tuple.

Returns

OneFormerModelOutput or tuple(torch.FloatTensor)

A OneFormerModelOutput or a tuple oftorch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various elements depending on the configuration (OneFormerConfig) and inputs.

The OneFormerModel forward method, overrides the __call__ special method.

Although the recipe for forward pass needs to be defined within this function, one should call the Moduleinstance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

• encoder_hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings + one for the output of each stage) of shape (batch_size, num_channels, height, width). Hidden-states (also called feature maps) of the encoder model at the output of each stage.
• pixel_decoder_hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings + one for the output of each stage) of shape (batch_size, num_channels, height, width). Hidden-states (also called feature maps) of the pixel decoder model at the output of each stage.
• transformer_decoder_hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings + one for the output of each stage) of shape (batch_size, sequence_length, hidden_size). Hidden-states (also called feature maps) of the transformer decoder at the output of each stage.
• transformer_decoder_object_queries (torch.FloatTensor of shape (batch_size, num_queries, hidden_dim)) — Output object queries from the last layer in the transformer decoder.
• transformer_decoder_contrastive_queries (torch.FloatTensor of shape (batch_size, num_queries, hidden_dim)) — Contrastive queries from the transformer decoder.
• transformer_decoder_mask_predictions (torch.FloatTensor of shape (batch_size, num_queries, height, width)) — Mask Predictions from the last layer in the transformer decoder.
• transformer_decoder_class_predictions (torch.FloatTensor of shape (batch_size, num_queries, num_classes+1)) — Class Predictions from the last layer in the transformer decoder.
• transformer_decoder_auxiliary_predictions (Tuple of Dict of str, torch.FloatTensor, optional) — Tuple of class and mask predictions from each layer of the transformer decoder.
• text_queries (torch.FloatTensor, optional of shape (batch_size, num_queries, hidden_dim)) — Text queries derived from the input text list used for calculating contrastive loss during training.
• task_token (torch.FloatTensor of shape (batch_size, hidden_dim)) — 1D task token to condition the queries.
• attentions (tuple(tuple(torch.FloatTensor)), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of tuple(torch.FloatTensor) (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length). Self and Cross Attentions weights from transformer decoder.

Example:

import torch from PIL import Image import httpx from io import BytesIO from transformers import OneFormerProcessor, OneFormerModel

url = "http://images.cocodataset.org/val2017/000000039769.jpg" with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read()))

processor = OneFormerProcessor.from_pretrained("shi-labs/oneformer_ade20k_swin_tiny") model = OneFormerModel.from_pretrained("shi-labs/oneformer_ade20k_swin_tiny") inputs = processor(image, ["semantic"], return_tensors="pt")

with torch.no_grad(): ... outputs = model(**inputs)

mask_predictions = outputs.transformer_decoder_mask_predictions class_predictions = outputs.transformer_decoder_class_predictions

f"👉 Mask Predictions Shape: {list(mask_predictions.shape)}, Class Predictions Shape: {list(class_predictions.shape)}" '👉 Mask Predictions Shape: [1, 150, 128, 171], Class Predictions Shape: [1, 150, 151]'

OneFormerForUniversalSegmentation

class transformers.OneFormerForUniversalSegmentation

< source >

( config: OneFormerConfig )

Parameters

• config (OneFormerConfig) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out thefrom_pretrained() method to load the model weights.

OneFormer Model for instance, semantic and panoptic image segmentation.

This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.)

This model is also a PyTorch torch.nn.Module subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.

forward

< source >

( pixel_values: Tensor task_inputs: Tensor text_inputs: torch.Tensor | None = None mask_labels: list[torch.Tensor] | None = None class_labels: list[torch.Tensor] | None = None pixel_mask: torch.Tensor | None = None output_auxiliary_logits: bool | None = None output_hidden_states: bool | None = None output_attentions: bool | None = None return_dict: bool | None = None **kwargs ) → OneFormerForUniversalSegmentationOutput or tuple(torch.FloatTensor)

Parameters

• pixel_values (torch.Tensor of shape (batch_size, num_channels, image_size, image_size)) — The tensors corresponding to the input images. Pixel values can be obtained usingOneFormerImageProcessor. See OneFormerImageProcessor.__call__() for details (OneFormerProcessor usesOneFormerImageProcessor for processing images).
• task_inputs (torch.FloatTensor of shape (batch_size, sequence_length)) — Task inputs. Task inputs can be obtained using AutoImageProcessor. See OneFormerProcessor.call()for details.
• text_inputs (list[torch.Tensor], optional) — Tensor of shape (num_queries, sequence_length) to be fed to a model
• mask_labels (list[torch.Tensor], optional) — List of mask labels of shape (num_labels, height, width) to be fed to a model
• class_labels (list[torch.LongTensor], optional) — list of target class labels of shape (num_labels, height, width) to be fed to a model. They identify the labels of mask_labels, e.g. the label of mask_labels[i][j] if class_labels[i][j].
• pixel_mask (torch.Tensor of shape (batch_size, height, width), optional) — Mask to avoid performing attention on padding pixel values. Mask values selected in [0, 1]:
  • 1 for pixels that are real (i.e. not masked),
  • 0 for pixels that are padding (i.e. masked).
    What are attention masks?
• output_auxiliary_logits (bool, optional) — Whether or not to output auxiliary logits.
• output_hidden_states (bool, optional) — Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for more detail.
• output_attentions (bool, optional) — Whether or not to return the attentions tensors of all attention layers. See attentions under returned tensors for more detail.
• return_dict (bool, optional) — Whether or not to return a ModelOutput instead of a plain tuple.

A OneFormerForUniversalSegmentationOutput or a tuple oftorch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various elements depending on the configuration (OneFormerConfig) and inputs.

The OneFormerForUniversalSegmentation forward method, overrides the __call__ special method.

Although the recipe for forward pass needs to be defined within this function, one should call the Moduleinstance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

• loss (torch.Tensor, optional) — The computed loss, returned when labels are present.
• class_queries_logits (torch.FloatTensor, optional, defaults to None) — A tensor of shape (batch_size, num_queries, num_labels + 1) representing the proposed classes for each query. Note the + 1 is needed because we incorporate the null class.
• masks_queries_logits (torch.FloatTensor, optional, defaults to None) — A tensor of shape (batch_size, num_queries, height, width) representing the proposed masks for each query.
• auxiliary_predictions (List of Dict of str, torch.FloatTensor, optional) — List of class and mask predictions from each layer of the transformer decoder.
• encoder_hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings + one for the output of each stage) of shape (batch_size, num_channels, height, width). Hidden-states (also called feature maps) of the encoder model at the output of each stage.
• pixel_decoder_hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings + one for the output of each stage) of shape (batch_size, num_channels, height, width). Hidden-states (also called feature maps) of the pixel decoder model at the output of each stage.
• transformer_decoder_hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings + one for the output of each stage) of shape (batch_size, sequence_length, hidden_size). Hidden-states (also called feature maps) of the transformer decoder at the output of each stage.
• transformer_decoder_object_queries (torch.FloatTensor of shape (batch_size, num_queries, hidden_dim)) — Output object queries from the last layer in the transformer decoder.
• transformer_decoder_contrastive_queries (torch.FloatTensor of shape (batch_size, num_queries, hidden_dim)) — Contrastive queries from the transformer decoder.
• transformer_decoder_mask_predictions (torch.FloatTensor of shape (batch_size, num_queries, height, width)) — Mask Predictions from the last layer in the transformer decoder.
• transformer_decoder_class_predictions (torch.FloatTensor of shape (batch_size, num_queries, num_classes+1)) — Class Predictions from the last layer in the transformer decoder.
• transformer_decoder_auxiliary_predictions (List of Dict of str, torch.FloatTensor, optional) — List of class and mask predictions from each layer of the transformer decoder.
• text_queries (torch.FloatTensor, optional of shape (batch_size, num_queries, hidden_dim)) — Text queries derived from the input text list used for calculating contrastive loss during training.
• task_token (torch.FloatTensor of shape (batch_size, hidden_dim)) — 1D task token to condition the queries.
• attentions (tuple(tuple(torch.FloatTensor)), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of tuple(torch.FloatTensor) (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length). Self and Cross Attentions weights from transformer decoder.

Example:

Universal segmentation example:

from transformers import OneFormerProcessor, OneFormerForUniversalSegmentation from PIL import Image import httpx from io import BytesIO import torch

processor = OneFormerProcessor.from_pretrained("shi-labs/oneformer_ade20k_swin_tiny") model = OneFormerForUniversalSegmentation.from_pretrained("shi-labs/oneformer_ade20k_swin_tiny")

url = ( ... "https://huggingface.co/datasets/hf-internal-testing/fixtures_ade20k/resolve/main/ADE_val_00000001.jpg" ... ) with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read()))

inputs = processor(image, ["semantic"], return_tensors="pt")

with torch.no_grad(): ... outputs = model(**inputs)

class_queries_logits = outputs.class_queries_logits masks_queries_logits = outputs.masks_queries_logits

predicted_semantic_map = processor.post_process_semantic_segmentation( ... outputs, target_sizes=[(image.height, image.width)] ... )[0] f"👉 Semantic Predictions Shape: {list(predicted_semantic_map.shape)}" '👉 Semantic Predictions Shape: [512, 683]'

inputs = processor(image, ["instance"], return_tensors="pt")

with torch.no_grad(): ... outputs = model(**inputs)

class_queries_logits = outputs.class_queries_logits masks_queries_logits = outputs.masks_queries_logits

predicted_instance_map = processor.post_process_instance_segmentation( ... outputs, target_sizes=[(image.height, image.width)] ... )[0]["segmentation"] f"👉 Instance Predictions Shape: {list(predicted_instance_map.shape)}" '👉 Instance Predictions Shape: [512, 683]'

inputs = processor(image, ["panoptic"], return_tensors="pt")

with torch.no_grad(): ... outputs = model(**inputs)

class_queries_logits = outputs.class_queries_logits masks_queries_logits = outputs.masks_queries_logits

predicted_panoptic_map = processor.post_process_panoptic_segmentation( ... outputs, target_sizes=[(image.height, image.width)] ... )[0]["segmentation"] f"👉 Panoptic Predictions Shape: {list(predicted_panoptic_map.shape)}" '👉 Panoptic Predictions Shape: [512, 683]'

Update on GitHub