Depth Anything (original) (raw)

Depth Anything is designed to be a foundation model for monocular depth estimation (MDE). It is jointly trained on labeled and ~62M unlabeled images to enhance the dataset. It uses a pretrained DINOv2 model as an image encoder to inherit its existing rich semantic priors, and DPT as the decoder. A teacher model is trained on unlabeled images to create pseudo-labels. The student model is trained on a combination of the pseudo-labels and labeled images. To improve the student model’s performance, strong perturbations are added to the unlabeled images to challenge the student model to learn more visual knowledge from the image.

You can find all the original Depth Anything checkpoints under the Depth Anything collection.

Click on the Depth Anything models in the right sidebar for more examples of how to apply Depth Anything to different vision tasks.

The example below demonstrates how to obtain a depth map with Pipeline or the AutoModel class.

import torch from transformers import pipeline

pipe = pipeline(task="depth-estimation", model="LiheYoung/depth-anything-base-hf", torch_dtype=torch.bfloat16, device=0) pipe("http://images.cocodataset.org/val2017/000000039769.jpg")["depth"]

Notes

• DepthAnythingV2, released in June 2024, uses the same architecture as Depth Anything and is compatible with all code examples and existing workflows. It uses synthetic data and a larger capacity teacher model to achieve much finer and robust depth predictions.

DepthAnythingConfig

class transformers.DepthAnythingConfig

< source >

( backbone_config = None backbone = None use_pretrained_backbone = False use_timm_backbone = False backbone_kwargs = None patch_size = 14 initializer_range = 0.02 reassemble_hidden_size = 384 reassemble_factors = [4, 2, 1, 0.5] neck_hidden_sizes = [48, 96, 192, 384] fusion_hidden_size = 64 head_in_index = -1 head_hidden_size = 32 depth_estimation_type = 'relative' max_depth = None **kwargs )

Parameters

• backbone_config (Union[Dict[str, Any], PretrainedConfig], optional) — The configuration of the backbone model. Only used in case is_hybrid is True or in case you want to leverage the AutoBackbone API.
• backbone (str, optional) — Name of backbone to use when backbone_config is None. If use_pretrained_backbone is True, this will load the corresponding pretrained weights from the timm or transformers library. If use_pretrained_backboneis False, this loads the backbone’s config and uses that to initialize the backbone with random weights.
• use_pretrained_backbone (bool, optional, defaults to False) — Whether to use pretrained weights for the backbone.
• use_timm_backbone (bool, optional, defaults to False) — Whether or not to use the timm library for the backbone. If set to False, will use the AutoBackboneAPI.
• backbone_kwargs (dict, optional) — Keyword arguments to be passed to AutoBackbone when loading from a checkpoint e.g. {'out_indices': (0, 1, 2, 3)}. Cannot be specified if backbone_config is set.
• patch_size (int, optional, defaults to 14) — The size of the patches to extract from the backbone features.
• initializer_range (float, optional, defaults to 0.02) — The standard deviation of the truncated_normal_initializer for initializing all weight matrices.
• reassemble_hidden_size (int, optional, defaults to 384) — The number of input channels of the reassemble layers.
• reassemble_factors (List[int], optional, defaults to [4, 2, 1, 0.5]) — The up/downsampling factors of the reassemble layers.
• neck_hidden_sizes (List[str], optional, defaults to [48, 96, 192, 384]) — The hidden sizes to project to for the feature maps of the backbone.
• fusion_hidden_size (int, optional, defaults to 64) — The number of channels before fusion.
• head_in_index (int, optional, defaults to -1) — The index of the features to use in the depth estimation head.
• head_hidden_size (int, optional, defaults to 32) — The number of output channels in the second convolution of the depth estimation head.
• depth_estimation_type (str, optional, defaults to "relative") — The type of depth estimation to use. Can be one of ["relative", "metric"].
• max_depth (float, optional) — The maximum depth to use for the “metric” depth estimation head. 20 should be used for indoor models and 80 for outdoor models. For “relative” depth estimation, this value is ignored.

This is the configuration class to store the configuration of a DepthAnythingModel. It is used to instantiate a DepthAnything model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the DepthAnythingLiheYoung/depth-anything-small-hf architecture.

Configuration objects inherit from PretrainedConfig and can be used to control the model outputs. Read the documentation from PretrainedConfig for more information.

Example:

from transformers import DepthAnythingConfig, DepthAnythingForDepthEstimation

configuration = DepthAnythingConfig()

model = DepthAnythingForDepthEstimation(configuration)

configuration = model.config

Serializes this instance to a Python dictionary. Override the default to_dict(). Returns:Dict[str, any]: Dictionary of all the attributes that make up this configuration instance,

DepthAnythingForDepthEstimation

class transformers.DepthAnythingForDepthEstimation

< source >

( config )

Parameters

• config (DepthAnythingForDepthEstimation) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out thefrom_pretrained() method to load the model weights.

Depth Anything Model with a depth estimation head on top (consisting of 3 convolutional layers) e.g. for KITTI, NYUv2.

This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.)

This model is also a PyTorch torch.nn.Module subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.

forward

< source >

( pixel_values: FloatTensor labels: typing.Optional[torch.LongTensor] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None ) → transformers.modeling_outputs.DepthEstimatorOutput or tuple(torch.FloatTensor)

Parameters

• pixel_values (torch.FloatTensor of shape (batch_size, num_channels, image_size, image_size)) — The tensors corresponding to the input images. Pixel values can be obtained using{image_processor_class}. See {image_processor_class}.__call__ for details ({processor_class} uses{image_processor_class} for processing images).
• labels (torch.LongTensor of shape (batch_size, height, width), optional) — Ground truth depth estimation maps for computing the loss.
• output_attentions (bool, optional) — Whether or not to return the attentions tensors of all attention layers. See attentions under returned tensors for more detail.
• output_hidden_states (bool, optional) — Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for more detail.
• return_dict (bool, optional) — Whether or not to return a ModelOutput instead of a plain tuple.

A transformers.modeling_outputs.DepthEstimatorOutput or a tuple oftorch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various elements depending on the configuration (DepthAnythingConfig) and inputs.

• loss (torch.FloatTensor of shape (1,), optional, returned when labels is provided) — Classification (or regression if config.num_labels==1) loss.
• predicted_depth (torch.FloatTensor of shape (batch_size, height, width)) — Predicted depth for each pixel.
• hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape (batch_size, num_channels, height, width).
  Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.
• attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, patch_size, sequence_length).
  Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads.

The DepthAnythingForDepthEstimation forward method, overrides the __call__ special method.

Although the recipe for forward pass needs to be defined within this function, one should call the Moduleinstance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Examples:

from transformers import AutoImageProcessor, AutoModelForDepthEstimation import torch import numpy as np from PIL import Image import requests

url = "http://images.cocodataset.org/val2017/000000039769.jpg" image = Image.open(requests.get(url, stream=True).raw)

image_processor = AutoImageProcessor.from_pretrained("LiheYoung/depth-anything-small-hf") model = AutoModelForDepthEstimation.from_pretrained("LiheYoung/depth-anything-small-hf")

inputs = image_processor(images=image, return_tensors="pt")

with torch.no_grad(): ... outputs = model(**inputs)

post_processed_output = image_processor.post_process_depth_estimation( ... outputs, ... target_sizes=[(image.height, image.width)], ... )

predicted_depth = post_processed_output[0]["predicted_depth"] depth = predicted_depth * 255 / predicted_depth.max() depth = depth.detach().cpu().numpy() depth = Image.fromarray(depth.astype("uint8"))

< > Update on GitHub