CLAP (original) (raw)

Overview

The CLAP model was proposed in Large Scale Contrastive Language-Audio pretraining with feature fusion and keyword-to-caption augmentation by Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor Berg-Kirkpatrick, Shlomo Dubnov.

CLAP (Contrastive Language-Audio Pretraining) is a neural network trained on a variety of (audio, text) pairs. It can be instructed in to predict the most relevant text snippet, given an audio, without directly optimizing for the task. The CLAP model uses a SWINTransformer to get audio features from a log-Mel spectrogram input, and a RoBERTa model to get text features. Both the text and audio features are then projected to a latent space with identical dimension. The dot product between the projected audio and text features is then used as a similar score.

The abstract from the paper is the following:

Contrastive learning has shown remarkable success in the field of multimodal representation learning. In this paper, we propose a pipeline of contrastive language-audio pretraining to develop an audio representation by combining audio data with natural language descriptions. To accomplish this target, we first release LAION-Audio-630K, a large collection of 633,526 audio-text pairs from different data sources. Second, we construct a contrastive language-audio pretraining model by considering different audio encoders and text encoders. We incorporate the feature fusion mechanism and keyword-to-caption augmentation into the model design to further enable the model to process audio inputs of variable lengths and enhance the performance. Third, we perform comprehensive experiments to evaluate our model across three tasks: text-to-audio retrieval, zero-shot audio classification, and supervised audio classification. The results demonstrate that our model achieves superior performance in text-to-audio retrieval task. In audio classification tasks, the model achieves state-of-the-art performance in the zeroshot setting and is able to obtain performance comparable to models’ results in the non-zero-shot setting. LAION-Audio-6

This model was contributed by Younes Belkada and Arthur Zucker . The original code can be found here.

ClapConfig

class transformers.ClapConfig

< source >

( text_config = None audio_config = None logit_scale_init_value = 14.285714285714285 projection_dim = 512 projection_hidden_act = 'relu' initializer_factor = 1.0 **kwargs )

Parameters

• text_config (dict, optional) — Dictionary of configuration options used to initialize ClapTextConfig.
• audio_config (dict, optional) — Dictionary of configuration options used to initialize ClapAudioConfig.
• logit_scale_init_value (float, optional, defaults to 14.29) — The initial value of the logit_scale parameter. Default is used as per the original CLAP implementation.
• projection_dim (int, optional, defaults to 512) — Dimensionality of text and audio projection layers.
• projection_hidden_act (str, optional, defaults to "relu") — Activation function for the projection layers.
• initializer_factor (float, optional, defaults to 1.0) — Factor to scale the initialization of the model weights.
• kwargs (optional) — Dictionary of keyword arguments.

ClapConfig is the configuration class to store the configuration of a ClapModel. It is used to instantiate a CLAP model according to the specified arguments, defining the text model and audio model configs. Instantiating a configuration with the defaults will yield a similar configuration to that of the CLAPlaion/clap-htsat-fused architecture.

Configuration objects inherit from PretrainedConfig and can be used to control the model outputs. Read the documentation from PretrainedConfig for more information.

Example:

from transformers import ClapConfig, ClapModel

configuration = ClapConfig()

model = ClapModel(configuration)

configuration = model.config

from transformers import ClapTextConfig, ClapAudioConfig

config_text = ClapTextConfig() config_audio = ClapAudioConfig()

config = ClapConfig.from_text_audio_configs(config_text, config_audio)

from_text_audio_configs

< source >

( text_config: ClapTextConfig audio_config: ClapAudioConfig **kwargs ) → ClapConfig

An instance of a configuration object

Instantiate a ClapConfig (or a derived class) from clap text model configuration and clap audio model configuration.

ClapTextConfig

class transformers.ClapTextConfig

< source >

( vocab_size = 50265 hidden_size = 768 num_hidden_layers = 12 num_attention_heads = 12 intermediate_size = 3072 hidden_act = 'gelu' hidden_dropout_prob = 0.1 attention_probs_dropout_prob = 0.1 max_position_embeddings = 514 type_vocab_size = 1 initializer_factor = 1.0 layer_norm_eps = 1e-12 projection_dim = 512 pad_token_id = 1 bos_token_id = 0 eos_token_id = 2 position_embedding_type = 'absolute' use_cache = True projection_hidden_act = 'relu' **kwargs )

Parameters

• vocab_size (int, optional, defaults to 30522) — Vocabulary size of the CLAP model. Defines the number of different tokens that can be represented by theinputs_ids passed when calling ClapTextModel.
• hidden_size (int, optional, defaults to 768) — Dimensionality of the encoder layers and the pooler layer.
• num_hidden_layers (int, optional, defaults to 12) — Number of hidden layers in the Transformer encoder.
• num_attention_heads (int, optional, defaults to 12) — Number of attention heads for each attention layer in the Transformer encoder.
• intermediate_size (int, optional, defaults to 3072) — Dimensionality of the “intermediate” (often named feed-forward) layer in the Transformer encoder.
• hidden_act (str or Callable, optional, defaults to "relu") — The non-linear activation function (function or string) in the encoder and pooler. If string, "relu","relu", "silu" and "relu_new" are supported.
• hidden_dropout_prob (float, optional, defaults to 0.1) — The dropout probability for all fully connected layers in the embeddings, encoder, and pooler.
• attention_probs_dropout_prob (float, optional, defaults to 0.1) — The dropout ratio for the attention probabilities.
• max_position_embeddings (int, optional, defaults to 512) — The maximum sequence length that this model might ever be used with. Typically set this to something large just in case (e.g., 512 or 1024 or 2048).
• type_vocab_size (int, optional, defaults to 2) — The vocabulary size of the token_type_ids passed when calling ClapTextModel.
• layer_norm_eps (float, optional, defaults to 1e-12) — The epsilon used by the layer normalization layers.
• position_embedding_type (str, optional, defaults to "absolute") — Type of position embedding. Choose one of "absolute", "relative_key", "relative_key_query". For positional embeddings use "absolute". For more information on "relative_key", please refer toSelf-Attention with Relative Position Representations (Shaw et al.). For more information on "relative_key_query", please refer to Method 4 in Improve Transformer Models with Better Relative Position Embeddings (Huang et al.).
• is_decoder (bool, optional, defaults to False) — Whether the model is used as a decoder or not. If False, the model is used as an encoder.
• use_cache (bool, optional, defaults to True) — Whether or not the model should return the last key/values attentions (not used by all models). Only relevant if config.is_decoder=True.
• projection_hidden_act (str, optional, defaults to "relu") — The non-linear activation function (function or string) in the projection layer. If string, "gelu","relu", "silu" and "gelu_new" are supported.
• projection_dim (int, optional, defaults to 512) — Dimension of the projection head of the ClapTextModelWithProjection.

This is the configuration class to store the configuration of a ClapTextModel. It is used to instantiate a CLAP model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the CLAPcalp-hsat-fused architecture.

Configuration objects inherit from PretrainedConfig and can be used to control the model outputs. Read the documentation from PretrainedConfig for more information.

Examples:

from transformers import ClapTextConfig, ClapTextModel

configuration = ClapTextConfig()

model = ClapTextModel(configuration)

configuration = model.config

ClapAudioConfig

class transformers.ClapAudioConfig

< source >

( window_size = 8 num_mel_bins = 64 spec_size = 256 hidden_act = 'gelu' patch_size = 4 patch_stride = [4, 4] num_classes = 527 hidden_size = 768 projection_dim = 512 depths = [2, 2, 6, 2] num_attention_heads = [4, 8, 16, 32] enable_fusion = False hidden_dropout_prob = 0.1 fusion_type = None patch_embed_input_channels = 1 flatten_patch_embeds = True patch_embeds_hidden_size = 96 enable_patch_layer_norm = True drop_path_rate = 0.0 attention_probs_dropout_prob = 0.0 qkv_bias = True mlp_ratio = 4.0 aff_block_r = 4 num_hidden_layers = 4 projection_hidden_act = 'relu' layer_norm_eps = 1e-05 initializer_factor = 1.0 **kwargs )

Parameters

• window_size (int, optional, defaults to 8) — Image size of the spectrogram
• num_mel_bins (int, optional, defaults to 64) — Number of mel features used per frames. Should correspond to the value used in the ClapProcessor class.
• spec_size (int, optional, defaults to 256) — Desired input size of the spectrogram that the model supports. It can be different from the output of theClapFeatureExtractor, in which case the input features will be resized. Corresponds to the image_sizeof the audio models.
• hidden_act (str, optional, defaults to "gelu") — The non-linear activation function (function or string) in the encoder and pooler. If string, "gelu","relu", "silu" and "gelu_new" are supported.
• patch_size (int, optional, defaults to 4) — Patch size for the audio spectrogram
• patch_stride (list, optional, defaults to [4, 4]) — Patch stride for the audio spectrogram
• num_classes (int, optional, defaults to 527) — Number of classes used for the head training
• hidden_size (int, optional, defaults to 768) — Hidden size of the output of the audio encoder. Correspond to the dimension of the penultimate layer’s output,which is sent to the projection MLP layer.
• projection_dim (int, optional, defaults to 512) — Hidden size of the projection layer.
• depths (list, optional, defaults to [2, 2, 6, 2]) — Depths used for the Swin Layers of the audio model
• num_attention_heads (list, optional, defaults to [4, 8, 16, 32]) — Number of attention heads used for the Swin Layers of the audio model
• enable_fusion (bool, optional, defaults to False) — Whether or not to enable patch fusion. This is the main contribution of the authors, and should give the best results.
• hidden_dropout_prob (float, optional, defaults to 0.1) — The dropout probability for all fully connected layers in the encoder.
• fusion_type ([type], optional) — Fusion type used for the patch fusion.
• patch_embed_input_channels (int, optional, defaults to 1) — Number of channels used for the input spectrogram
• flatten_patch_embeds (bool, optional, defaults to True) — Whether or not to flatten the patch embeddings
• patch_embeds_hidden_size (int, optional, defaults to 96) — Hidden size of the patch embeddings. It is used as the number of output channels.
• enable_patch_layer_norm (bool, optional, defaults to True) — Whether or not to enable layer normalization for the patch embeddings
• drop_path_rate (float, optional, defaults to 0.0) — Drop path rate for the patch fusion
• attention_probs_dropout_prob (float, optional, defaults to 0.0) — The dropout ratio for the attention probabilities.
• qkv_bias (bool, optional, defaults to True) — Whether or not to add a bias to the query, key, value projections.
• mlp_ratio (float, optional, defaults to 4.0) — Ratio of the mlp hidden dim to embedding dim.
• aff_block_r (int, optional, defaults to 4) — downsize_ratio used in the AudioFF block
• num_hidden_layers (int, optional, defaults to 4) — Number of hidden layers in the Transformer encoder.
• projection_hidden_act (str, optional, defaults to "relu") — The non-linear activation function (function or string) in the projection layer. If string, "gelu","relu", "silu" and "gelu_new" are supported.
• layer_norm_eps ([type], optional, defaults to 1e-05) — The epsilon used by the layer normalization layers.
• initializer_factor (float, optional, defaults to 1.0) — A factor for initializing all weight matrices (should be kept to 1, used internally for initialization testing).

This is the configuration class to store the configuration of a ClapAudioModel. It is used to instantiate a CLAP audio encoder according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the audio encoder of the CLAPlaion/clap-htsat-fused architecture.

Configuration objects inherit from PretrainedConfig and can be used to control the model outputs. Read the documentation from PretrainedConfig for more information.

Example:

from transformers import ClapAudioConfig, ClapAudioModel

configuration = ClapAudioConfig()

model = ClapAudioModel(configuration)

configuration = model.config

ClapFeatureExtractor

( feature_size = 64 sampling_rate = 48000 hop_length = 480 max_length_s = 10 fft_window_size = 1024 padding_value = 0.0 return_attention_mask = False frequency_min: float = 0 frequency_max: float = 14000 top_db: typing.Optional[int] = None truncation: str = 'fusion' padding: str = 'repeatpad' **kwargs )

Parameters

• feature_size (int, optional, defaults to 64) — The feature dimension of the extracted Mel spectrograms. This corresponds to the number of mel filters (n_mels).
• sampling_rate (int, optional, defaults to 48000) — The sampling rate at which the audio files should be digitalized expressed in hertz (Hz). This only serves to warn users if the audio fed to the feature extractor does not have the same sampling rate.
• hop_length (int,optional, defaults to 480) — Length of the overlapping windows for the STFT used to obtain the Mel Spectrogram. The audio will be split in smaller frames with a step of hop_length between each frame.
• max_length_s (int, optional, defaults to 10) — The maximum input length of the model in seconds. This is used to pad the audio.
• fft_window_size (int, optional, defaults to 1024) — Size of the window (in samples) on which the Fourier transform is applied. This controls the frequency resolution of the spectrogram. 400 means that the fourier transform is computed on windows of 400 samples.
• padding_value (float, optional, defaults to 0.0) — Padding value used to pad the audio. Should correspond to silences.
• return_attention_mask (bool, optional, defaults to False) — Whether or not the model should return the attention masks corresponding to the input.
• frequency_min (float, optional, defaults to 0) — The lowest frequency of interest. The STFT will not be computed for values below this.
• frequency_max (float, optional, defaults to 14000) — The highest frequency of interest. The STFT will not be computed for values above this.
• top_db (float, optional) — The highest decibel value used to convert the mel spectrogram to the log scale. For more details see theaudio_utils.power_to_db function
• truncation (str, optional, defaults to "fusion") — Truncation pattern for long audio inputs. Two patterns are available:
  • fusion will use _random_mel_fusion, which stacks 3 random crops from the mel spectrogram and a downsampled version of the entire mel spectrogram. If config.fusion is set to True, shorter audios also need to to return 4 mels, which will just be a copy of the original mel obtained from the padded audio.
  • rand_trunc will select a random crop of the mel spectrogram.
• padding (str, optional, defaults to "repeatpad") — Padding pattern for shorter audio inputs. Three patterns were originally implemented:
  • repeatpad: the audio is repeated, and then padded to fit the max_length.
  • repeat: the audio is repeated and then cut to fit the max_length
  • pad: the audio is padded.

Constructs a CLAP feature extractor.

This feature extractor inherits from SequenceFeatureExtractor which contains most of the main methods. Users should refer to this superclass for more information regarding those methods.

This class extracts mel-filter bank features from raw speech using a custom numpy implementation of the Short Time Fourier Transform (STFT) which should match pytorch’s torch.stft equivalent.

( ) → Dict[str, Any]

Dictionary of all the attributes that make up this configuration instance, except for the mel filter banks, which do not need to be saved or printed as they are too long.

Serializes this instance to a Python dictionary.

ClapProcessor

Constructs a CLAP processor which wraps a CLAP feature extractor and a RoBerta tokenizer into a single processor.

ClapProcessor offers all the functionalities of ClapFeatureExtractor and RobertaTokenizerFast. See the__call__() and decode() for more information.

This method forwards all its arguments to RobertaTokenizerFast’s batch_decode(). Please refer to the docstring of this method for more information.

This method forwards all its arguments to RobertaTokenizerFast’s decode(). Please refer to the docstring of this method for more information.

ClapModel

class transformers.ClapModel

< source >

( config: ClapConfig )

Parameters

• config (ClapConfig) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the from_pretrained() method to load the model weights.

This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.)

This model is also a PyTorch torch.nn.Module subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.

forward

< source >

( input_ids: typing.Optional[torch.LongTensor] = None input_features: typing.Optional[torch.FloatTensor] = None is_longer: typing.Optional[torch.BoolTensor] = None attention_mask: typing.Optional[torch.Tensor] = None position_ids: typing.Optional[torch.LongTensor] = None return_loss: typing.Optional[bool] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None ) → transformers.models.clap.modeling_clap.ClapOutput or tuple(torch.FloatTensor)

Parameters

• input_ids (torch.LongTensor of shape (batch_size, sequence_length)) — Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it.
  Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() andPreTrainedTokenizer.call() for details.
  What are input IDs?
• attention_mask (torch.Tensor of shape (batch_size, sequence_length), optional) — Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
  • 1 for tokens that are not masked,
  • 0 for tokens that are masked.
    What are attention masks?
• position_ids (torch.LongTensor of shape (batch_size, sequence_length), optional) — Indices of positions of each input sequence tokens in the position embeddings. Selected in the range [0, config.max_position_embeddings - 1].
  What are position IDs?
• input_features (torch.FloatTensor of shape (batch_size, num_channels, height, width)) — Input audio features. This should be returned by the ClapFeatureExtractor class that you can also retrieve from AutoFeatureExtractor. See ClapFeatureExtractor.__call__() for details.
• return_loss (bool, optional) — Whether or not to return the contrastive loss.
• output_attentions (bool, optional) — Whether or not to return the attentions tensors of all attention layers. See attentions under returned tensors for more detail.
• output_hidden_states (bool, optional) — Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for more detail.
• return_dict (bool, optional) — Whether or not to return a ModelOutput instead of a plain tuple.

Returns

transformers.models.clap.modeling_clap.ClapOutput or tuple(torch.FloatTensor)

A transformers.models.clap.modeling_clap.ClapOutput or a tuple oftorch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various elements depending on the configuration (<class 'transformers.models.clap.configuration_clap.ClapConfig'>) and inputs.

• loss (torch.FloatTensor of shape (1,), optional, returned when return_loss is True) — Contrastive loss for audio-text similarity.
• logits_per_audio (torch.FloatTensor of shape (audio_batch_size, text_batch_size)) — The scaled dot product scores between audio_embeds and text_embeds. This represents the audio-text similarity scores.
• logits_per_text (torch.FloatTensor of shape (text_batch_size, audio_batch_size)) — The scaled dot product scores between text_embeds and audio_embeds. This represents the text-audio similarity scores.
• text_embeds (torch.FloatTensor of shape (batch_size, output_dim) — The text embeddings obtained by applying the projection layer to the pooled output of ClapTextModel.
• audio_embeds (torch.FloatTensor of shape (batch_size, output_dim) — The audio embeddings obtained by applying the projection layer to the pooled output of ClapAudioModel.
• text_model_output (BaseModelOutputWithPooling) — The output of the ClapTextModel.
• audio_model_output (BaseModelOutputWithPooling) — The output of the ClapAudioModel.

The ClapModel forward method, overrides the __call__ special method.

Although the recipe for forward pass needs to be defined within this function, one should call the Moduleinstance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Examples:

from datasets import load_dataset from transformers import AutoProcessor, ClapModel

dataset = load_dataset("hf-internal-testing/ashraq-esc50-1-dog-example") audio_sample = dataset["train"]["audio"][0]["array"]

model = ClapModel.from_pretrained("laion/clap-htsat-unfused") processor = AutoProcessor.from_pretrained("laion/clap-htsat-unfused")

input_text = ["Sound of a dog", "Sound of vaccum cleaner"]

inputs = processor(text=input_text, audios=audio_sample, return_tensors="pt", padding=True)

outputs = model(**inputs) logits_per_audio = outputs.logits_per_audio
probs = logits_per_audio.softmax(dim=-1)

get_text_features

< source >

( input_ids: typing.Optional[torch.Tensor] = None attention_mask: typing.Optional[torch.Tensor] = None position_ids: typing.Optional[torch.Tensor] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None ) → text_features (torch.FloatTensor of shape (batch_size, output_dim)

Parameters

• input_ids (torch.LongTensor of shape (batch_size, sequence_length)) — Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it.
  Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() andPreTrainedTokenizer.call() for details.
  What are input IDs?
• attention_mask (torch.Tensor of shape (batch_size, sequence_length), optional) — Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
  • 1 for tokens that are not masked,
  • 0 for tokens that are masked.
    What are attention masks?
• position_ids (torch.LongTensor of shape (batch_size, sequence_length), optional) — Indices of positions of each input sequence tokens in the position embeddings. Selected in the range [0, config.max_position_embeddings - 1].
  What are position IDs?
• output_attentions (bool, optional) — Whether or not to return the attentions tensors of all attention layers. See attentions under returned tensors for more detail.
• output_hidden_states (bool, optional) — Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for more detail.
• return_dict (bool, optional) — Whether or not to return a ModelOutput instead of a plain tuple.

Returns

text_features (torch.FloatTensor of shape (batch_size, output_dim)

The text embeddings obtained by applying the projection layer to the pooled output of ClapTextModel.

The ClapModel forward method, overrides the __call__ special method.

Although the recipe for forward pass needs to be defined within this function, one should call the Moduleinstance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Examples:

from transformers import AutoTokenizer, ClapModel

model = ClapModel.from_pretrained("laion/clap-htsat-unfused") tokenizer = AutoTokenizer.from_pretrained("laion/clap-htsat-unfused")

inputs = tokenizer(["the sound of a cat", "the sound of a dog"], padding=True, return_tensors="pt") text_features = model.get_text_features(**inputs)

get_audio_features

< source >

( input_features: typing.Optional[torch.Tensor] = None is_longer: typing.Optional[torch.Tensor] = None attention_mask: typing.Optional[torch.Tensor] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None ) → audio_features (torch.FloatTensor of shape (batch_size, output_dim)

Parameters

• input_features (torch.FloatTensor of shape (batch_size, num_channels, height, width)) — Input audio features. This should be returned by the ClapFeatureExtractor class that you can also retrieve from AutoFeatureExtractor. See ClapFeatureExtractor.__call__() for details.
• is_longer (torch.FloatTensor, of shape (batch_size, 1), optional) — Whether the audio clip is longer than max_length. If True, a feature fusion will be enabled to enhance the features.
• output_attentions (bool, optional) — Whether or not to return the attentions tensors of all attention layers. See attentions under returned tensors for more detail.
• output_hidden_states (bool, optional) — Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for more detail.
• return_dict (bool, optional) — Whether or not to return a ModelOutput instead of a plain tuple.

Returns

audio_features (torch.FloatTensor of shape (batch_size, output_dim)

The audio embeddings obtained by applying the projection layer to the pooled output of ClapAudioModel.

The ClapModel forward method, overrides the __call__ special method.

Although the recipe for forward pass needs to be defined within this function, one should call the Moduleinstance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Examples:

from transformers import AutoFeatureExtractor, ClapModel import torch

model = ClapModel.from_pretrained("laion/clap-htsat-unfused") feature_extractor = AutoFeatureExtractor.from_pretrained("laion/clap-htsat-unfused") random_audio = torch.rand((16_000)) inputs = feature_extractor(random_audio, return_tensors="pt") audio_features = model.get_audio_features(**inputs)

ClapTextModel

class transformers.ClapTextModel

< source >

( config add_pooling_layer = True )

The model can behave as an encoder (with only self-attention) as well as a decoder, in which case a layer of cross-attention is added between the self-attention layers, following the architecture described in Attention is all you need_ by Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser and Illia Polosukhin.

To behave as an decoder the model needs to be initialized with the is_decoder argument of the configuration set to True. To be used in a Seq2Seq model, the model needs to initialized with both is_decoder argument andadd_cross_attention set to True; an encoder_hidden_states is then expected as an input to the forward pass.

.. _ Attention is all you need: https://arxiv.org/abs/1706.03762

forward

< source >

( input_ids: typing.Optional[torch.Tensor] = None attention_mask: typing.Optional[torch.Tensor] = None token_type_ids: typing.Optional[torch.Tensor] = None position_ids: typing.Optional[torch.Tensor] = None head_mask: typing.Optional[torch.Tensor] = None inputs_embeds: typing.Optional[torch.Tensor] = None encoder_hidden_states: typing.Optional[torch.Tensor] = None encoder_attention_mask: typing.Optional[torch.Tensor] = None past_key_values: typing.Optional[typing.List[torch.FloatTensor]] = None use_cache: typing.Optional[bool] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None )

encoder_hidden_states (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size), optional): Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention if the model is configured as a decoder. encoder_attention_mask (torch.FloatTensor of shape (batch_size, sequence_length), optional): Mask to avoid performing attention on the padding token indices of the encoder input. This mask is used in the cross-attention if the model is configured as a decoder. Mask values selected in [0, 1]:

• 1 for tokens that are not masked,
• 0 for tokens that are masked. past_key_values (tuple(tuple(torch.FloatTensor)) of length config.n_layers with each tuple having 4 tensors of shape (batch_size, num_heads, sequence_length - 1, embed_size_per_head)): Contains precomputed key and value hidden states of the attention blocks. Can be used to speed up decoding.

If past_key_values are used, the user can optionally input only the last decoder_input_ids (those that don’t have their past key value states given to this model) of shape (batch_size, 1) instead of alldecoder_input_ids of shape (batch_size, sequence_length). use_cache (bool, optional): If set to True, past_key_values key value states are returned and can be used to speed up decoding (seepast_key_values).

ClapTextModelWithProjection

class transformers.ClapTextModelWithProjection

< source >

( config: ClapTextConfig )

Parameters

• config (ClapConfig) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the from_pretrained() method to load the model weights.

CLAP Text Model with a projection layer on top (a linear layer on top of the pooled output).

This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.)

This model is also a PyTorch torch.nn.Module subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.

forward

< source >

( input_ids: typing.Optional[torch.Tensor] = None attention_mask: typing.Optional[torch.Tensor] = None position_ids: typing.Optional[torch.Tensor] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None ) → transformers.models.clap.modeling_clap.ClapTextModelOutput or tuple(torch.FloatTensor)

Parameters

• input_ids (torch.LongTensor of shape (batch_size, sequence_length)) — Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it.
  Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() andPreTrainedTokenizer.call() for details.
  What are input IDs?
• attention_mask (torch.Tensor of shape (batch_size, sequence_length), optional) — Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
  • 1 for tokens that are not masked,
  • 0 for tokens that are masked.
    What are attention masks?
• position_ids (torch.LongTensor of shape (batch_size, sequence_length), optional) — Indices of positions of each input sequence tokens in the position embeddings. Selected in the range [0, config.max_position_embeddings - 1].
  What are position IDs?
• output_attentions (bool, optional) — Whether or not to return the attentions tensors of all attention layers. See attentions under returned tensors for more detail.
• output_hidden_states (bool, optional) — Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for more detail.
• return_dict (bool, optional) — Whether or not to return a ModelOutput instead of a plain tuple.

Returns

transformers.models.clap.modeling_clap.ClapTextModelOutput or tuple(torch.FloatTensor)

A transformers.models.clap.modeling_clap.ClapTextModelOutput or a tuple oftorch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various elements depending on the configuration (<class 'transformers.models.clap.configuration_clap.ClapTextConfig'>) and inputs.

• text_embeds (torch.FloatTensor of shape (batch_size, output_dim) optional returned when model is initialized with with_projection=True) — The text embeddings obtained by applying the projection layer to the pooler_output.
• last_hidden_state (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size)) — Sequence of hidden-states at the output of the last layer of the model.
• hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).
  Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.
• attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
  Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads.

The ClapTextModelWithProjection forward method, overrides the __call__ special method.

Although the recipe for forward pass needs to be defined within this function, one should call the Moduleinstance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Examples:

from transformers import AutoTokenizer, ClapTextModelWithProjection

model = ClapTextModelWithProjection.from_pretrained("laion/clap-htsat-unfused") tokenizer = AutoTokenizer.from_pretrained("laion/clap-htsat-unfused")

inputs = tokenizer(["a sound of a cat", "a sound of a dog"], padding=True, return_tensors="pt")

outputs = model(**inputs) text_embeds = outputs.text_embeds

ClapAudioModel

class transformers.ClapAudioModel

< source >

( config: ClapAudioConfig )

forward

< source >

( input_features: typing.Optional[torch.FloatTensor] = None is_longer: typing.Optional[torch.BoolTensor] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None ) → transformers.modeling_outputs.BaseModelOutputWithPooling or tuple(torch.FloatTensor)

Parameters

• input_features (torch.FloatTensor of shape (batch_size, num_channels, height, width)) — Input audio features. This should be returned by the ClapFeatureExtractor class that you can also retrieve from AutoFeatureExtractor. See ClapFeatureExtractor.__call__() for details.
• is_longer (torch.FloatTensor, of shape (batch_size, 1), optional) — Whether the audio clip is longer than max_length. If True, a feature fusion will be enabled to enhance the features.
• output_attentions (bool, optional) — Whether or not to return the attentions tensors of all attention layers. See attentions under returned tensors for more detail.
• output_hidden_states (bool, optional) — Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for more detail.
• return_dict (bool, optional) — Whether or not to return a ModelOutput instead of a plain tuple.

A transformers.modeling_outputs.BaseModelOutputWithPooling or a tuple oftorch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various elements depending on the configuration (<class 'transformers.models.clap.configuration_clap.ClapAudioConfig'>) and inputs.

• last_hidden_state (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size)) — Sequence of hidden-states at the output of the last layer of the model.
• pooler_output (torch.FloatTensor of shape (batch_size, hidden_size)) — Last layer hidden-state of the first token of the sequence (classification token) after further processing through the layers used for the auxiliary pretraining task. E.g. for BERT-family of models, this returns the classification token after processing through a linear layer and a tanh activation function. The linear layer weights are trained from the next sentence prediction (classification) objective during pretraining.
• hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).
  Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.
• attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
  Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads.

The ClapAudioModel forward method, overrides the __call__ special method.

Although the recipe for forward pass needs to be defined within this function, one should call the Moduleinstance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Examples:

from datasets import load_dataset from transformers import AutoProcessor, ClapAudioModel

dataset = load_dataset("hf-internal-testing/ashraq-esc50-1-dog-example") audio_sample = dataset["train"]["audio"][0]["array"]

model = ClapAudioModel.from_pretrained("laion/clap-htsat-fused") processor = AutoProcessor.from_pretrained("laion/clap-htsat-fused")

inputs = processor(audios=audio_sample, return_tensors="pt")

outputs = model(**inputs) last_hidden_state = outputs.last_hidden_state

ClapAudioModelWithProjection

class transformers.ClapAudioModelWithProjection

< source >

( config: ClapAudioConfig )

Parameters

• config (ClapConfig) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the from_pretrained() method to load the model weights.

CLAP Audio Model with a projection layer on top (a linear layer on top of the pooled output).

This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.)

This model is also a PyTorch torch.nn.Module subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.

forward

< source >

( input_features: typing.Optional[torch.FloatTensor] = None is_longer: typing.Optional[torch.BoolTensor] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None ) → transformers.models.clap.modeling_clap.ClapAudioModelOutput or tuple(torch.FloatTensor)

Parameters

• input_features (torch.FloatTensor of shape (batch_size, num_channels, height, width)) — Input audio features. This should be returned by the ClapFeatureExtractor class that you can also retrieve from AutoFeatureExtractor. See ClapFeatureExtractor.__call__() for details.
• is_longer (torch.FloatTensor, of shape (batch_size, 1), optional) — Whether the audio clip is longer than max_length. If True, a feature fusion will be enabled to enhance the features.
• output_attentions (bool, optional) — Whether or not to return the attentions tensors of all attention layers. See attentions under returned tensors for more detail.
• output_hidden_states (bool, optional) — Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for more detail.
• return_dict (bool, optional) — Whether or not to return a ModelOutput instead of a plain tuple.

Returns

transformers.models.clap.modeling_clap.ClapAudioModelOutput or tuple(torch.FloatTensor)

A transformers.models.clap.modeling_clap.ClapAudioModelOutput or a tuple oftorch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various elements depending on the configuration (<class 'transformers.models.clap.configuration_clap.ClapAudioConfig'>) and inputs.

• audio_embeds (torch.FloatTensor of shape (batch_size, hidden_size)) — The Audio embeddings obtained by applying the projection layer to the pooler_output.
• last_hidden_state (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size)) — Sequence of hidden-states at the output of the last layer of the model.
• attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
  Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads.
• hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).
  Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.

The ClapAudioModelWithProjection forward method, overrides the __call__ special method.

Although the recipe for forward pass needs to be defined within this function, one should call the Moduleinstance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Examples:

from datasets import load_dataset from transformers import ClapAudioModelWithProjection, ClapProcessor

model = ClapAudioModelWithProjection.from_pretrained("laion/clap-htsat-fused") processor = ClapProcessor.from_pretrained("laion/clap-htsat-fused")

dataset = load_dataset("hf-internal-testing/ashraq-esc50-1-dog-example") audio_sample = dataset["train"]["audio"][0]["array"]

inputs = processor(audios=audio_sample, return_tensors="pt") outputs = model(**inputs) audio_embeds = outputs.audio_embeds

< > Update on GitHub