rasyosef/splade-mini · Hugging Face (original) (raw)

SPLADE-Mini

This is a SPLADE sparse retrieval model based on BERT-Mini (11M) that was trained by distilling a Cross-Encoder on the MSMARCO dataset. The cross-encoder used was ms-marco-MiniLM-L6-v2.

This tiny SPLADE model is 6x smaller than Naver's official splade-v3-distilbert while having 85% of it's performance on the MSMARCO benchmark. This model is small enough to be used without a GPU on a dataset of a few thousand documents.

Performance

The splade models were evaluated on 55 thousand queries and 8.84 million documents from the MSMARCO dataset.

Size (# Params) MRR@10 (MS MARCO dev)
BM25 - 18.0
rasyosef/splade-tiny 4.4M 30.9
rasyosef/splade-mini 11.2M 34.1
rasyosef/splade-small 28.8M 35.4
naver/splade-v3-distilbert 67.0M 38.7

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SparseEncoder

# Download from the 🤗 Hub
model = SparseEncoder("yosefw/SPLADE-BERT-Mini-BS256-distil")
# Run inference
queries = [
    "common law implied warranty",
]
documents = [
    'The law recognizes two basic kinds of warrantiesimplied warranties and express warranties. Implied Warranties. Implied warranties are unspoken, unwritten promises, created by state law, that go from you, as a seller or merchant, to your customers.',
    'An implied warranty is a contract law term for certain assurances that are presumed in the sale of products or real property.',
    'The implied warranty of fitness for a particular purpose is a promise that the law says you, as a seller, make when your customer relies on your advice that a product can be used for some specific purpose.',
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings.shape, document_embeddings.shape)
# [1, 30522] [3, 30522]

# Get the similarity scores for the embeddings
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[22.4364, 22.7160, 21.7330]])

Model Details

Model Description

Model Sources

Full Model Architecture

SparseEncoder(
  (0): MLMTransformer({'max_seq_length': 512, 'do_lower_case': False, 'architecture': 'BertForMaskedLM'})
  (1): SpladePooling({'pooling_strategy': 'max', 'activation_function': 'relu', 'word_embedding_dimension': 30522})
)

More

Click to expand

Evaluation

Metrics

Sparse Information Retrieval

Metric Value
dot_accuracy@1 0.5018
dot_accuracy@3 0.8286
dot_accuracy@5 0.9194
dot_accuracy@10 0.9746
dot_precision@1 0.5018
dot_precision@3 0.2839
dot_precision@5 0.191
dot_precision@10 0.1026
dot_recall@1 0.4868
dot_recall@3 0.8148
dot_recall@5 0.9096
dot_recall@10 0.9709
dot_ndcg@10 0.7457
dot_mrr@10 0.6749
dot_map@100 0.6708
query_active_dims 22.585
query_sparsity_ratio 0.9993
corpus_active_dims 174.852
corpus_sparsity_ratio 0.9943

Training Details

Training Dataset

Unnamed Dataset

{  
    "loss": "SparseMarginMSELoss",  
    "document_regularizer_weight": 0.12,  
    "query_regularizer_weight": 0.2  
}

Training Hyperparameters

Non-Default Hyperparameters

All Hyperparameters

Click to expand

Training Logs

Epoch Step Training Loss dot_ndcg@10
1.0 15625 9.3147 0.7353
2.0 31250 7.5267 0.7429
3.0 46875 6.3289 0.7457

Framework Versions

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

SpladeLoss

@misc{formal2022distillationhardnegativesampling,
      title={From Distillation to Hard Negative Sampling: Making Sparse Neural IR Models More Effective},
      author={Thibault Formal and Carlos Lassance and Benjamin Piwowarski and Stéphane Clinchant},
      year={2022},
      eprint={2205.04733},
      archivePrefix={arXiv},
      primaryClass={cs.IR},
      url={https://arxiv.org/abs/2205.04733},
}

SparseMarginMSELoss

@misc{hofstätter2021improving,
    title={Improving Efficient Neural Ranking Models with Cross-Architecture Knowledge Distillation},
    author={Sebastian Hofstätter and Sophia Althammer and Michael Schröder and Mete Sertkan and Allan Hanbury},
    year={2021},
    eprint={2010.02666},
    archivePrefix={arXiv},
    primaryClass={cs.IR}
}

FlopsLoss

@article{paria2020minimizing,
    title={Minimizing flops to learn efficient sparse representations},
    author={Paria, Biswajit and Yeh, Chih-Kuan and Yen, Ian EH and Xu, Ning and Ravikumar, Pradeep and P{'o}czos, Barnab{'a}s},
    journal={arXiv preprint arXiv:2004.05665},
    year={2020}
}

Downloads last month

1,210

Model tree for rasyosef/splade-mini

Dataset used to train rasyosef/splade-mini

microsoft/ms_marco Viewer • Updated Jan 4, 2024• 1.11M • 21.6k • 238

Collection including rasyosef/splade-mini

Papers for rasyosef/splade-mini

Evaluation results