Paper page - Matryoshka Representation Learning (original) (raw)
Published on May 26, 2022
Abstract
Matryoshka Representation Learning (MRL) adapts learned representations to varying computational constraints, offering smaller embedding sizes, faster retrievals, and enhanced accuracy in diverse tasks and modalities.
Learned representations are a central component in modern ML systems, serving a multitude of downstream tasks. When training such representations, it is often the case that computational and statistical constraints for each downstream task are unknown. In this context rigid, fixed capacity representations can be either over or under-accommodating to the task at hand. This leads us to ask: can we design a flexible representation that can adapt to multiple downstream tasks with varying computational resources? Our main contribution is Matryoshka Representation Learning (MRL) which encodes information at different granularities and allows a single embedding to adapt to the computational constraints of downstream tasks. MRL minimally modifies existing representation learning pipelines and imposes no additional cost during inference and deployment. MRL learns coarse-to-fine representations that are at least as accurate and rich as independently trained low-dimensional representations. The flexibility within the learned Matryoshka Representations offer: (a) up to 14x smaller embedding size for ImageNet-1K classification at the same level of accuracy; (b) up to 14x real-world speed-ups for large-scale retrieval on ImageNet-1K and 4K; and (c) up to 2% accuracy improvements forlong-tail few-shot classification, all while being as robust as the original representations. Finally, we show that MRL extends seamlessly to web-scale datasets (ImageNet, JFT) across various modalities -- vision (ViT, ResNet), vision + language (ALIGN) and language (BERT). MRL code and pretrained models are open-sourced at https://github.com/RAIVNLab/MRL.
View arXiv page View PDF Add to collection
Get this paper in your agent:
hf papers read 2205.13147
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash
Models citing this paper 1,000+
nomic-ai/nomic-embed-text-v1.5 Sentence Similarity • 0.1B • Updated 2 days ago • 10.7M • 793
nomic-ai/nomic-embed-text-v2-moe Sentence Similarity • 0.5B • Updated Apr 1, 2025 • 1.86M • 464
NovaSearch/stella_en_1.5B_v5 Sentence Similarity • Updated Jul 28, 2025 • 32k • 261
NovaSearch/stella_en_400M_v5 Sentence Similarity • 0.4B • Updated Jul 28, 2025 • 1.23M • 233
Browse 1,000+ models citing this paper
Datasets citing this paper 0
No dataset linking this paper
Cite arxiv.org/abs/2205.13147 in a dataset README.md to link it from this page.