CohereLabs/wikipedia-2023-11-embed-multilingual-v3 · Datasets at Hugging Face (original) (raw)

Multilingual Embeddings for Wikipedia in 300+ Languages

This dataset contains the wikimedia/wikipedia dataset dump from 2023-11-01 from Wikipedia in all 300+ languages.

The individual articles have been chunked and embedded with the state-of-the-art multilingual Cohere Embed V3 embedding model. This enables an easy way to semantically search across all of Wikipedia or to use it as a knowledge source for your RAG application. In total is it close to 250M paragraphs / embeddings.

You can also use the model to perform cross-lingual search: Enter your search query in any language and get the most relevant results back.

Loading the dataset

Loading the document embeddings

You can either load the dataset like this:

from datasets import load_dataset
lang = "simple" #Use the Simple English Wikipedia subset
docs = load_dataset("CohereLabs/wikipedia-2023-11-embed-multilingual-v3", lang, split="train")

Or you can also stream it without downloading it before:

from datasets import load_dataset
lang = "simple" #Use the Simple English Wikipedia subset
docs = load_dataset("CohereLabs/wikipedia-2023-11-embed-multilingual-v3", lang, split="train", streaming=True)
for doc in docs:
    doc_id = doc['_id']
    title = doc['title']
    text = doc['text']
    emb = doc['emb']

Note, depending on the language, the download can be quite large.

Search

A full search example (on the first 1,000 paragraphs):

#Run: pip install cohere datasets numpy
from datasets import load_dataset
import numpy as np
import cohere

co = cohere.Client("YOUR_COHERE_API_KEY")  # Add your cohere API key from www.cohere.com
lang = "simple"
top_k = 3

#Load at max 1000 chunks + embeddings
max_docs = 1000
docs_stream = load_dataset(f"CohereLabs/wikipedia-2023-11-embed-multilingual-v3", lang, split="train", streaming=True)

docs = []
doc_embeddings = []

for doc in docs_stream:
    docs.append(doc)
    doc_embeddings.append(doc['emb'])
    if len(docs) >= max_docs:
        break

doc_embeddings = np.asarray(doc_embeddings)

query = 'Who was Alan Turing'
response = co.embed(texts=[query], model='embed-multilingual-v3.0', input_type="search_query")
query_embedding = response.embeddings 
query_embedding = np.asarray(query_embedding)

# Compute dot score between query embedding and document embeddings
dot_scores = np.matmul(query_embedding, doc_embeddings.transpose())[0]
top_k_hits = np.argpartition(dot_scores, -top_k)[-top_k:].tolist()

# Sort top_k_hits by dot score
top_k_hits.sort(key=lambda x: dot_scores[x], reverse=True)

# Print results
print("Query:", query)
for doc_id in top_k_hits:
    print(docs[doc_id]['title'])
    print(docs[doc_id]['text'])
    print(docs[doc_id]['url'], "\n")

Overview

The following table contains all language codes together with the total numbers of passages.

Downloads last month

11,472

Models trained or fine-tuned on CohereLabs/wikipedia-2023-11-embed-multilingual-v3