GitHub - mistralai/mistral-common: Official inference library for pre-processing of Mistral models (original) (raw)

What is it?

mistral-common is a set of tools to help you work with Mistral AI models.

We open-source the tokenizers, validation and normalization code that can be used with our models.

This ensures that you can take full advantage of our models for the following features:

• tokenization of text, images and tools calls.
• validation and normalization of requests, messages, tool calls, and responses. This is built on top of the Pydantic library.

We also version our tokenizers to guarantee backward compatibility for the models that we release.

For who ?

This library is for you if you want to:

• use our models in your own application.
• build your own models and want to use the same tokenization and validation code as we do.

How to use it ?

You can install the library using pip:

pip install mistral-common

We propose different dependencies to install depending on your needs:

• image: to use the image tokenizers.
• audio: to use the audio tokenizers.
• hf-hub: to download the tokenizers from the Hugging Face Hub.
• sentencepiece: to allow the use of SentencePiece tokenizers. This is now optional as we only release Tekken tokenizers for recent models.
• [Experimental] server: to use our tokenizers in a server mode.

Each dependency is optional and can be installed separately or all together using the following commands:

pip install "mistral-common[image]" pip install "mistral-common[audio]" pip install "mistral-common[hf-hub]" pip install "mistral-common[sentencepiece]" pip install "mistral-common[server]" pip install "mistral-common[image,audio,hf-hub,sentencepiece,server]"

For more information, please refer to the documentation.

How to contribute ?

We welcome contributions to this library. All of our features are tested to ensure best usage. But if you encounter a bug, find difficulties in using mistral-common, please open an issue on our GitHub repository.

If you want to contribute to the code, please fork the repository and open a pull request. We will review your changes and merge them if they are relevant to the library.

Clone the repository

git clone https://github.com//mistral-common.git cd mistral-common

Create a virtual environment and install the dependencies

uv venv source .venv/bin/activate uv sync --frozen --all-extras --group dev # --group docs if you want to build the documentation

Install the pre-commit hooks

uv run pre-commit install

License

This library is licensed under the Apache 2.0 License. See the LICENCE file for more information.

You must not use this library or our models in a manner that infringes, misappropriates, or otherwise violates any third party’s rights, including intellectual property rights.