SciPhi/SciPhi-Self-RAG-Mistral-7B-32k · Hugging Face (original) (raw)

Instructions to use SciPhi/SciPhi-Self-RAG-Mistral-7B-32k with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

• Libraries
• Transformers
  How to use SciPhi/SciPhi-Self-RAG-Mistral-7B-32k with Transformers:

Use a pipeline as a high-level helper

from transformers import pipeline
pipe = pipeline("text-generation", model="SciPhi/SciPhi-Self-RAG-Mistral-7B-32k")

Load model directly

from transformers import AutoTokenizer, AutoModelForMultimodalLM
tokenizer = AutoTokenizer.from_pretrained("SciPhi/SciPhi-Self-RAG-Mistral-7B-32k")
model = AutoModelForMultimodalLM.from_pretrained("SciPhi/SciPhi-Self-RAG-Mistral-7B-32k")

• Inference
• Notebooks
• Google Colab
• Kaggle
• Local Apps Settings
• vLLM
  How to use SciPhi/SciPhi-Self-RAG-Mistral-7B-32k with vLLM:

Install from pip and serve model

Install vLLM from pip:

pip install vllm

Start the vLLM server:

vllm serve "SciPhi/SciPhi-Self-RAG-Mistral-7B-32k"

Call the server using curl (OpenAI-compatible API):

curl -X POST "http://localhost:8000/v1/completions" \
-H "Content-Type: application/json" \
--data '{
"model": "SciPhi/SciPhi-Self-RAG-Mistral-7B-32k",
"prompt": "Once upon a time,",
"max_tokens": 512,
"temperature": 0.5
}'

Use Docker

docker model run hf.co/SciPhi/SciPhi-Self-RAG-Mistral-7B-32k

• SGLang
  How to use SciPhi/SciPhi-Self-RAG-Mistral-7B-32k with SGLang:

Install from pip and serve model

Install SGLang from pip:

pip install sglang

Start the SGLang server:

python3 -m sglang.launch_server \
--model-path "SciPhi/SciPhi-Self-RAG-Mistral-7B-32k" \
--host 0.0.0.0 \
--port 30000

Call the server using curl (OpenAI-compatible API):

curl -X POST "http://localhost:30000/v1/completions" \
-H "Content-Type: application/json" \
--data '{
"model": "SciPhi/SciPhi-Self-RAG-Mistral-7B-32k",
"prompt": "Once upon a time,",
"max_tokens": 512,
"temperature": 0.5
}'

Use Docker images

docker run --gpus all \
--shm-size 32g \
-p 30000:30000 \
-v ~/.cache/huggingface:/root/.cache/huggingface \
--env "HF_TOKEN=" \
--ipc=host \
lmsysorg/sglang:latest \
python3 -m sglang.launch_server \
--model-path "SciPhi/SciPhi-Self-RAG-Mistral-7B-32k" \
--host 0.0.0.0 \
--port 30000

Call the server using curl (OpenAI-compatible API):

curl -X POST "http://localhost:30000/v1/completions" \
-H "Content-Type: application/json" \
--data '{
"model": "SciPhi/SciPhi-Self-RAG-Mistral-7B-32k",
"prompt": "Once upon a time,",
"max_tokens": 512,
"temperature": 0.5
}'

• Docker Model Runner
  How to use SciPhi/SciPhi-Self-RAG-Mistral-7B-32k with Docker Model Runner:
  docker model run hf.co/SciPhi/SciPhi-Self-RAG-Mistral-7B-32k
• Browse Quantizations
  to use this model in llama.cpp, Ollama, LM Studio, or any compatible app.

SciPhi-Self-RAG-Mistral-7B-32k is a Large Language Model (LLM) fine-tuned from Mistral-7B-v0.1. This model underwent the fine-tuning process described in the SciPhi-Mistral-7B-32k model card. It then underwent further fine-tuning on the recently released self-rag dataset. Other RAG-related instruct datasets were mixed in during this process in an effort to keep the tone of the current model. This model benchmarks well, but it needs further tuning to be an excellent conversationalist.

Benchmark Results:

SciPhi-AI is available via a free hosted API, though the exposed model can vary. Currently, SciPhi-Self-RAG-Mistral-7B-32k is available. More details can be found in the docs here.

Recommended Chat Formatting

We recommend mapping such that

messages = [
    {
        "role": "system",
        "content": "You are a friendly chatbot who always responds in the style of a pirate",
    },
    {"role": "user", "content": "How many helicopters can a human eat in one sitting?"},
]

goes to --->

### System:
You are a friendly chatbot who always responds in the style of a pirate

### Instruction:
How many helicopters can a human eat in one sitting?

### Response:
...

Here is a sample implementation that does this and combines with RAG context retrieval.

def get_chat_completion(
    self, conversation: list[dict], generation_config: GenerationConfig
) -> str:
    self._check_stop_token(generation_config.stop_token)
    prompt = ""
    added_system_prompt = False
    for message in conversation:
        if message["role"] == "system":
            prompt += f"### System:\n{SciPhiLLMInterface.ALPACA_CHAT_SYSTEM_PROMPT}. Further, the assistant is given the following additional instructions - {message['content']}\n\n"
            added_system_prompt = True
        elif message["role"] == "user":
            last_user_message = message["content"]
            prompt += f"### Instruction:\n{last_user_message}\n\n"
        elif message["role"] == "assistant":
            prompt += f"### Response:\n{message['content']}\n\n"

    if not added_system_prompt:
        prompt = f"### System:\n{SciPhiLLMInterface.ALPACA_CHAT_SYSTEM_PROMPT}.\n\n{prompt}"

    context = self.rag_interface.get_contexts([last_user_message])[0]
    prompt += f"### Response:\n{SciPhiFormatter.RETRIEVAL_TOKEN} {SciPhiFormatter.INIT_PARAGRAPH_TOKEN}{context}{SciPhiFormatter.END_PARAGRAPH_TOKEN}"
    latest_completion = self.model.get_instruct_completion(
        prompt, generation_config
    ).strip()

    return SciPhiFormatter.remove_cruft(latest_completion)

Model Architecture

Base Model: Mistral-7B-v0.1

Architecture Features:

• Transformer-based model
• Grouped-Query Attention
• Sliding-Window Attention
• Byte-fallback BPE tokenizer

References

1. Asai, A., Wu, Z., Wang, Y., Sil, A., & Hajishirzi, H. (2023). Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection. arXiv preprint arXiv:2310.11511.
2. Lian, W., Goodson, B., Wang, G., Pentland, E., Cook, A., Vong, C., & Teknium. (2023). MistralOrca: Mistral-7B Model Instruct-tuned on Filtered OpenOrcaV1 GPT-4 Dataset. HuggingFace repository. Link
3. Mukherjee, S., Mitra, A., Jawahar, G., Agarwal, S., Palangi, H., & Awadallah, A. (2023). Orca: Progressive Learning from Complex Explanation Traces of GPT-4. arXiv preprint arXiv:2306.02707.
4. Longpre, S., Hou, L., Vu, T., Webson, A., Chung, H. W., Tay, Y., Zhou, D., Le, Q. V., Zoph, B., Wei, J., & Roberts, A. (2023). The Flan Collection: Designing Data and Methods for Effective Instruction Tuning. arXiv preprint arXiv:2301.13688.
5. Mistral AI. (2023). Model Card for Mistral-7B-v0.1. The Mistral-7B-v0.1 Large Language Model (LLM) is a pretrained generative text model with 7 billion parameters. Mistral-7B-v0.1 outperforms Llama 2 13B on all benchmarks tested. For full details, please refer to the paper and release blog post. Model Architecture: Transformer with Grouped-Query Attention, Sliding-Window Attention, and Byte-fallback BPE tokenizer. Link

Acknowledgements

Thank you to the AI Alignment Lab, vikp, jph00 and others who contributed to this work.

Model tree for SciPhi/SciPhi-Self-RAG-Mistral-7B-32k

Papers for SciPhi/SciPhi-Self-RAG-Mistral-7B-32k