Mistral Small Instruct 2409 EU AI Act Compliance Profile

Model Card for Mistral-Small-Instruct-2409

Mistral-Small-Instruct-2409 is an instruct fine-tuned version with the following characteristics:

• 22B parameters
• Vocabulary to 32768
• Supports function calling
• 32k sequence length

Usage Examples

vLLM (recommended)

We recommend using this model with the vLLM library to implement production-ready inference pipelines.

Installation

Make sure you install vLLM >= v0.6.1.post1:

Also make sure you have mistral_common >= 1.4.1 installed:

You can also make use of a ready-to-go docker image.

Offline

from vllm import LLM
from vllm.sampling_params import SamplingParams

model_name = "mistralai/Mistral-Small-Instruct-2409"

sampling_params = SamplingParams(max_tokens=8192)

# note that running Mistral-Small on a single GPU requires at least 44 GB of GPU RAM
# If you want to divide the GPU requirement over multiple devices, please add *e.g.* `tensor_parallel=2`
llm = LLM(model=model_name, tokenizer_mode="mistral", config_format="mistral", load_format="mistral")

prompt = "How often does the letter r occur in Mistral?"

messages = [
    {
        "role": "user",
        "content": prompt
    },
]

outputs = llm.chat(messages, sampling_params=sampling_params)

print(outputs[0].outputs[0].text)

Server

You can also use Mistral Small in a server/client setting.

1. Spin up a server:

Note: Running Mistral-Small on a single GPU requires at least 44 GB of GPU RAM.

If you want to divide the GPU requirement over multiple devices, please add e.g. --tensor_parallel=2

1. And pin

Your obligations as a Mistral Small Instruct 2409 deployer

Track your progress

Sign up to track obligations, receive deadline reminders, and generate audit packages.

Tags

Languages

Library