Mistral Small 3.1 24B Instruct 2503 int4 AutoRound awq sym EU AI Act Compliance Profile

Model Details

This model is an int4 model with group_size 128 and symmetric quantization of mistralai/Mistral-Small-3.1-24B-Instruct-2503 generated by intel/auto-round algorithm.

Please follow the license of the original model.

INT4 Inference

Requirements

transformers>=4.52.0, auto-round>0.5.1

Note: There is no official HuggingFace sample code of the original model. The following code may have issues.

from transformers import AutoProcessor, Mistral3ForConditionalGeneration, AutoTokenizer
from huggingface_hub import hf_hub_download
import torch
from datetime import datetime, timedelta

model_id = "OPEA/Mistral-Small-3.1-24B-Instruct-2503-int4-AutoRound-awq-sym"

def load_system_prompt(repo_id: str, filename: str) -> str:
    file_path = hf_hub_download(repo_id=repo_id, filename=filename)
    with open(file_path, "r") as file:
        system_prompt = file.read()
    today = datetime.today().strftime("%Y-%m-%d")
    yesterday = (datetime.today() - timedelta(days=1)).strftime("%Y-%m-%d")
    model_name = repo_id.split("/")[-1]
    return system_prompt.format(name=model_name, today=today, yesterday=yesterday)


SYSTEM_PROMPT = load_system_prompt(model_id, "SYSTEM_PROMPT.txt")

model = Mistral3ForConditionalGeneration.from_pretrained(
    model_id, torch_dtype=torch.float16, device_map="auto"
).eval()

processor = AutoProcessor.from_pretrained(model_id)
tokenizer = AutoTokenizer.from_pretrained(model_id)
url = "https://huggingface.co/datasets/patrickvonplaten/random_img/resolve/main/europe.png"

prompt = "Which of the depicted countries has the best food? Which the second and third and fourth? Name the country, its color on the map and one its city that is visible on the map, but is not the capital. Make absolutely sure to only name a city that can be seen on the map."

messages = [
    {"role": "system", "content": SYSTEM_PROMPT},

Your obligations as a Mistral Small 3.1 24B Instruct 2503 int4 AutoRound awq sym deployer

Track your progress

Sign up to track obligations, receive deadline reminders, and generate audit packages.

Tags

Base Model