gemma 3 12b it INT4 EU AI Act Compliance Profile

INT4 google/gemma-3-12b-it model

• Developed by: pytorch
• License: apache-2.0
• Quantized from Model : google/gemma-3-12b-it
• Quantization Method : INT4
• Terms of Use: [Terms][terms]

Inference with vLLM

Install vllm nightly and torchao nightly to get some recent changes:

Serving

Then we can serve with the following command:

# Server
export MODEL=pytorch/gemma-3-12b-it-INT4
VLLM_DISABLE_COMPILE_CACHE=1 vllm serve $MODEL --tokenizer $MODEL -O3

# Client
curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{
  "model": "pytorch/gemma-3-12b-it-INT4",
  "messages": [
    {"role": "user", "content": "Give me a short introduction to large language models."}
  ],
  "temperature": 0.6,
  "top_p": 0.95,
  "top_k": 20,
  "max_tokens": 32768
}'

Note: please use VLLM_DISABLE_COMPILE_CACHE=1 to disable compile cache when running this code, e.g. VLLM_DISABLE_COMPILE_CACHE=1 python example.py, since there are some issues with the composability of compile in vLLM and torchao, this is expected be resolved in pytorch 2.8.

Inference with Transformers

Install the required packages:

pip install git+https://github.com/huggingface/transformers@main
pip install torchao
pip install torch
pip install accelerate

Example:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "pytorch/gemma-3-12b-it-INT4"

# load the tokenizer and the model
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype="auto",
    device_map="auto"
)

# prepare the model input
prompt = "Give me a short introduction to large language model."
messages = [
    {"role": "user", "content": prompt}
]
text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=T

Your obligations as a gemma 3 12b it INT4 deployer

Track your progress

Sign up to track obligations, receive deadline reminders, and generate audit packages.

Tags

Languages

Library

Base Model