Qwen3.5 9B AWQ EU AI Act Compliance Profile

Qwen3.5-9B-AWQ

Base model: Qwen/Qwen3.5-9B

This repo quantizes the model using data-free quantization technique.

【Dependencies / Installation】

vllm>=0.16.0rc2.dev447
transformers>=5.3.0.dev0

As of 2026-02-25, make sure your system has cuda12.8 installed.

Then, create a fresh Python environment (e.g. python3.12 venv) and run:

pip install -U vllm --pre --index-url https://pypi.org/simple --extra-index-url https://wheels.vllm.ai/nightly

# Note: important for old containers, or could face run time error like
# subprocess.CalledProcessError: 
# Command '['ninja', '-v', '-C', 
# '/root/.cache/flashinfer/0.6.3/90a/cached_ops/trtllm_comm',
# '-f', '/root/.cache/flashinfer/0.6.3/90a/cached_ops/trtllm_comm/build.ninja']'
# returned non-zero exit status 1. 
rm -rf ~/.cache/flashinfer

# upgrade transformers so that applications could properly execute tool calls
pip install -U "transformers @ git+https://github.com/huggingface/transformers.git@f2ba019"
# locate modeling_rope_utils.py line 651 to fix a simple bug
TF_FILE="$(python -m pip show transformers | awk -F': ' '/^Location:/{print $2}')/transformers/modeling_rope_utils.py" && echo "$TF_FILE"
NEW_LINE='            ignore_keys_at_rope_validation = set(ignore_keys_at_rope_validation) | {"partial_rotary_factor"}' \
perl -i.bak -pe 'if ($. == 651) { $_ = $ENV{NEW_LINE} . "\n" }' "$TF_FILE"

vLLM Official Guide

【vLLM Startup Command】

Your obligations as a Qwen3.5 9B AWQ deployer

Track your progress

Sign up to track obligations, receive deadline reminders, and generate audit packages.

Tags

Library

Base Model