Qwen3.5 397B A17B int4 AutoRound EU AI Act Compliance Profile

Model Details

This model is a int4 model with group_size 128 of Qwen/Qwen3.5-397B-A17B generated by intel/auto-round. Please follow the license of the original model.

The model is quantized via RTN mode

vllm Infernece Example

pip install git+https://github.com/vllm-project/vllm.git@main


pip install git+https://github.com/huggingface/transformers.git

vllm serve Intel/Qwen3.5-397B-A17B-int4-AutoRound  --port 8000   --tensor-parallel-size 1  --max-model-len 2048 --reasoning-parser qwen3 --served-model-name qwen

curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d ' {
    "model": "qwen",
    "messages": [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": "Summarize Qwen 3.5 in one sentence."}
    ],
    "temperature": 1,
    "max_tokens": 512
  } '

Transformers Inference

Transformers v5.2.0

auto-round and This pr is required https://github.com/intel/auto-round/pull/1476

from transformers import AutoProcessor, Qwen3_5MoeForConditionalGeneration
model_name = "Intel/Qwen3.5-397B-A17B-int4-AutoRound"

model = Qwen3_5MoeForConditionalGeneration.from_pretrained(model_name, dtype="auto",
                                                                device_map="auto")
processor = AutoProcessor.from_pretrained(model_name)

messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "image",
                "image": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-VL/assets/demo.jpeg",
            },
            {"type": "text", "text": "Describe this image in short."},
        ],
    }
]


inputs = processor.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    return_dict=True,
    return_tensors="pt"
)
inputs = inputs.to(model.device)


generated_i

Your obligations as a Qwen3.5 397B A17B int4 AutoRound deployer

Track your progress

Sign up to track obligations, receive deadline reminders, and generate audit packages.

Tags

Base Model