Qwen3 8B WMXFP4FP8 AMXFP4FP8 AMP KVFP8 EU AI Act Compliance Profile

Qwen3-8B-WMXFP4FP8-AMXFP4FP8-AMP-KVFP8

• Introduction

  This model was created by applying Quark with calibration samples from Pile dataset.

• Quantization Stragegy

  • Quantized Layers: All linear layers excluding "lm_head"
  • Weight: Auto Mixed Precision quantized by Quark, each weight has either quantization scheme in candidates of
    • FP8 symmetric per-tensor
    • OCP Microscaling (MX) FP4
  • Activation: Auto Mixed Precision quantized by Quark, each activation input has the same quantization scheme with weight, i.e., in candidates of
    • FP8 symmetric per-tensor
    • OCP Microscaling (MX) FP4
  • KV Cache: FP8 symmetric per-tensor

• Quick Start

1. Download and install Quark
2. [TODO] We will provide example script(s) to run auto mixed precision (AMP) quantizations later.

Deployment

The Quark quantized Auto Mixed Precision (AMP) models are now supported to be easily deployed in vLLM backend (vLLM-compatible).

Evaluation

The quantization evaluation results are conducted in pseudo-quantization mode, which may slightly differ from the actual quantized inference accuracy. These results are provided for reference only.

Evaluation scores

| Quant scheme | arc challenge (↑)<br>(acc) | | gsm8k (↑)<br>(strict-match) | | mmlu (↑)<br>(acc) | | winogrande (↑)<br>(acc) | | |--------------|-----------------------------|-------|-----------------------------|-------|-------------------|-------|--------------------------|-------| | | absolute value | recovery rate | absolute value | recovery rate | absolute value | recovery rate | absolute value | recovery rate | | BF16 | 0.5597 | 100.0% | 0.8552 | 100.0% | 0.7296 | 100.0% | 0.6780 | 100.0% | | FP8 | 0.5563

Your obligations as a Qwen3 8B WMXFP4FP8 AMXFP4FP8 AMP KVFP8 deployer

Track your progress

Sign up to track obligations, receive deadline reminders, and generate audit packages.

Tags

Base Model