Qwen2.5-1.5B-quantized.w8a8 EU AI Act Compliance Profile

Qwen2.5-1.5B-quantized.w8a8

Model Overview

• Model Architecture: Qwen2
  • Input: Text
  • Output: Text
• Model Optimizations:
  • Activation quantization: INT8
  • Weight quantization: INT8
• Intended Use Cases: Intended for commercial and research use multiple languages. Similarly to Qwen2.5-1.5B, this models is intended for assistant-like chat.
• Out-of-scope: Use in any manner that violates applicable laws or regulations (including trade compliance laws).
• Release Date: 10/09/2024
• Version: 1.0
• License(s): apache-2.0
• Model Developers: Neural Magic

Quantized version of Qwen2.5-1.5B. It achieves an average score of 58.34 on the OpenLLM benchmark (version 1), whereas the unquantized model achieves 58.48.

Model Optimizations

This model was obtained by quantizing the weights of Qwen2.5-1.5B to INT8 data type. This optimization reduces the number of bits used to represent weights and activations from 16 to 8, reducing GPU memory requirements (by approximately 50%) and increasing matrix-multiply compute throughput (by approximately 2x). Weight quantization also reduces disk size requirements by approximately 50%.

Only weights and activations of the linear operators within transformers blocks are quantized. Weights are quantized with a symmetric static per-channel scheme, where a fixed linear scaling factor is applied between INT8 and floating point representations for each output channel dimension. Activations are quantized with a symmetric dynamic per-token scheme, computing a linear scaling factor at runtime for each token between INT8 and floating point representations.

Deployment

This model can be deployed efficiently using the [

Your obligations as a Qwen2.5-1.5B-quantized.w8a8 deployer

Track your progress

Sign up to track obligations, receive deadline reminders, and generate audit packages.

Tags

Languages

Base Model