Llama-3.2-1B-Instruct-FP8 EU AI Act Compliance Profile

Llama-3.2-1B-Instruct-FP8

Model Overview

• Model Architecture: Llama-3
  • Input: Text
  • Output: Text
• Model Optimizations:
  • Activation quantization: FP8
  • Weight quantization: FP8
• Intended Use Cases: Intended for commercial and research use multiple languages. Similarly to Llama-3.2-1B-Instruct, this models is intended for assistant-like chat.
• Out-of-scope: Use in any manner that violates applicable laws or regulations (including trade compliance laws).
• Release Date: 9/25/2024
• Version: 1.0
• License(s): Llama3.2
• Model Developers: Neural Magic

Quantized version of Llama-3.2-1B-Instruct. It achieves scores within 1.0% of the scores of the unquantized model for MMLU, ARC-Challenge, GSM-8k, Hellaswag, Winogrande and TruthfulQA.

Model Optimizations

This model was obtained by quantizing the weights of Llama-3.2-1B-Instruct to FP8 data type. This optimization reduces the number of bits used to represent weights and activations from 16 to 8, reducing GPU memory requirements (by approximately 50%) and increasing matrix-multiply compute throughput (by approximately 2x). Weight quantization also reduces disk size requirements by approximately 50%.

Only weights and activations of the linear operators within transformers blocks are quantized. Weights are quantized with a symmetric static per-channel scheme, where a fixed linear scaling factor is applied between FP8 and floating point representations for each output channel dimension. Activations are quantized with a symmetric per-tensor scheme, where a fixed linear scaling factor is applied between FP8 and floating point representations for the entire activation tensor. Weights are quantized by rounding to nearest FP8 representation. The [llm-compressor](https://github.com/vl

Your obligations as a Llama-3.2-1B-Instruct-FP8 deployer

Track your progress

Sign up to track obligations, receive deadline reminders, and generate audit packages.

Tags

Languages

Base Model