Llama 3.3 70B Instruct FP8 block EU AI Act Compliance Profile

Llama-3.3-70B-Instruct-FP8-block

Model Overview

• Model Architecture: LlamaForCausalLM
  • Input: Text
  • Output: Text
• Model Optimizations:
  • Weight quantization: FP8
  • Activation quantization: FP8
• Release Date:
• Version: 1.0
• Model Developers:: Red Hat

Quantized version of meta-llama/Llama-3.3-70B-Instruct.

Model Optimizations

This model was obtained by quantizing the weights and activations of meta-llama/Llama-3.3-70B-Instruct to FP8 data type. This optimization reduces the number of bits per parameter from 16 to 8, reducing the disk size and GPU memory requirements by approximately 50%. Only the weights and activations of the linear operators within transformers blocks of the language model are quantized.

Deployment

Use with vLLM

1. Initialize vLLM server:

1. Send requests to the server:

from openai import OpenAI

# Modify OpenAI's API key and API base to use vLLM's API server.
openai_api_key = "EMPTY"
openai_api_base = "http://<your-server-host>:8000/v1"

client = OpenAI(
    api_key=openai_api_key,
    base_url=openai_api_base,
)

model = "nm-testing/Llama-3.3-70B-Instruct-FP8-block"


messages = [
    {"role": "user", "content": "Explain quantum mechanics clearly and concisely."},
]

outputs = client.chat.completions.create(
    model=model,
    messages=messages,
)

generated_text = outputs.choices[0].message.content
print(generated_text)

Creation

This model was quantized using the llm-compressor library as shown below.

from transformers import AutoProcessor, LlamaForCausalLM

from llmcompressor import oneshot
from llmcompressor.modeling import replace_mod

Your obligations as a Llama 3.3 70B Instruct FP8 block deployer

Track your progress

Sign up to track obligations, receive deadline reminders, and generate audit packages.

Tags

Base Model