ultravox-v0_5-llama-3_2-1b EU AI Act Compliance Profile

Model Card for Ultravox

Ultravox is a multimodal Speech LLM built around a pretrained Llama3.2-1B-Instruct and whisper-large-v3-turbo backbone.

See https://ultravox.ai for the GitHub repo and more information.

Model Details

Model Description

Ultravox is a multimodal model that can consume both speech and text as input (e.g., a text system prompt and voice user message). The input to the model is given as a text prompt with a special <|audio|> pseudo-token, and the model processor will replace this magic token with embeddings derived from the input audio. Using the merged embeddings as input, the model will then generate output text as usual.

In a future revision of Ultravox, we plan to expand the token vocabulary to support generation of semantic and acoustic audio tokens, which can then be fed to a vocoder to produce voice output. No preference tuning has been applied to this revision of the model.

• Developed by: Fixie.ai
• License: MIT

Model Sources

• Repository: https://ultravox.ai
• Demo: See repo

Usage

Think of the model as an LLM that can also hear and understand speech. As such, it can be used as a voice agent, and also to do speech-to-speech translation, analysis of spoken audio, etc.

To use the model, try the following:

# pip install transformers peft librosa

import transformers
import numpy as np
import librosa

pipe = transformers.pipeline(model='fixie-ai/ultravox-v0_5-llama-3_2-1b', trust_remote_code=True)

path = "<path-to-input-audio>"  # TODO: pass the audio here
audio, sr = librosa.load(path, sr=16000)


turns = [
  {
    "role": "system",
    "content": "You are a friendly and helpful character. You love to answer questions for people."
  },
]
pipe({'audio': audio, 'turns': turns, 'sampling_rate': sr}, max_new_tokens=30)

Training Details

The model uses a pre-trained [Llama3.2

Your obligations as a ultravox-v0_5-llama-3_2-1b deployer

Track your progress

Sign up to track obligations, receive deadline reminders, and generate audit packages.

Tags

Languages

Library