ultraVAD EU AI Act Compliance Profile

Model Card: UltraVAD

UltraVAD is a context-aware, audio-native endpointing model. It estimates the probability that a speaker has finished their turn in real time by fusing recent dialog text with the user’s audio. UltraVAD consumes the dialogue history and the last user audio turn, then produces a probability for the end-of-turn token <|eot_id|>.

Model Details

• Developer: Ultravox.ai
• Type: Context-aware audio–text fusion endpointing
• Backbone: Llama-8B (post-trained)
• Languages (26): ar, bg, zh, cs, da, nl, en, fi, fr, de, el, hi, hu, it, ja, pl, pt, ro, ru, sk, es, sv, ta, tr, uk, vi

What it predicts. UltraVAD computes the probability P(<eot_id> | context, user_audio)

Sources

• Website/Repo: https://ultravox.ai
• Demo: https://demo.ultravox.ai/
• Benchmark: https://huggingface.co/datasets/fixie-ai/turntaking-contextual-tts
• Blog: https://www.ultravox.ai/blog/ultravad-is-now-open-source-introducing-the-first-context-aware-audio-native-endpointing-model

Usage

Use UltraVAD as a turn-taking oracle in voice agents. Run it alongside a lightweight streaming VAD; when short silences are detected, call UltraVAD and trigger your agent’s response once the <eot> probability crosses your threshold.

import transformers
import torch
import librosa
import os

pipe = transformers.pipeline(model='fixie-ai/ultraVAD', trust_remote_code=True, device="cpu")

sr = 16000
wav_path = os.path.join(os.path.dirname(__file__), "sample.wav")
audio, sr = librosa.load(wav_path, sr=sr)

turns = [
  {"role": "assistant", "content": "Hi, how are you?"},
]

# Build model inputs via pipeline preprocess
inputs = {"audio": audio, "turns": turns, "sampling_rate": sr}
model_inputs = pipe.preprocess(inputs)

# Move tensors to model device
device = next(pipe.model.parameters()).device
model_inputs = {k: (v.to(device) if hasattr(v, "to") else v) for k, v in model_inputs.items()}

# Forward pass (no generation)
with torch.

Your obligations as a ultraVAD deployer

Track your progress

Sign up to track obligations, receive deadline reminders, and generate audit packages.

Tags

Library