pe a frame large EU AI Act Compliance Profile

Perception Encoder Audio Frame (PE-A-Frame)

PE-A-Frame is a state-of-the-art audio-text embedding model. For text, the model produces a single embedding. For audio, it produces a sequence of embeddings (onen for every 40ms of audio). These embeddings can then be used for audio event localization. For convienience, model outputs temporal spans (start and end timestamps) indicating when that event (freeform audio description) occurs in the audio.

Model Description

PE-A-Frame uses contrastive learning to align frame-level audio representations with text descriptions. The model can identify precise time ranges when described audio events occur

Model Variants

We release multiple model checkpoints with varying sizes:

| Model | Parameters | |-------|------------| | pe-a-frame-small | 450M | | pe-a-frame-base | 560M | | pe-a-frame-large | 1.4B |

Quick Start

Basic Usage: Audio Event Localization

import torch
from core.audio_visual_encoder import PEAudioFrame, PEAudioFrameTransform

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

# Load model and transform
model = PEAudioFrame.from_config("pe-a-frame-large", pretrained=True).to(device)
transform = PEAudioFrameTransform.from_config("pe-a-frame-large")

# Define audio file and event descriptions
audio_file = "office_conversation.wav"
descriptions = ["a person talking", "keyboard typing", "phone ringing"]

# Process inputs
inputs = transform(audio=[audio_file], text=descriptions).to(device)

# Run inference
with torch.inference_mode():
    outputs = model(**inputs, return_spans=True)

# Print detected time spans for each event
for description, spans in zip(descriptions, outputs.spans):
    if spans:
        span_str = ", ".join([f"({start:.2f}s, {end:.2f}s)" for start, end in spans])
        print(f'"{descripti

Your obligations as a pe a frame large deployer

Track your progress

Sign up to track obligations, receive deadline reminders, and generate audit packages.

Tags