spear xlarge speech audio EU AI Act Compliance Profile

SPEAR XLarge (speech + general audio)

This is the SPEAR XLarge dual-domain (speech + general audio) model. The model adopts a Zipformer backbone with 597M parameters consisting of 13 Zipformer stacks. It generates 1280-dimmensional representations at approximately 50~Hz.

The model was pre-trained on 197k hours of mixture data of English speech and general audio, among which 184k hours are speech data, and the rest 13k hours are general audio data. It achieves state-of-the-art performance on SUPERB benchmark and competitive performance on HEAR.

The speech data consists of the following datasets:

| Dataset | Duration (hours) | | ---------- | ------- | | Libriheavy | ~50k | | Gigaspeech | ~10k | | VoxPopuli (en) | ~24k | | yodas-granary | ~100k |

The audio data consists of the following datasets:

| Dataset | Duration (hours) | | ---------- | ------- | | AudioSet | ~5k | | Freesound | ~2.8k | | Music4all | ~1k | | VGGSound | ~ 0.5k | | MTG-Jamendo | ~ 3.8k |

Note: The model is pretrained on 16kHz sampled speech/audio data. When using the model make sure that your input is also sampled at 16kHz.

Paper

Authors: Xiaoyu Yang, Yifan Yang, Zengrui Jin, Ziyun Cui, Wen Wu, Baoxiang Li, Chao Zhang, Phil Woodland

Abstract Self-Supervised Learning (SSL) excels at learning generic representations of acoustic signals, yet prevailing methods remain domain-specific, tailored to either speech or general audio, hindering the development of a unified representation model with a comprehensive capability over both domains. To address this, we present SPEAR (SPEech and Audio Representations), the first SSL framework to successfully learn unified speech and audio representations from a mixture of speech and audio data. SPEAR proposes a unified pre-training objective based on masked

Your obligations as a spear xlarge speech audio deployer

Track your progress

Sign up to track obligations, receive deadline reminders, and generate audit packages.

Tags