tts-0.75b-en-public EU AI Act Compliance Profile

Model Card for Kyutai TTS (public data)

See also the pre-print research paper, the project page, the GitHub repository, and the evaluation pipeline.

This is a model for streaming text-to-speech (TTS). Unlike offline text-to-speech, where the model needs the entire text to produce the audio, our model starts to output audio as soon as the first few words from the text have been given as input. This model was trained on a mixed of public TTS datasets, allowing for a fair comparisons with other methods.

Model Details

The model architecture is a hierarchical Transformer that consumes tokenized text and generateds audio tokenized by Mimi, see the Moshi paper. The frame rate is 12.5 Hz and each audio frame is represented by 16 audio tokens. You cannot use less tokens at inference. The backbone model is 300M parameters, and the depth transformer is 450M parameters and uses partial weight sharing similar to Hibiki. The audio is shifted by 16 steps (1.28 sec.) with respect to the text, and the model uses an acoustic/semantic delay of 2.

Model Description

Kyutai TTS is a decoder-only model for streaming speech-to-text. It leverages the multistream architecture of Moshi to model text stream based on the speech stream. The text stream is shifted w.r.t. the audio stream to allow the model to predict text tokens based on the input audio.

• Developed by: Kyutai
• Model type: Streaming Text-To-Speech.
• Language(s) (NLP): English
• License: Model weights are licensed under CC-BY 4.0
• Repository: GitHub

Uses

Direct Use

This model is able to perform streaming text-to-speech generation. This model allows for voice cloning through pref

Your obligations as a tts-0.75b-en-public deployer

Track your progress

Sign up to track obligations, receive deadline reminders, and generate audit packages.

Tags

Languages

Library