moshiko-pytorch-bf16 EU AI Act Compliance Profile

Model Card for Moshi

Moshi is a speech-text foundation model and full-duplex spoken dialogue framework

Model Details

Pytorch version quantized in bf16 precision.

Model Description

Moshi is a speech-text foundation model that casts spoken dialogue as speech-to-speech generation. Starting from a text language model backbone, Moshi generates speech as tokens from the residual quantizer of a neural audio codec, while modeling separately its own speech and that of the user into parallel streams. This allows for the removal of explicit speaker turns, and the modeling of arbitrary conversational dynamics. Moshi also predicts time-aligned text tokens as a prefix to audio tokens. This “Inner Monologue” method significantly improves the linguistic quality of generated speech and provides streaming speech recognition and text-to-speech. As a result, Moshi is the first real-time full-duplex spoken large language model, with a theoretical latency of 160ms, 200ms in practice.

• Developed by: Kyutai
• Model type: Multimodal speech-text foundation model
• Language(s) (NLP): English
• License: CC-BY

Model Sources

• Repository: repo
• Paper: paper
• Demo: demo

Uses

Direct Use

The model can be used as a conversational agent for casual conversations, basic facts and advice (e.g. recipes, trivia), roleplay, etc. However, the model has limited abilities for complex tasks and cannot access tools, but rather focues on natural, low-latency interactions.

Downstream Use

Some components of the model can be used independently or repurposed relatively easily. For instance the Mimi codec is a state-of-the-art audio neural codec that combines semantic and acoustic information into audio tokens running at 12Hz and a bitrate of 1.1kbps, which make it particularly adapted to train speech language models or text-to-speech systems..

Your obligations as a moshiko-pytorch-bf16 deployer

Track your progress

Sign up to track obligations, receive deadline reminders, and generate audit packages.

Tags

Languages

Library