s2t medium mustc multilingual st EU AI Act Compliance Profile

S2T-MEDIUM-MUSTC-MULTILINGUAL-ST

s2t-medium-mustc-multilingual-st is a Speech to Text Transformer (S2T) model trained for end-to-end Multilingual Speech Translation (ST). The S2T model was proposed in this paper and released in this repository

Model description

S2T is a transformer-based seq2seq (encoder-decoder) model designed for end-to-end Automatic Speech Recognition (ASR) and Speech Translation (ST). It uses a convolutional downsampler to reduce the length of speech inputs by 3/4th before they are fed into the encoder. The model is trained with standard autoregressive cross-entropy loss and generates the transcripts/translations autoregressively.

Intended uses & limitations

This model can be used for end-to-end English speech to French text translation. See the model hub to look for other S2T checkpoints.

How to use

As this a standard sequence to sequence transformer model, you can use the generate method to generate the transcripts by passing the speech features to the model.

For multilingual speech translation models, eos_token_id is used as the decoder_start_token_id and the target language id is forced as the first generated token. To force the target language id as the first generated token, pass the forced_bos_token_id parameter to the generate() method. The following example shows how to transate English speech to French and German text using the facebook/s2t-medium-mustc-multilingual-st checkpoint.

Note: The Speech2TextProcessor object uses torchaudio to extract the filter bank features. Make sure to install the torchaudio package before running this example.

You could either install those as extra speech dependancies with pip install transformers"[speech, sentencepiece]" or install the packages seperatly wit

Your obligations as a s2t medium mustc multilingual st deployer

Track your progress

Sign up to track obligations, receive deadline reminders, and generate audit packages.

Tags

Languages

Library