japanese avhubert base noise pt EU AI Act Compliance Profile

reazon-research/japanese-avhubert-base_noise_pt

This is a Japanese AVHuBERT (Audio-Visual Hidden Unit BERT) model pretrained with noise augmentation in the last iteration. AVHuBERT is a self-supervised model for AVSR (Audio-Visual Speech Recognition), robust for a noisy environment by leveraging both audio and visual inputs.

This model is pretrained on approximately 2,250h of Japanese audio-visual dataset with noise augmentation using environment noise, music, babble noise, and speech mixup.

Usage

Please install dependencies first.

$ pip install git+https://github.com/reazon-research/ReazonSpeech.git#subdirectory=pkg/avsr

Using transformers directly

from transformers import AutoFeatureExtractor, AutoModel

extractor = AutoFeatureExtractor.from_pretrained("reazon-research/japanese-avhubert-base_noise_pt", trust_remote_code=True)
model = AutoModel.from_pretrained("reazon-research/japanese-avhubert-base_noise_pt", trust_remote_code=True)

inputs = extractor(raw_audio="path/to/audio", raw_video="path/to/video")
# If mouth extraction is not performed, you can add `extract_mouth=True`
inputs = extractor(raw_audio="path/to/audio", raw_video="path/to/video", extract_mouth=True)

outputs = model(**inputs)

Using reazonspeech.avsr package

If you do not want to use trust_remote_code, please install reazonspeech.avsr.

from reazonspeech.avsr import AVHubertFeatureExtractor, AVHubertModel

extractor = AVHubertFeatureExtractor.from_pretrained("reazon-research/japanese-avhubert-base_noise_pt")
model = AVHubertModel.from_pretrained("reazon-research/japanese-avhubert-base_noise_pt")

inputs = extractor(raw_audio="path/to/audio", raw_video="path/to/video")
# If mouth extraction is not performed, you can add `extract_mouth=True`
inputs = extractor(raw_audio="path/to/audio", raw_video="path/to/video", extract_mouth=True)

outputs = model(**inputs)

Citation

Your obligations as a japanese avhubert base noise pt deployer

Track your progress

Sign up to track obligations, receive deadline reminders, and generate audit packages.

Tags

Languages

Library