whisper large v3 word intelligibility synth aug pooled EU AI Act Compliance Profile

Whisper Large V3 - Word Intelligibility (Synth Aug Pooled)

Fine-tuned Whisper Large V3 for word-level intelligibility prediction.

Model Description

This model predicts word-level intelligibility scores (0-5 scale) from audio. It uses a finetuned Whisper decoder with a regression head on decoder hidden states.

Technical changes from previous version

1. Mean pooled ratings: Trained on mean-pooled rater scores instead of individual rater scores
2. Synthetic data augmentation: Training data augmented with word omissions and word substitutions

Evaluation

100 audio samples covering various scenarios (word omissions, word substitutions, etc.):

| Metric | Prod | This Model | |--------|-------|------------| | PCC | 0.720 | 0.838 | | SCC | 0.717 | 0.762 | | MAE | 0.985 | 0.635 |

Training Details

• Base model: openai/whisper-large-v3
• Encoder: Frozen
• Trainable params: ~637M (decoder) + 344K (head)

Usage

import torch
import torch.nn as nn
from transformers import WhisperForConditionalGeneration, WhisperProcessor

REPO = "boldvoice/whisper-large-v3-word-intelligibility-synth-aug-pooled"

# Load model
model = WhisperForConditionalGeneration.from_pretrained(REPO)
processor = WhisperProcessor.from_pretrained(REPO)
device = "cuda" if torch.cuda.is_available() else "cpu"
model.to(device)
model.eval()

# Load regression head
from huggingface_hub import hf_hub_download

head_path = hf_hub_download(REPO, "head.pt")
head_data = torch.load(head_path, map_location=device, weights_only=False)

class DecoderHiddenHead(nn.Module):
    def __init__(self, hidden_dim=1280, dropout=0.1):
        super().__init__()
        self.head = nn.Sequential(
            nn.Linear(hidden_dim, 256), nn.ReLU(), nn.Dropout(dropout),
            nn.Linear(256, 64), nn.ReLU(), nn.Dropout(dropout),
            nn.Linear(64, 1), nn.Sigmoid()
        )

    def forward(self, x):
        return self.head(x).squeeze(-1)

head = Dec

Your obligations as a whisper large v3 word intelligibility synth aug pooled deployer

Track your progress

Sign up to track obligations, receive deadline reminders, and generate audit packages.

Tags