punct cap seg 47 language EU AI Act Compliance Profile

Model Overview

This model accepts as input lower-cased, unpunctuated, unsegmented text in 47 languages and performs punctuation restoration, true-casing (capitalization), and sentence boundary detection (segmentation).

All languages are processed with the same algorithm with no need for language tags or language-specific branches in the graph. This includes continuous-script and non-continuous script languages, predicting language-specific punctuation, etc.

This model is fun to play with, but the results could be better. I would recommend these newer, better models:

• Better English model
• Better Romance languages model
• Better 47-language

Usage

The easy way to use this model is to install punctuators:

pip install punctuators

Running the following script should load this model and run some texts:

from punctuators.models import PunctCapSegModelONNX

# Instantiate this model
# This will download the ONNX and SPE models. To clean up, delete this model from your HF cache directory.
m = PunctCapSegModelONNX.from_pretrained("pcs_47lang")

# Define some input texts to punctuate
input_texts: List[str] = [
    "hola mundo cómo estás estamos bajo el sol y hace mucho calor santa coloma abre los huertos urbanos a las escuelas de la ciudad",
    "hello friend how's it going it's snowing outside right now in connecticut a large storm is moving in",
    "未來疫苗將有望覆蓋3歲以上全年齡段美國與北約軍隊已全部撤離還有鐵路公路在內的各項基建的來源都將枯竭",
    "በባለፈው ሳምንት ኢትዮጵያ ከሶማሊያ 3 ሺህ ወታደሮቿንም እንዳስወጣች የሶማሊያው ዳልሳን ሬድዮ ዘግቦ ነበር ጸጥታ ሃይሉና ህዝቡ ተቀናጅቶ በመስራቱ በመዲናዋ ላይ የታቀደው የጥፋት ሴራ ከሽፏል",
    "all human beings are born free and equal in dignity and rights they are endowed with reason and conscience and should a

Your obligations as a punct cap seg 47 language deployer

Track your progress

Sign up to track obligations, receive deadline reminders, and generate audit packages.

Tags

Languages

Library