thainer corpus v2 base model EU AI Act Compliance Profile

This is a Named Entity Recognition model that trained with Thai NER v2.0 Corpus

Training script and split data: https://zenodo.org/record/7761354

The model was trained by WangchanBERTa base model.

Validation from the Validation set

• Precision: 0.830336794125095
• Recall: 0.873701039168665
• F1: 0.8514671513892494
• Accuracy: 0.9736483416628805

Test from the Test set

• Precision: 0.8199168093956447
• Recall: 0.8781446540880503
• F1: 0.8480323927622422
• Accuracy: 0.9724346779516247

Download: HuggingFace Hub

Read more: Thai NER v2.0

Inference

Huggingface doesn't support inference token classification for Thai and It will give wrong tag. You must using this code.

from transformers import AutoTokenizer
from transformers import AutoModelForTokenClassification
from pythainlp.tokenize import word_tokenize # pip install pythainlp
import torch

name="pythainlp/thainer-corpus-v2-base-model"
tokenizer = AutoTokenizer.from_pretrained(name)
model = AutoModelForTokenClassification.from_pretrained(name)

sentence="ฉันชื่อ นางสาวมะลิวา บุญสระดี อาศัยอยู่ที่อำเภอนางรอง จังหวัดบุรีรัมย์ อายุ 23 ปี เพิ่งเรียนจบจาก มหาวิทยาลัยขอนแก่น และนี่คือข้อมูลปลอมชื่อคนไม่มีอยู่จริง อายุ 23 ปี"
cut=word_tokenize(sentence.replace(" ", "<_>"))
inputs=tokenizer(cut,is_split_into_words=True,return_tensors="pt")

ids = inputs["input_ids"]
mask = inputs["attention_mask"]
# forward pass
outputs = model(ids, attention_mask=mask)
logits = outputs[0]

predictions = torch.argmax(logits, dim=2)
predicted_token_class = [model.config.id2label[t.item()] for t in predictions[0]]

def fix_span_error(words,ner):
    _ner = []
    _ner=ner
    _new_tag=[]
    for i,j in zip(words,_ner):
        #print(i,j)

Your obligations as a thainer corpus v2 base model deployer

Track your progress

Sign up to track obligations, receive deadline reminders, and generate audit packages.

Tags

Languages

Library