Legal BERTimbau base EU AI Act Compliance Profile

Legal_BERTimbau

Introduction

Legal_BERTimbau Large is a fine-tuned BERT model based on BERTimbau Large.

"BERTimbau Base is a pretrained BERT model for Brazilian Portuguese that achieves state-of-the-art performances on three downstream NLP tasks: Named Entity Recognition, Sentence Textual Similarity and Recognizing Textual Entailment. It is available in two sizes: Base and Large.

For further information or requests, please go to BERTimbau repository."

The performance of Language Models can change drastically when there is a domain shift between training and test data. In order create a Portuguese Language Model adapted to a Legal domain, the original BERTimbau model was submitted to a fine-tuning stage where it was performed 1 "PreTraining" epoch over 30 000 legal Portuguese Legal documents available online.

Available models

| Model | Arch. | #Layers | #Params | | ---------------------------------------- | ---------- | ------- | ------- | | rufimelo/Legal-BERTimbau-base | BERT-Base |12 |110M| | rufimelo/Legal-BERTimbau-large | BERT-Large | 24 | 335M |

Usage

from transformers import AutoTokenizer, AutoModelForMaskedLM

tokenizer = AutoTokenizer.from_pretrained("rufimelo/Legal-BERTimbau-base")

model = AutoModelForMaskedLM.from_pretrained("rufimelo/Legal-BERTimbau-base")

Masked language modeling prediction example

from  transformers  import  pipeline
from  transformers  import  AutoTokenizer, AutoModelForMaskedLM

tokenizer = AutoTokenizer.from_pretrained("rufimelo/Legal-BERTimbau-base")
model = AutoModelForMaskedLM.from_pretrained("rufimelo/Legal-BERTimbau-base")

pipe = pipeline('fill-mask', model=model, tokenizer=tokenizer)
pipe('O advogado apresentou [MASK] para o juíz')
# [{'score': 0.5034703612327576, 
#'token': 8190, 
#'token_str': 'recurso',

Your obligations as a Legal BERTimbau base deployer

Track your progress

Sign up to track obligations, receive deadline reminders, and generate audit packages.

Tags

Languages

Library