juribert base EU AI Act Compliance Profile

JuriBERT: A Masked-Language Model Adaptation for French Legal Text

Introduction

JuriBERT is a set of BERT models (tiny, mini, small and base) pre-trained from scratch on French legal-domain specific corpora. JuriBERT models are pretrained on 6.3GB of legal french raw text from two different sources: the first dataset is crawled from Légifrance and the other one consists of anonymized court’s decisions and the pleadings from the Court of Cassation (mémoires ampliatifs). The latter contains more than 100k long documents from different court cases.

It is now on Hugging Face in four different versions with varying number of parameters.

JuriBERT Pre-trained models

| Model | #params | Architecture | |--------------------------------|--------------------------------|-------| | dascim/juribert-tiny | 6M | Tiny (L=2, H=128, A=2) | | dascim/juribert-mini | 15M | Mini (L=4, H=256, A=4)| | dascim/juribert-small | 42M | Small (L=6, H=512, A=8)| | dascim/juribert-base | 110M | Base (L=12, H=768, A=12) |

JuriBERT Usage

Load JuriBERT and its sub-word tokenizer :

from transformers import AutoModel, AutoTokenizer

# You can replace "juribert-base" with any other model from the table, e.g. "dascim/juribert-small".
tokenizer = AutoTokenizer.from_pretrained("dascim/juribert-base")
juribert = AutoModel.from_pretrained("dascim/juribert-base")

juribert.eval()  # disable dropout (or leave in train mode to finetune)

Filling masks using pipeline

from transformers import pipeline 

juribert_fill_mask  = pipeline("fill-mask", model="dascim/juribert-base", tokenizer="dascim/juribert-base")
results = juribert_fill_mask("la chambre <mask> est une chambre de la cour de cassation.")
# results
# [{'score': 0.3455437421798706, 'token': 579, 'token_str': ' civile', 'sequence': 'la chambre civile est une chamb

Your obligations as a juribert base deployer

Track your progress

Sign up to track obligations, receive deadline reminders, and generate audit packages.

Tags

Languages

Library