flaubert small cased EU AI Act Compliance Profile

FlauBERT: Unsupervised Language Model Pre-training for French

FlauBERT is a French BERT trained on a very large and heterogeneous French corpus. Models of different sizes are trained using the new CNRS (French National Centre for Scientific Research) Jean Zay supercomputer.

Along with FlauBERT comes FLUE: an evaluation setup for French NLP systems similar to the popular GLUE benchmark. The goal is to enable further reproducible experiments in the future and to share models and progress on the French language.For more details please refer to the official website.

FlauBERT models

| Model name | Number of layers | Attention Heads | Embedding Dimension | Total Parameters | | :------: | :---: | :---: | :---: | :---: | | flaubert-small-cased | 6 | 8 | 512 | 54 M | | flaubert-base-uncased | 12 | 12 | 768 | 137 M | | flaubert-base-cased | 12 | 12 | 768 | 138 M | | flaubert-large-cased | 24 | 16 | 1024 | 373 M |

Note: flaubert-small-cased is partially trained so performance is not guaranteed. Consider using it for debugging purpose only.

Using FlauBERT with Hugging Face's Transformers

import torch
from transformers import FlaubertModel, FlaubertTokenizer

# Choose among ['flaubert/flaubert_small_cased', 'flaubert/flaubert_base_uncased', 
#               'flaubert/flaubert_base_cased', 'flaubert/flaubert_large_cased']
modelname = 'flaubert/flaubert_base_cased' 

# Load pretrained model and tokenizer
flaubert, log = FlaubertModel.from_pretrained(modelname, output_loading_info=True)
flaubert_tokenizer = FlaubertTokenizer.from_pretrained(modelname, do_lowercase=False)
# do_lowercase=False if using cased models, True if using uncased ones

sentence = "Le chat mange une pomme."
token_ids = torch.tensor([flaubert_tokenizer.encode(sentence)])

last_layer = flaubert(toke

Your obligations as a flaubert small cased deployer

Track your progress

Sign up to track obligations, receive deadline reminders, and generate audit packages.

Tags

Languages

Library