DrBERT 7GB EU AI Act Compliance Profile

DrBERT: A Robust Pre-trained Model in French for Biomedical and Clinical domains

In recent years, pre-trained language models (PLMs) achieve the best performance on a wide range of natural language processing (NLP) tasks. While the first models were trained on general domain data, specialized ones have emerged to more effectively treat specific domains. In this paper, we propose an original study of PLMs in the medical domain on French language. We compare, for the first time, the performance of PLMs trained on both public data from the web and private data from healthcare establishments. We also evaluate different learning strategies on a set of biomedical tasks. Finally, we release the first specialized PLMs for the biomedical field in French, called DrBERT, as well as the largest corpus of medical data under free license on which these models are trained.

1. DrBERT models

DrBERT is a French RoBERTa trained on a open source corpus of French medical crawled textual data called NACHOS. Models with different amount of data from differents public and private sources are trained using the CNRS (French National Centre for Scientific Research) Jean Zay French supercomputer. Only the weights of the models trained using exclusively open-sources data are publicly released to prevent any personnal information leak and to follow the european GDPR laws :

| Model name | Corpus | Number of layers | Attention Heads | Embedding Dimension | Sequence Length | Model URL | | :------: | :---: | :---: | :---: | :---: | :---: | :---: | | DrBERT-7-GB-cased-Large | NACHOS 7 GB | 24 | 16 | 1024 | 512 | HuggingFace | | DrBERT-7-GB-cased | NACHOS 7 GB | 12 | 12 | 768 | 512 | HuggingFace | | `DrBERT-4-G

Your obligations as a DrBERT 7GB deployer

Track your progress

Sign up to track obligations, receive deadline reminders, and generate audit packages.

Tags

Languages

Library