BioClinical ModernBERT large EU AI Act Compliance Profile

BioClinical ModernBERT

BioClinical ModernBERT is available in two sizes: base (150M parameters) and large (396M parameters). The model training checkpoints can be found here, and our code is available in our GitHub repository.

A step-by-step guide for reproducing our model training with your own data is available here!

Table of Contents

1. Model Summary
2. Usage
3. Training
4. Evaluation
5. License
6. Citation

Model Summary

BioClinical ModernBERT is a domain-adapted encoder that builds on ModernBERT base and large, incorporating long-context processing and substantial improvements in speed and performance for biomedical and clinical NLP. BioClinical ModernBERT is trained on the largest biomedical and clinical corpus to date, with over 53.5 billion tokens, and addresses a key limitation of prior clinical encoders by leveraging 20 datasets from diverse institutions, domains, and geographic regions, rather than relying on data from a single source.

Usage

You can use these models directly with the transformers library starting from v4.48.0:

pip install -U transformers>=4.48.0

Since BioClinical ModernBERT is a Masked Language Model (MLM), you can use the fill-mask pipeline or load it via AutoModelForMaskedLM. To use BioClinical ModernBERT for downstream tasks like classification, retrieval, or QA, fine-tune it following standard BERT fine-tuning recipes.

**⚠️ If your GPU supports it, we recommend us

Your obligations as a BioClinical ModernBERT large deployer

Track your progress

Sign up to track obligations, receive deadline reminders, and generate audit packages.

Tags

Languages

Library

Base Model