ModernBERT-large EU AI Act Compliance Profile

ModernBERT

Table of Contents

1. Model Summary
2. Usage
3. Evaluation
4. Limitations
5. Training
6. License
7. Citation

Model Summary

ModernBERT is a modernized bidirectional encoder-only Transformer model (BERT-style) pre-trained on 2 trillion tokens of English and code data with a native context length of up to 8,192 tokens. ModernBERT leverages recent architectural improvements such as:

• Rotary Positional Embeddings (RoPE) for long-context support.
• Local-Global Alternating Attention for efficiency on long inputs.
• Unpadding and Flash Attention for efficient inference.

ModernBERT’s native long context length makes it ideal for tasks that require processing long documents, such as retrieval, classification, and semantic search within large corpora. The model was trained on a large corpus of text and code, making it suitable for a wide range of downstream tasks, including code retrieval and hybrid (text + code) semantic search.

It is available in the following sizes:

• ModernBERT-base - 22 layers, 149 million parameters
• ModernBERT-large - 28 layers, 395 million parameters

For more information about ModernBERT, we recommend our release blog post for a high-level overview, and our arXiv pre-print for in-depth information.

ModernBERT is a collaboration between Answer.AI, LightOn, and friends.

Usage

You can use these models directly with the transformers library starting from v4.48.0:

pip install -U transformers>=4.48.0

Since ModernBERT is a Masked Language Model (MLM), you can use the fill-mask pipeline or load it via AutoModelForMaskedLM. To use ModernBERT for downstream tasks like c

Your obligations as a ModernBERT-large deployer

Track your progress

Sign up to track obligations, receive deadline reminders, and generate audit packages.

Tags

Languages

Library