Clinical ModernBERT EU AI Act Compliance Profile

Clinical ModernBERT

Clinical ModernBERT is a state-of-the-art encoder-based transformer tailored specifically for biomedical and clinical text handling context length up to 8192 tokens. Building on the innovations introduced by ModernBERT, this model extends the context window to 8,192 tokens and incorporates domain-specific vocabulary refinements. It is designed to produce semantically rich representations that capture both the nuanced syntax of biomedical literature and the intricate semantics of clinical narratives.

Usage

Pretrained model weights and tokenizer artifacts are provided to facilitate easy integration with your downstream biomedical NLP tasks:

from transformers import AutoModel, AutoTokenizer

model = AutoModel.from_pretrained('Simonlee711/Clinical_ModernBERT')
tokenizer = AutoTokenizer.from_pretrained('Simonlee711/Clinical_ModernBERT')

Model Overview

Below is a table summarizing ModernBERT's key architectural components and their benefits:

| Feature | Description | Benefit | |------------------------------|-----------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------| | Extended Context Length | Processes sequences up to 8,192 tokens. | Captures long-range dependencies and full document contexts, essential for complex linguistic tasks. | | GeGLU Activation | Uses the GeGLU activation, a gated variant of GeLU. | Enhances non-linear representation and model stability by allowing controlled inform

Your obligations as a Clinical ModernBERT deployer

Track your progress

Sign up to track obligations, receive deadline reminders, and generate audit packages.

Tags

Languages

Library

Base Model