BiomedVLP CXR BERT specialized EU AI Act Compliance Profile

CXR-BERT-specialized

CXR-BERT is a chest X-ray (CXR) domain-specific language model that makes use of an improved vocabulary, novel pretraining procedure, weight regularization, and text augmentations. The resulting model demonstrates improved performance on radiology natural language inference, radiology masked language model token prediction, and downstream vision-language processing tasks such as zero-shot phrase grounding and image classification.

First, we pretrain CXR-BERT-general from a randomly initialized BERT model via Masked Language Modeling (MLM) on abstracts PubMed and clinical notes from the publicly-available MIMIC-III and MIMIC-CXR. In that regard, the general model is expected be applicable for research in clinical domains other than the chest radiology through domain specific fine-tuning.

CXR-BERT-specialized is continually pretrained from CXR-BERT-general to further specialize in the chest X-ray domain. At the final stage, CXR-BERT is trained in a multi-modal contrastive learning framework, similar to the CLIP framework. The latent representation of [CLS] token is utilized to align text/image embeddings.

Model variations

| Model | Model identifier on HuggingFace | Vocabulary | Note | | ------------------------------------------------- | ----------------------------------------------------------------------------------------------------------- | -------------- | --------------------------------------------------------- | | CXR-BERT-general | [microsoft/BiomedVLP-CXR-

Your obligations as a BiomedVLP CXR BERT specialized deployer

Track your progress

Sign up to track obligations, receive deadline reminders, and generate audit packages.

Tags

Languages

Library