BiomedVLP BioViL T EU AI Act Compliance Profile

BioViL-T

BioViL-T is a domain-specific vision-language model designed to analyze chest X-rays (CXRs) and radiology reports. It was trained using a temporal multi-modal pre-training procedure, which distinguishes it from its predecessor model (BioViL). In detail, BioViL-T takes advantage of the temporal structure between data points, resulting in improved downstream performance on multiple benchmarks, while using the same training dataset as its predecessor. In particular, the resultant model displays significant improvement in embedding temporal information present in the image and text modalities (see results), as well as in the joint space. The canonical model can be adapted to both single- and multi-image downstream applications including: natural language inference, phrase-grounding, image/text classification, and language decoding.

The corresponding BERT language model is trained in two stages: First, we pretrain CXR-BERT-general from a randomly initialized BERT model via Masked Language Modeling (MLM) on PubMed abstracts and clinical notes from the publicly-available MIMIC-III and MIMIC-CXR. The general model can be fine-tuned for research in other clinical domains by adjusting the parameters specific to the target domain. In the second stage, BioViL-T is continually pretrained from CXR-BERT-general using a multi-modal pre-training procedure by utilising radiology reports and sequences of chest X-rays. We utilise the latent representation of [CLS] token to align text and image embeddings.

Language model variations

| Model | Model identifier on HuggingFace

Your obligations as a BiomedVLP BioViL T deployer

Track your progress

Sign up to track obligations, receive deadline reminders, and generate audit packages.

Tags

Languages

Library