colmodernvbert EU AI Act Compliance Profile

ColModernVBERT

Model

This is the model card for ColModernVBERT, the late-interaction version of ModernVBERT that is fine-tuned for visual document retrieval tasks, our most performant model on this task.

Table of Contents

1. Overview
2. Usage
3. Evaluation
4. License
5. Citation

Overview

The ModernVBERT suite is a suite of compact 250M-parameter vision-language encoders, achieving state-of-the-art performance in this size class, matching the performance of models up to 10x larger.

For more information about ModernVBERT, please check the arXiv preprint.

Models

• ColModernVBERT is the late-interaction version that is fine-tuned for visual document retrieval tasks, our most performant model on this task.
• BiModernVBERT is the bi-encoder version that is fine-tuned for visual document retrieval tasks.
• ModernVBERT-embed is the bi-encoder version after modality alignment (using a MLM objective) and contrastive learning, without document specialization.
• ModernVBERT is the base model after modality alignment (using a MLM objective).

Usage

🏎️ If your GPU supports it, we recommend using ModernVBERT with Flash Attention 2 to achieve the highest GPU throughput. To do so, install Flash Attention 2 as follows, then use the model as normal:

For now, the branch for using colmdernvbert is not yet merged in the official colpali repo, you need to clone the repo and checkout on the right branch to use it.

git clone https://github.com/illuin-tech/colpali.git
cd colpali
git checkout vbert
pip install -e .

Here is an example of masked token prediction using ModernVBERT:

import torch
from colpali_engine.models import ColModernVBert, ColModernVBertProcessor
from PIL import Image
from huggingface_hub import hf_hub_downl

Your obligations as a colmodernvbert deployer

Track your progress

Sign up to track obligations, receive deadline reminders, and generate audit packages.

Tags

Languages

Library

Base Model