colpali v1.2 EU AI Act Compliance Profile

ColPali: Visual Retriever based on PaliGemma-3B with ColBERT strategy

ColPali is a model based on a novel model architecture and training strategy based on Vision Language Models (VLMs) to efficiently index documents from their visual features. It is a PaliGemma-3B extension that generates ColBERT- style multi-vector representations of text and images. It was introduced in the paper ColPali: Efficient Document Retrieval with Vision Language Models and first released in this repository

Version specificity

This version is trained with colpali-engine==0.2.0 but can be loaded for any version >=0.2.0.

Compared to vidore/colpali, this version is trained with right padding for queries to fix unwanted tokens in the query encoding. It also stems from the fixed vidore/colpaligemma-3b-pt-448-base to guarantee deterministic projection layer initialization. It was trained for 5 epochs, with in-batch negatives and hard mined negatives and a warmup of 1000 steps (10x longer) to help reduce non-english language collapse.

Data is the same as the ColPali data described in the paper.

Model Description

This model is built iteratively starting from an off-the-shelf SigLIP model. We finetuned it to create BiSigLIP and fed the patch-embeddings output by SigLIP to an LLM, PaliGemma-3B to create BiPali.

One benefit of inputting image patch embeddings through a language model is that they are natively mapped to a latent space similar to

Your obligations as a colpali v1.2 deployer

Track your progress

Sign up to track obligations, receive deadline reminders, and generate audit packages.

Tags

Languages

Library

Base Model