cde small v2 EU AI Act Compliance Profile

cde-small-v2

[!NOTE] Note on parameter count: Although HuggingFace reports the size of this model as 281M params, really it can be thought of as 140M. That's because our weights actually contain the weights of two models (dubbed "first stage" and "second stage"), and only the second-stage model is used to compute embeddings at search time.

<a href="https://github.com/jxmorris12/cde">Github</a>

Our new model that naturally integrates "context tokens" into the embedding process. As of January 13th, 2025, cde-small-v2 is the best small model (under 400M params) on the MTEB leaderboard for text embedding models, with an average score of 65.58.

👉 <b><a href="https://colab.research.google.com/drive/1r8xwbp7_ySL9lP-ve4XMJAHjidB9UkbL?usp=sharing">Try on Colab</a></b> <br> 👉 <b><a href="https://arxiv.org/abs/2410.02525">Contextual Document Embeddings (ArXiv)</a></b>

How to use cde-small-v2

Our embedding model needs to be used in two stages. The first stage is to gather some dataset information by embedding a subset of the corpus using our "first-stage" model. The second stage is to actually embed queries and documents, conditioning on the corpus information from the first stage. Note that we can do the first stage part offline and only use the second-stage weights at inference time.

With Transformers

Loading the model

Our model can be loaded using transformers out-of-the-box with "trust remote code" enabled. We use the default BERT uncased tokenizer:

import transformers

model = transformers.AutoModel.from_pretrained("jxm/cde-small-v2", trust_remote_code=True)
tokenizer = transformers.AutoTokenizer.from_pretrained("answerdotai/ModernBERT-base")

Note on prefixes

Nota bene: Like all state-of-the-art em

Your obligations as a cde small v2 deployer

Track your progress

Sign up to track obligations, receive deadline reminders, and generate audit packages.

Tags

Library

Base Model