bert hash nano embeddings EU AI Act Compliance Profile

BERT Hash Nano Embeddings

This is a BERT Hash Nano model fined-tuned using sentence-transformers. It maps sentences & paragraphs to a 128-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

This model is an alternative to MUVERA fixed-dimensional encoding with ColBERT models. MUVERA encoding enables encoding the multi-vector outputs of ColBERT into single dense vector outputs. While this is a great step, the main issue with MUVERA is that it tends to need wide vectors to be effective (5K - 10K dimensional vectors). bert-hash-nano-embeddings outputs 128-dimensional vectors.

The training dataset is a subset of this embedding training collection. The training workflow was a two step distillation process as follows.

• Distill embeddings from the larger mixedbread-ai/mxbai-embed-xsmall-v1 model using this model distillation script from Sentence Transformers.
• Build a distilled dataset of teacher scores using the mixedbread-ai/mxbai-rerank-xsmall-v1 cross-encoder for a random sample of the training dataset mentioned above.
• Further fine-tune the model on the distilled dataset using KLDivLoss.

Usage (txtai)

This model can be used to build embeddings databases with txtai for semantic search and/or as a knowledge source for retrieval augmented generation (RAG).

impo

Your obligations as a bert hash nano embeddings deployer

Track your progress

Sign up to track obligations, receive deadline reminders, and generate audit packages.

Tags

Languages

Library

Base Model