bge m3 onnx EU AI Act Compliance Profile

BGE-M3 converted to ONNX weights with HF Optimum, to be compatible, for example, with ONNX Runtime.

This ONNX model outputs dense, sparse and ColBERT embedding representations all at once. The output is a list of numpy arrays in previously mentioned order of representations.

Note: dense and ColBERT embeddings are normalized like the default behavior in the original FlagEmbedding library, if you want unnormalized outputs you can modify the code in bgem3_model.py and re-run the ONNX export with export_onnx.py script.

This ONNX model also has "O2" level graph optimizations applied, you can read more about optimization levels here. If you want ONNX model with different optimization or without optimizations, you can re-run the ONNX export script export_onnx.py with appropriate optimization argument.

Usage with ONNX Runtime (Python)

If you haven't already, you can install the ONNX Runtime Python library with pip:

pip install onnxruntime==1.17.0

For tokenization, you can for example use HF Transformers by installing it with pip:

pip install transformers==4.37.2

Clone this repository with Git LFS to get the ONNX model files.

You can then use the model to compute embeddings, as follows:

import onnxruntime as ort
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("BAAI/bge-m3")
ort_session = ort.InferenceSession("model.onnx")

inputs = tokenizer("BGE M3 is an embedding model supporting dense retrieval, lexical matching and multi-vector interaction.", padding="longest", return_tensors="np")
inputs_onnx = {k: ort.OrtValue.ortvalue_from_numpy(v) for k, v in inputs.items()}

outputs = ort_session.run(None, inputs_onnx)

Note: you can use following sparse token weight processor from FlagEmbedding to get same the output for the

Your obligations as a bge m3 onnx deployer

Track your progress

Sign up to track obligations, receive deadline reminders, and generate audit packages.

Tags

Library