opensearch neural sparse encoding multilingual v1 EU AI Act Compliance Profile

opensearch-neural-sparse-encoding-multilingual-v1

Select the model

The model should be selected considering search relevance, model inference and retrieval efficiency(FLOPS). We benchmark models' performance on MIRACL benchmark (we exclude th since the uncased backbone can not encode it). We recommend to use it with max_ratio pruning.

| Model | Inference-free for Retrieval | Model Parameters | AVG NDCG@10 | AVG FLOPS | AVG EMB SIZE | |-------|------------------------------|------------------|-------------|-----------| -----------| | opensearch-neural-sparse-encoding-multilingual-v1 | ✔️ | 160M | 0.629 | 1.3 | 138 | | opensearch-neural-sparse-encoding-multilingual-v1; prune_ratio 0.1 | ✔️ | 160M | 0.626 | 0.8 | 75 |

Overview

• Paper: Towards Competitive Search Relevance For Inference-Free Learned Sparse Retrievers
• Fine-tuning sample: opensearch-sparse-model-tuning-sample

This is a learned sparse retrieval model. It encodes the documents to 105879 dimensional sparse vectors. For queries, it just use a tokenizer and a weight look-up table to generate sparse vectors. The non-zero dimension index means the corresponding token in the vocabulary, and the weight means the importance of the token. And the similarity score is the inner product of query/document sparse vectors.

OpenSearch neural sparse feature supports learned sparse retrieval with lucene inverted index. Link: https://opensearch.org/docs/latest/query-dsl/specialized/neural-sparse/. The indexing and search can be performed with OpenSearch high-level API.

Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers
`

Your obligations as a opensearch neural sparse encoding multilingual v1 deployer

Track your progress

Sign up to track obligations, receive deadline reminders, and generate audit packages.

Tags

Languages

Library