ViDense EU AI Act Compliance Profile

Table of contents

• Introduce
• Usage
• Performance
• Contact
• Support The Project
• Citation

Introduce

ViDense is a VietNamese Embedding Model. Fine-tuned and enhanced with tailored methods, ViDense incorporates advanced techniques to optimize performance for text embeddings in various applications.

Model Configuration and Methods:

• Base Model: FacebookAI/xlm-roberta-large
• Trained for 10 epochs with a train batch size of 2048.
• Utilizes a 3-phase training approach, where the best checkpoint from each phase serves as the base model for the next.
• Position Encoding: Rotary Position Encoding
• Attention: Blockwise Parallel Transformer
• Pooling: Mean Pooling
• Momentum Encoder: Incorporates MoCo (Momentum Contrast) to enhance in-batch negative sampling.
• Rank Encoder: Introduces a Rank Encoder to account for transitive positive relationships. By considering positives of positives as relevant to the anchor, it reranks the corpus using the Spearman metric and integrates Spearman weights into the loss calculation for improved ranking.
• Loss Function: Cross Entropy Loss

Usage

import torch
from transformers import AutoModel, AutoTokenizer


def avg_pooling(attention_mask, outputs):
    last_hidden = outputs.last_hidden_state
    return (last_hidden * attention_mask.unsqueeze(-1)).sum(1) / attention_mask.sum(-1).unsqueeze(-1)


tokenizer = AutoTokenizer.from_pretrained('namdp-ptit/ViDense')
model = AutoModel.from_pretrained('namdp-ptit/ViDense')

sentences = [
    'Tỉnh nào có diện tích lớn nhất Việt Nam',
    'Tỉnh nào có diện tích nhỏ nhất Việt Nam',
    'Tỉnh nào có diện tích rộng nhất Việt Nam'
]

inputs = tokenizer(sentences, return_tensors='pt', padding=True)

with torch.no_grad():
    outputs = mode

Your obligations as a ViDense deployer

Track your progress

Sign up to track obligations, receive deadline reminders, and generate audit packages.

Tags

Languages

Library

Base Model