IndicBERTv2 MLM only EU AI Act Compliance Profile

IndicBERT

A multilingual language model trained on IndicCorp v2 and evaluated on IndicXTREME benchmark. The model has 278M parameters and is available in 23 Indic languages and English. The models are trained with various objectives and datasets. The list of models are as follows:

• IndicBERT-MLM [Model] - A vanilla BERT style model trained on IndicCorp v2 with the MLM objective
  • +Samanantar [Model] - TLM as an additional objective with Samanantar Parallel Corpus [Paper] | [Dataset]
  • +Back-Translation [Model] - TLM as an additional objective by translating the Indic parts of IndicCorp v2 dataset into English w/ IndicTrans model [Model]
• IndicBERT-SS [Model] - To encourage better lexical sharing among languages we convert the scripts from Indic languages to Devanagari and train a BERT style model with the MLM objective

Run Fine-tuning

Fine-tuning scripts are based on transformers library. Create a new conda environment and set it up as follows:

conda create -n finetuning python=3.9
pip install -r requirements.txt

All the tasks follow the same structure, please check individual files for detailed hyper-parameter choices. The following command runs the fine-tuning for a task:

python IndicBERT/fine-tuning/$TASK_NAME/$TASK_NAME.py \
    --model_name_or_path=$MODEL_NAME \
    --do_train

Arguments:

• MODEL_NAME: name of the model to fine-tune, can be a local path or a model from the HuggingFace Model Hub
• TASK_NAME: one of [ner, paraphrase, qa, sentiment, xcopa, xnli, flores]

For MASSIVE task, please use the inst

Your obligations as a IndicBERTv2 MLM only deployer

Track your progress

Sign up to track obligations, receive deadline reminders, and generate audit packages.

Tags

Languages

Library