muril large cased EU AI Act Compliance Profile

MuRIL Large

Multilingual Representations for Indian Languages : A BERT Large (24L) model pre-trained on 17 Indian languages, and their transliterated counterparts.

Overview

This model uses a BERT large architecture [1] pretrained from scratch using the Wikipedia [2], Common Crawl [3], PMINDIA [4] and Dakshina [5] corpora for 17 [6] Indian languages.

We use a training paradigm similar to multilingual bert, with a few modifications as listed:

• We include translation and transliteration segment pairs in training as well.
• We keep an exponent value of 0.3 and not 0.7 for upsampling, shown to enhance low-resource performance. [7]

See the Training section for more details.

Training

The MuRIL model is pre-trained on monolingual segments as well as parallel segments as detailed below :

• Monolingual Data : We make use of publicly available corpora from Wikipedia and Common Crawl for 17 Indian languages.
• Parallel Data : We have two types of parallel data :
  • Translated Data : We obtain translations of the above monolingual corpora using the Google NMT pipeline. We feed translated segment pairs as input. We also make use of the publicly available PMINDIA corpus.
  • Transliterated Data : We obtain transliterations of Wikipedia using the IndicTrans [8] library. We feed transliterated segment pairs as input. We also make use of the publicly available Dakshina dataset.

We keep an exponent value of 0.3 to calculate duplication multiplier values for upsampling of lower resourced languages and set dupe factors accordingly. Note, we limit transliterated pairs to Wikipedia only.

The model was trained using a self-supervised masked language modeling task. We do whole word masking with a maximum of 80 predictions. The model was trained for 1500K steps, with a batch size of 8192, and a max sequence length of 512.

Trainable parameters

All parameters in the module are trainable, and fine-tuning all p

Your obligations as a muril large cased deployer

Track your progress

Sign up to track obligations, receive deadline reminders, and generate audit packages.

Tags

Library