nepaliBERT EU AI Act Compliance Profile

NEPALI BERT

Masked Language Model for nepali language trained on nepali news scrapped from different nepali news website. The data set contained about 10 million of nepali sentences mainly related to nepali news.

This model is a fine-tuned version of Bert Base Uncased on dataset composed of different news scrapped from nepali news portals comprising of 4.6 GB of textual data. It achieves the following results on the evaluation set:

• Loss: 1.0495

Model description

Pretraining done on bert base architecture.

Intended uses & limitations

This transformer model can be used for any NLP tasks related to Devenagari language. At the time of training, this is the state of the art model developed for Devanagari dataset. Intrinsic evaluation with Perplexity of 8.56 achieves this state of the art whereas extrinsit evaluation done on sentiment analysis of Nepali tweets outperformed other existing masked language models on Nepali dataset.

Training and evaluation data

THe training corpus is developed using 85467 news scrapped from different job portals. This is a preliminary dataset for the experimentation. THe corpus size is about 4.3 GB of textual data. Similary evaluation data contains few news articles about 12 mb of textual data.

Training procedure

For the pretraining of masked language model, Trainer API from Huggingface is used. The pretraining took about 3 days 8 hours 57 minutes. Training was done on Tesla V100 GPU. With 640 Tensor Cores, Tesla V100 is the world's first GPU to break the 100 teraFLOPS (TFLOPS) barrier of deep learning performance. This GPU was faciliated by Kathmandu University (KU) supercomputer. Thanks to KU administration.

Usage

Your obligations as a nepaliBERT deployer

Track your progress

Sign up to track obligations, receive deadline reminders, and generate audit packages.