t5 v1.1 base dutch cased EU AI Act Compliance Profile

t5-v1.1-base-dutch-cased

A T5 sequence to sequence model pre-trained from scratch on cleaned Dutch 🇳🇱🇧🇪 mC4.

This t5-v1.1 model has 247M parameters. It was pre-trained with masked language modeling (denoise token span corruption) objective on the dataset mc4_nl_cleaned config full for 2 epoch(s) and a duration of 6d6h, with a sequence length of 1024, batch size 64 and 1210154 total steps (79B tokens). Pre-training evaluation loss and accuracy are 0,96 and 0,78. Refer to the evaluation section below for a comparison of the pre-trained models on summarization and translation.

• Pre-trained T5 models need to be finetuned before they can be used for downstream tasks, therefore the inference widget on the right has been turned off.
• For a demo of the Dutch CNN summarization models, head over to the Hugging Face Spaces for the Netherformer 📰 example application!

Please refer to the original T5 papers and Scale Efficiently papers for more information about the T5 architecture and configs, though it must be noted that this model (t5-v1.1-base-dutch-cased) is unrelated to these projects and not an 'official' checkpoint.

• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer by Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J. Liu.
• Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers by Yi Tay, Mostafa Dehghani, Jinfeng Rao, William Fedus, Samira Abnar, Hyung Won Chung, Sharan Narang, Dani Yogatama, Ashish Vaswani, Donald Metzler.

Tokenizer

The model uses a cased SentencePiece tokenizer configured with the `Nmt

Your obligations as a t5 v1.1 base dutch cased deployer

Track your progress

Sign up to track obligations, receive deadline reminders, and generate audit packages.

Tags

Languages

Library