squeezebert uncased EU AI Act Compliance Profile

language: en license: bsd datasets:

• bookcorpus
• wikipedia

SqueezeBERT pretrained model

This model, squeezebert-uncased, is a pretrained model for the English language using a masked language modeling (MLM) and Sentence Order Prediction (SOP) objective. SqueezeBERT was introduced in this paper. This model is case-insensitive. The model architecture is similar to BERT-base, but with the pointwise fully-connected layers replaced with grouped convolutions. The authors found that SqueezeBERT is 4.3x faster than bert-base-uncased on a Google Pixel 3 smartphone.

Pretraining

Pretraining data

• BookCorpus, a dataset consisting of thousands of unpublished books
• English Wikipedia

Pretraining procedure

The model is pretrained using the Masked Language Model (MLM) and Sentence Order Prediction (SOP) tasks. (Author's note: If you decide to pretrain your own model, and you prefer to train with MLM only, that should work too.)

From the SqueezeBERT paper:

We pretrain SqueezeBERT from scratch (without distillation) using the LAMB optimizer, and we employ the hyperparameters recommended by the LAMB authors: a global batch size of 8192, a learning rate of 2.5e-3, and a warmup proportion of 0.28. Following the LAMB paper's recommendations, we pretrain for 56k steps with a maximum sequence length of 128 and then for 6k steps with a maximum sequence length of 512.

Finetuning

The SqueezeBERT paper results from 2 approaches to finetuning the model:

• "finetuning without bells and whistles" -- after pretraining the SqueezeBERT model, finetune it on each GLUE task
• "finetuning with bells and whistles" -- after pretraining the SqueezeBERT model, finetune it on a MNLI with distillation from a teacher model. Then, use the MNLI-finetuned SqueezeBERT mode

Your obligations as a squeezebert uncased deployer

Track your progress

Sign up to track obligations, receive deadline reminders, and generate audit packages.

Tags

Library