ibert roberta base EU AI Act Compliance Profile

I-BERT base model

This model, ibert-roberta-base, is an integer-only quantized version of RoBERTa, and was introduced in this paper. I-BERT stores all parameters with INT8 representation, and carries out the entire inference using integer-only arithmetic. In particular, I-BERT replaces all floating point operations in the Transformer architectures (e.g., MatMul, GELU, Softmax, and LayerNorm) with closely approximating integer operations. This can result in upto 4x inference speed up as compared to floating point counterpart when tested on an Nvidia T4 GPU. The best model parameters searched via quantization-aware finetuning can be then exported (e.g., to TensorRT) for integer-only deployment of the model.

Finetuning Procedure

Finetuning of I-BERT consists of 3 stages: (1) Full-precision finetuning from the pretrained model on a down-stream task, (2) model quantization, and (3) integer-only finetuning (i.e., quantization-aware training) of the quantized model.

Full-precision finetuning

Full-precision finetuning of I-BERT is similar to RoBERTa finetuning. For instance, you can run the following command to finetune on the MRPC text classification task.

Model Quantization

Once you are done with full-precision finetuning, open up config.json in your checkpoint directory and set the quantize attribute as true.

Your obligations as a ibert roberta base deployer

Track your progress

Sign up to track obligations, receive deadline reminders, and generate audit packages.

Tags

Library