toxic comment model EU AI Act Compliance Profile

Model description

This model is a fine-tuned version of the DistilBERT model to classify toxic comments.

How to use

You can use the model with the following code.

from transformers import AutoModelForSequenceClassification, AutoTokenizer, TextClassificationPipeline

model_path = "martin-ha/toxic-comment-model"
tokenizer = AutoTokenizer.from_pretrained(model_path)
model = AutoModelForSequenceClassification.from_pretrained(model_path)

pipeline =  TextClassificationPipeline(model=model, tokenizer=tokenizer)
print(pipeline('This is a test text.'))

Limitations and Bias

This model is intended to use for classify toxic online classifications. However, one limitation of the model is that it performs poorly for some comments that mention a specific identity subgroup, like Muslim. The following table shows a evaluation score for different identity group. You can learn the specific meaning of this metrics here. But basically, those metrics shows how well a model performs for a specific group. The larger the number, the better.

| subgroup | subgroup_size | subgroup_auc | bpsn_auc | bnsp_auc | | ----------------------------- | ----------------- | ---------------- | ------------ | ------------ | | muslim | 108 | 0.689 | 0.811 | 0.88 | | jewish | 40 | 0.749 | 0.86 | 0.825 | | homosexual_gay_or_lesbian | 56 | 0.795 | 0.706 | 0.972 | | black | 84 | 0.866 | 0.758 | 0.975 | | white | 112 | 0.876 | 0.784 | 0.97 | | female | 306 |

Your obligations as a toxic comment model deployer

Track your progress

Sign up to track obligations, receive deadline reminders, and generate audit packages.

Tags

Languages