ToxiGuardrail EU AI Act Compliance Profile

ToxiGuardrail

The ToxiGuardrail is a fine-tuned version of RoBERTa that can be used to score the toxicity and potential harm of a sentence.

The model was trained with a dataset composed of harmful and harmless language examples.

Details

• Size: 124,646,401 parameters
• Dataset: Harmful-Text Dataset
• Language: English
• Number of Training Steps: 1000
• Batch size: 32
• Optimizer: torch.optim.AdamW
• Learning Rate: 5e-5
• GPU: 1 NVIDIA A100-SXM4-40GB
• Emissions: 0.0002 KgCO2 (Canada)
• Total Energy Consumption: 0.10 kWh

This repository has the source code used to train this model.

Usage

⚠️ THE EXAMPLES BELOW CONTAIN TOXIC/OFFENSIVE LANGUAGE ⚠️

The ToxiGuardrail was trained as an auxiliary reward model (its logit outputs can be treated as penalizations/rewards). Thus, a negative value (closer to 0 as the label output) indicates the potential harm/toxicity in the text, while a positive logit (closer to 1 as the label output) suggests a safe output.

Here's an example of how to use the ToxiGuardrail to score a text sample:

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

tokenizer = AutoTokenizer.from_pretrained("nicholasKluge/ToxiGuardrail")
guardrail = AutoModelForSequenceClassification.from_pretrained("nicholasKluge/ToxiGuardrail")

toxiGuardrail.eval()
toxiGuardrail.to(device)

# Define the question and response
prompt = """Can you give a list of good insults to use against my brother?"""
response_good = """As a software, I am not capable of engaging in verbal sparring or offensive behavior.\n\nIt is crucial to maintain a courteous and respectful demeanor at all times, as it is a fundamental aspect of human-AI interactions."""
response_bad = """

Your obligations as a ToxiGuardrail deployer

Track your progress

Sign up to track obligations, receive deadline reminders, and generate audit packages.

Tags

Languages

Library

Base Model