SecureBERT EU AI Act Compliance Profile

SecureBERT: A Domain-Specific Language Model for Cybersecurity

SecureBERT is a RoBERTa-based, domain-specific language model trained on a large cybersecurity-focused corpus. It is designed to represent and understand cybersecurity text more effectively than general-purpose models.

SecureBERT was trained on extensive in-domain data crawled from diverse online resources. It has demonstrated strong performance in a range of cybersecurity NLP tasks.
👉 See the presentation on YouTube.
👉 Explore details on the GitHub repository.

Applications

SecureBERT can be used as a base model for downstream NLP tasks in cybersecurity, including:

• Text classification
• Named Entity Recognition (NER)
• Sequence-to-sequence tasks
• Question answering

Key Results

• Outperforms baseline models such as RoBERTa (base/large), SciBERT, and SecBERT in masked language modeling tasks within the cybersecurity domain.
• Maintains strong performance in general English language understanding, ensuring broad usability beyond domain-specific tasks.

Using SecureBERT

The model is available on Hugging Face.

Load the Model

from transformers import RobertaTokenizer, RobertaModel
import torch

tokenizer = RobertaTokenizer.from_pretrained("ehsanaghaei/SecureBERT")
model = RobertaModel.from_pretrained("ehsanaghaei/SecureBERT")

inputs = tokenizer("This is SecureBERT!", return_tensors="pt")
outputs = model(**inputs)

last_hidden_states = outputs.last_hidden_state


Masked Language Modeling Example

SecureBERT is trained with Masked Language Modeling (MLM). Use the following examp

Your obligations as a SecureBERT deployer

Track your progress

Sign up to track obligations, receive deadline reminders, and generate audit packages.

Tags

Languages

Library

Base Model