RuModernBERT base EU AI Act Compliance Profile

RuModernBERT-base

The Russian version of the modernized bidirectional encoder-only Transformer model, ModernBERT. RuModernBERT was pre-trained on approximately 2 trillion tokens of Russian, English, and code data with a context length of up to 8,192 tokens, using data from the internet, books, scientific sources, and social media.

| | Model Size | Hidden Dim | Num Layers | Vocab Size | Context Length | Task | |------------------------------------------------------------------------------:|:----------:|:----------:|:----------:|:----------:|:--------------:|:---------:| | deepvk/RuModernBERT-small | 35M | 384 | 12 | 50368 | 8192 | Masked LM | | deepvk/RuModernBERT-base [this] | 150M | 768 | 22 | 50368 | 8192 | Masked LM |

Notice ⚠️

The patched tokenizer is provided under the patched-tokenizer revision.

We observed that several Russian lowercase letters were split into multiple subword tokens. This can be problematic for tasks like Named Entity Recognition (NER), where it is important that the first token of a word is a semantically meaningful unit.

To address this, we release a patched revision of the tokenizer with minimal but targeted change. Six common Russian lowercase letters (а, е, и, н, о, т) are now encoded as single tokens. These tokens were assigned to [unusedX] slots in the vocabulary. Corresponding BPE merges were added to ensure proper single-token encoding during inference. To preserve compatibility with the pretrained model each new token was initialized with the embedding of its uppercase counterpart both in tok_embedding and lm_head. To preven

Your obligations as a RuModernBERT base deployer

Track your progress

Sign up to track obligations, receive deadline reminders, and generate audit packages.

Tags

Languages

Library