modernbert ja 310m EU AI Act Compliance Profile

ModernBERT-Ja-310M

This repository provides Japanese ModernBERT trained by SB Intuitions.

ModernBERT is a new variant of the BERT model that combines local and global attention, allowing it to handle long sequences while maintaining high computational efficiency. It also incorporates modern architectural improvements, such as RoPE.

Our ModernBERT-Ja-310M is trained on a high-quality corpus of Japanese and English text comprising 4.09T tokens, featuring a vocabulary size of 102,400 and a sequence length of 8,192 tokens.

How to Use

You can use our models directly with the transformers library v4.48.0 or higher:

pip install -U "transformers>=4.48.0"

Additionally, if your GPUs support Flash Attention 2, we recommend using our models with Flash Attention 2.

Example Usage

import torch
from transformers import AutoModelForMaskedLM, AutoTokenizer, pipeline

model = AutoModelForMaskedLM.from_pretrained("sbintuitions/modernbert-ja-310m", torch_dtype=torch.bfloat16)
tokenizer = AutoTokenizer.from_pretrained("sbintuitions/modernbert-ja-310m")
fill_mask = pipeline("fill-mask", model=model, tokenizer=tokenizer)

results = fill_mask("おはようございます、今日の天気は<mask>です。")

for result in results:
    print(result)
# {'score': 0.54296875, 'token': 16416, 'token_str': '晴れ', 'sequence': 'おはようございます、今日の天気は晴れです。'}
# {'score': 0.17578125, 'token': 28933, 'token_str': '曇り', 'sequence': 'おはようございます、今日の天気は曇りです。'}
# {'score': 0.13671875, 'token': 92339, 'token_str': 'くもり', 'sequence': 'おはようございます、今日の天気はくもりです。'}
# {'score': 0.06494140625, 'token': 2988, 'token_str': '雨', 'sequence': 'おはようございます、今日の天気は雨です。'}
# {'score': 0.019775390625, 'token': 10547, 'token_str': '晴', 'sequence': 'おはようございます、今日の天気は晴です。'}

Model Series

We provide ModernBERT-Ja in several model sizes. Below is a summary of each m

Your obligations as a modernbert ja 310m deployer

Track your progress

Sign up to track obligations, receive deadline reminders, and generate audit packages.

Tags

Languages

Library