polish roberta 8k EU AI Act Compliance Profile

A Polish language model built on the RoBERTa architecture, supporting context length of up to 8192 tokens. Encoder-type models can be fine-tuned to solve various text prediction tasks such as classification, regression, sequence tagging, or retrieval. In such tasks, they are usually faster and more parameter-efficient than LLMs. This model was initialized from the polish-roberta-large-v2 checkpoint, originally supporting 512-token context, and then adapted to handle longer texts. The model training process was as follows:

• In the first stage, the positional embedding layer was extended from 512 to 8192 tokens. The model was then trained on a corpus of approximately 150 billion tokens for one epoch. All model weights were frozen, and only the positional embedding layer was trained. The goal of this stage was to adapt the new layer without making drastic changes to the rest of the model's weights.
• In the second stage, all model weights were trained for 4 epochs. To improve training efficiency, we added support for Flash Attention 2 and contamination-free packing. Documents were packed into sequences of exactly 8192 tokens.
• Since the model's ability to solve both short and long prediction tasks changed during training, the final model is a merge of three checkpoints from different training stages — specifically from epochs 2, 3, and 4 — which ensured a balanced performance across different types of tasks.

Evaluation - Fine-Tuning

The main advantage of the new model lies in its ability to process longer texts than previously existing Polish encoders. For short classification tasks, the model maintains performance comparable to polish-roberta-large-v2. We analyzed the capabilities of our model by performing fine-tuning on 25 different tasks in Polish. For comparison, we also included the two most popula

Your obligations as a polish roberta 8k deployer

Track your progress

Sign up to track obligations, receive deadline reminders, and generate audit packages.

Tags

Languages