Crystal EU AI Act Compliance Profile

CrystalCoder

Crystal is a 7B parameter language model, distinctively trained on the SlimPajama and StarCoder datasets. This model excels in balancing natural language processing and coding capabilities. Despite being trained on a smaller dataset of 1.4 trillion tokens—compared to LLaMA 2's 2 trillion—Crystal surpasses LLaMA 2 in some challenging English and coding tasks. It demonstrates superior performance in benchmarks like MMLU, HumanEval, and MBPP. By comparing Crystal with other similar work, Crystal is quite balance on language and coding tasks. Crystal is part of LLM360's Pebble model series.

Note: Crystal was formerly known as CrystalCoder.

| Performance on Standard Benchmarks | |------------------------------------------------|

Notes

• We compute all evaluation metrics ourselves.

• Language benchmarks are computed following the convention of the Huggingface Leaderboard, which means AI2 Reasoning Challenge in 25-shot, HellaSwag in 10-shot, MMLU computed in 5-shot, TruthfulQA in 0-shot.

• As reported in prior work, the choice of temperature affect the programming metrics a lot, we evaluate all models with the following temperature:

  • Scores for HumanEval is computed with a temperature of 0.2
  • Scores for MBPP is computed with a temperature of 0.1

• For detailed token breakdown of Crystal dataset, refer to the Crystal dataset repository.

About LLM360

LLM360 is an initiative for comprehensive and fully open-sourced LLMs, where all training details, model checkpoints, intermediate results, and additional analyses are mad

Your obligations as a Crystal deployer

Track your progress

Sign up to track obligations, receive deadline reminders, and generate audit packages.

Tags

Languages

Library