Mistral Nemo Base 2407 EU AI Act Compliance Profile

Model Card for Mistral-Nemo-Base-2407

The Mistral-Nemo-Base-2407 Large Language Model (LLM) is a pretrained generative text model of 12B parameters trained jointly by Mistral AI and NVIDIA, it significantly outperforms existing models smaller or similar in size.

For more details about this model please refer to our release blog post.

Key features

• Released under the Apache 2 License
• Pre-trained and instructed versions
• Trained with a 128k context window
• Trained on a large proportion of multilingual and code data
• Drop-in replacement of Mistral 7B

Model Architecture

Mistral Nemo is a transformer model, with the following architecture choices:

• Layers: 40
• Dim: 5,120
• Head dim: 128
• Hidden dim: 14,436
• Activation Function: SwiGLU
• Number of heads: 32
• Number of kv-heads: 8 (GQA)
• Vocabulary size: 2**17 ~= 128k
• Rotary embeddings (theta = 1M)

Metrics

Main Benchmarks

| Benchmark | Score | | --- | --- | | HellaSwag (0-shot) | 83.5% | | Winogrande (0-shot) | 76.8% | | OpenBookQA (0-shot) | 60.6% | | CommonSenseQA (0-shot) | 70.4% | | TruthfulQA (0-shot) | 50.3% | | MMLU (5-shot) | 68.0% | | TriviaQA (5-shot) | 73.8% | | NaturalQuestions (5-shot) | 31.2% |

Multilingual Benchmarks (MMLU)

| Language | Score | | --- | --- | | French | 62.3% | | German | 62.7% | | Spanish | 64.6% | | Italian | 61.3% | | Portuguese | 63.3% | | Russian | 59.2% | | Chinese | 59.0% | | Japanese | 59.0% |

Usage

The model can be used with three different frameworks

• mistral_inference: See here
• transformers: See here
• NeMo: See nvidia/Mistral-NeMo-12B-Base

Mistral Inference

Install

It is recommended to use `mistralai/M

Your obligations as a Mistral Nemo Base 2407 deployer

Track your progress

Sign up to track obligations, receive deadline reminders, and generate audit packages.

Tags

Languages

Library