mini glm moe EU AI Act Compliance Profile

Mini GLM-4 MoE (0.5B)

A small GLM-4 MoE model (543M parameters) for testing and development. Uses the same Glm4MoeForCausalLM architecture as the full GLM-4-100B-A10B but with reduced dimensions.

This model is designed for testing MoE training pipelines in prime-rl without needing large pretrained checkpoints. It is small enough to run on a single GPU while exercising the same code paths as production models.

Architecture

| Parameter | Value | |---|---| | Parameters | 543M | | Hidden size | 1024 | | Layers | 24 | | Attention heads | 16 (4 KV heads) | | Routed experts | 8 | | Experts per token | 4 | | Shared experts | 1 | | MoE intermediate size | 256 | | Dense intermediate size | 2048 | | Dense layers (first-k) | 1 | | Vocab size | 151,552 | | Partial rotary factor | 0.5 | | Model type | glm4_moe |

The architecture mirrors THUDM/GLM-4-100B-A10B: the first layer is a dense MLP, and all subsequent layers use Mixture-of-Experts with a shared expert. Attention uses Grouped Query Attention (GQA) with partial rotary embeddings.

How this model was created

Step 1: Random initialization. A Glm4MoeConfig was instantiated with the small dimensions above and the HuggingFace Glm4MoeForCausalLM model was initialized with random weights. The tokenizer was copied from THUDM/GLM-4-9B-0414.

Step 2: Roundtrip verification. Before training, we verified that the HuggingFace and prime-rl custom implementations produce identical outputs on the same weights (max logits diff < 0.01), and that convert_to_hf / convert_to_prime state dict conversions are lossless.

Step 3: SFT warmup. The model was fine-tuned for 200 steps on [PrimeIntellect/Reverse-Text-SFT](https://huggingface.co/datasets/PrimeIntellect/Reverse-Text-

Your obligations as a mini glm moe deployer

Track your progress

Sign up to track obligations, receive deadline reminders, and generate audit packages.

Tags