stablelm 2 1 6b chat imatrix GGUF EU AI Act Compliance Profile

This is StableLM 2 Chat 1.6B, quantized with the help of imatrix so it could offer better performance for being quantized, and have quantization levels available for lower-memory devices to run. Kalomaze's "groups_merged.txt" was used for the importance matrix, with context set to 4,096 (the context length according to their paper).

Here's a chart that provides an approximation of the HellaSwag score (out of 1,000 tasks). Thanks to the randomization of tasks, it may be slightly unprecise: |Quantization|HellaSwag| |------------|---------| |IQ1_S |35.4% | |IQ1_M |38.7% | |IQ2_XXS |51.2% | |IQ2_XS |51.8% | |IQ2_S |56.8% | |IQ2_M |59.3% | |Q2_K_S |55.2% | |Q2_K |59.0% | |IQ3_XXS |60.8% | |Q4_0 |64.0% | |Q4_K_M |66.0% | |Q5_K_M |65.8% |

Original model card below.

StableLM 2 Chat 1.6B

Model Description

Stable LM 2 Chat 1.6B is a 1.6 billion parameter instruction tuned language model inspired by HugginFaceH4's Zephyr 7B training pipeline. The model is trained on a mix of publicly available datasets and synthetic datasets, utilizing Direct Preference Optimization (DPO).

Usage

StableLM 2 1.6B Chat uses the following ChatML format:

from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained('stabilityai/stablelm-2-1_6b-chat')
model = AutoModelForCausalLM.from_pretrained(
    'stabilityai/stablelm-2-1_6b-chat',
    device_map="auto",
)

prompt = [{'role': 'user', 'content': 'Implement snake game using pygame'}]
inputs = tokenizer.apply_chat_template(
    prompt,
    add_generation_prompt=True,

Your obligations as a stablelm 2 1 6b chat imatrix GGUF deployer

Track your progress

Sign up to track obligations, receive deadline reminders, and generate audit packages.

Tags

Languages