Llama 3.3 8B Instruct GGUF EU AI Act Compliance Profile

<span style="color: #7FFF7F;">Llama-3.3-8B-Instruct GGUF Models</span>

<span style="color: #7F7FFF;">Model Generation Details</span>

This model was generated using llama.cpp at commit 0f89d2ecf.

<span style="color: #7FFF7F;">Quantization Beyond the IMatrix</span>

I've been experimenting with a new quantization approach that selectively elevates the precision of key layers beyond what the default IMatrix configuration provides.

In my testing, standard IMatrix quantization underperforms at lower bit depths, especially with Mixture of Experts (MoE) models. To address this, I'm using the --tensor-type option in llama.cpp to manually "bump" important layers to higher precision. You can see the implementation here:
👉 Layer bumping with llama.cpp

While this does increase model file size, it significantly improves precision for a given quantization level.

I'd love your feedback—have you tried this? How does it perform for you?

Llama 3.3 8B Instruct

Yes, this is official, and yes, this is, to my knowledge, a real version of Llama 3.3 8B. (I think, anyways)

Facebook has a Llama API available that allows for inference of the other Llama models (L3.3 70B, L4 Scout and Maverick), but also includes a special, new (according to the original press release) "Llama 3.3 8B" that didn't exist anywhere else and was stuck behind the Facebook API!

However. The Llama API supports finetuning L3.3... and downloading the final model in HF format. Problem solved, r

Your obligations as a Llama 3.3 8B Instruct GGUF deployer

Track your progress

Sign up to track obligations, receive deadline reminders, and generate audit packages.

Tags