gemma3-vision-encoder EU AI Act Compliance Profile

Gemma 3 Vision Encoder (extracted)

This repository contains the SigLIP-family vision encoder extracted from google/gemma-3-4b-pt. It also includes the Gemma multimodal projector weights (state_dict) and a small metadata file.

Contents

• config.json, model.safetensors: the SigLIP vision encoder
• preprocessor_config.json: the image processor settings used by Gemma 3
• projector_state_dict.pt: PyTorch state dict for the Gemma projector
• projector_config.json: metadata (class, dims, token count if detected)
• NOTICE: Gemma Terms pointer

Basic usage (encoder as feature extractor)

from transformers import SiglipVisionModel, AutoImageProcessor
from PIL import Image
import torch

repo_id = "<your-username>/<your-repo>"
encoder = SiglipVisionModel.from_pretrained(repo_id).eval()
processor = AutoImageProcessor.from_pretrained(repo_id)

img = Image.open("test.jpg").convert("RGB")
inputs = processor(images=img, return_tensors='pt')
with torch.no_grad():
    feats = encoder(**inputs).last_hidden_state  # (B, Tv, Dv)
print(feats.shape)

Using the projector (Gemma-style multimodal path)

The projector here is provided as a state dict plus metadata. It is intended for users who are wiring a Gemma-style VLM, where the projector maps the vision sequence to a fixed number of image tokens at the LLM hidden size.

Two common paths:

1. Use with Transformers' Gemma 3 model: load the full VLM, then load this projector's state_dict into the model's multi_modal_projector module.

import torch
from transformers import Gemma3ForConditionalGeneration

repo_id = "<your-username>/<your-repo>"
vlm = Gemma3ForConditionalGeneration.from_pretrained('google/gemma-3-4b-pt', device_map='cpu')
sd = torch.load('projector_state_dict.pt', map_location='cpu')  # or from the repo checkout
vlm.multi_modal_projector.load_state_dict(sd, strict=False)
vlm.eval()

1. Recreate the projector module from the class name, instantiate it, an

Your obligations as a gemma3-vision-encoder deployer

Track your progress

Sign up to track obligations, receive deadline reminders, and generate audit packages.