DermLIP ViT B 16 EU AI Act Compliance Profile

DermLIP: Dermatology Language-Image Pretraining

Model Description

DermLIP is a vision-language model for dermatology, trained on the Derm1M dataset—the largest dermatological image-text corpus to date.

Model Details

• Model Type: Pretrained Vision-Language Model (CLIP-style)

• Architecture:

  • Vision encoder: ViT-B16
  • Text encoder: GPT2

• Resolution: 224×224 pixels

• Paper: https://arxiv.org/abs/2503.14911

• Repository: https://github.com/SiyuanYan1/Derm1M

• license: cc-by-nc-nd-4.0

Training Details

• Training data: 403,563 skin image-text pairs from Derm1M datasets. Images include both dermoscopic and clinical images.
• Training objective: image-text contrastive loss
• Hardware: 1 x Nvidia H200 （～40GB memory usage）
• Hours used: ~5 hours

Intended Uses

Primary Use Cases

• Zero-shot classification
• Few-shot learning
• Cross-modal retrieval
• Concept annotation/explanation

How to Use

Installation

First, clone the Derm1M repository:

git clone git@github.com:SiyuanYan1/Derm1M.git
cd Derm1M
···

Then install the package following the instruction in the repository.


### Quick Start
```python
import open_clip
from PIL import Image
import torch

# Load model with huggingface checkpoint
model, _, preprocess = open_clip.create_model_and_transforms(
    'hf-hub:redlessone/DermLIP_ViT-B-16'
)
model.eval()

# Initialize tokenizer
tokenizer = open_clip.get_tokenizer('hf-hub:redlessone/DermLIP_ViT-B-16')

# Read example image
image = preprocess(Image.open("your_skin_image.png")).unsqueeze(0)

# Define disease labels (example: PAD dataset classes)
PAD_CLASSNAMES = [
    "nevus",
    "basal cell carcinoma",
    "actinic keratosis",
    "seborrheic keratosis",
    "squamous cell carcinoma",
    "melanoma"
]

# Build text prompts
template = lambda c: f'This is a skin image of {c}'
text = tokenizer([template(c) for c in PAD_CLASSNAMES])

# Inference
with torch.no_grad

Your obligations as a DermLIP ViT B 16 deployer

Track your progress

Sign up to track obligations, receive deadline reminders, and generate audit packages.

Tags

Languages

Library