CLIP-convnext_base_w-laion2B-s13B-b82K-augreg EU AI Act Compliance Profile

Model Card for CLIP-convnext_base_w.laion2B-s13B-b82k-augreg

Table of Contents

1. Model Details
2. Uses
3. Training Details
4. Evaluation
5. Acknowledgements
6. Citation

Model Details

Model Description

A series of CLIP ConvNeXt-Base (w/ wide embed dim) models trained on subsets LAION-5B (https://laion.ai/blog/laion-5b/) using OpenCLIP (https://github.com/mlfoundations/open_clip).

Goals:

• Explore an alternative to ViT and ResNet (w/ AttentionPooling) CLIP models that scales well with model size and image resolution

Firsts:

• First known ConvNeXt CLIP models trained at scale in the range of CLIP ViT-B/16 and RN50x4 models
• First released model weights exploring increase of augmentation + regularization for image tower via adding (greater scale range of RRC, random erasing, stochastic depth)

The models utilize the timm ConvNeXt-Base model (convnext_base) as the image tower, and the same text tower as the RN50x4 (depth 12, embed dim 640) model from OpenAI CLIP. The base models are trained at 256x256 image resolution and roughly match the RN50x4 models on FLOPs and activation counts. The models with 320 in the name are trained at 320x320.

All models in this series were trained for 13B samples and have ImageNet Zero-Shot top-1 of >= 70.8%. Comparing to ViT-B/16 at 34B SS with zero-shot of 70.2% (68.1% for 13B SS) this suggests the ConvNeXt architecture may be more sample efficient in this range of model scale. More experiments needed to confirm.

| Model | Dataset | Resolution | AugReg | Top-1 ImageNet Zero-Shot (%) | | ----- | ------- | ---------- | ------------ | --------- | | convnext_base_w.laion2b_s13b_b82k | LAION-2B | 256x256 | RRC (0.9, 1.0) | 70.8 | | [convnext_base_w.laion2b_

Your obligations as a CLIP-convnext_base_w-laion2B-s13B-b82K-augreg deployer

Track your progress

Sign up to track obligations, receive deadline reminders, and generate audit packages.

Tags

Library