vit_base_patch16_clip_224.openai EU AI Act Compliance Profile

CLIP (OpenAI model for timm)

Model Details

The CLIP model was developed by researchers at OpenAI to learn about what contributes to robustness in computer vision tasks. The model was also developed to test the ability of models to generalize to arbitrary image classification tasks in a zero-shot manner. It was not developed for general model deployment - to deploy models like CLIP, researchers will first need to carefully study their capabilities in relation to the specific context they’re being deployed within.

This instance of the CLIP model is intended for loading in

• timm (https://github.com/rwightman/pytorch-image-models) and
• OpenCLIP (https://github.com/mlfoundations/open_clip) libraries.

Please see https://huggingface.co/openai/clip-vit-base-patch16 for use in Hugging Face Transformers.

Model Date

January 2021

Model Type

The model uses a ViT-B/16 Transformer architecture as an image encoder and uses a masked self-attention Transformer as a text encoder. These encoders are trained to maximize the similarity of (image, text) pairs via a contrastive loss. The original implementation had two variants: one using a ResNet image encoder and the other using a Vision Transformer. This repository has the variant with the Vision Transformer.

Documents

• Blog Post
• CLIP Paper

Model Use

Intended Use

The model is intended as a research output for research communities. We hope that this model will enable researchers to better understand and explore zero-shot, arbitrary image classification. We also hope it can be used for interdisciplinary studies of the potential impact of such models - the CLIP paper includes a discussion of potential downstream impacts to provide an example for this sort of analysis.

Primary intended uses

The primary intended users of these models are AI researchers. We primarily imagine the model will be used by researchers to better

Your obligations as a vit_base_patch16_clip_224.openai deployer

Track your progress

Sign up to track obligations, receive deadline reminders, and generate audit packages.

Tags

Library