flava-full EU AI Act Compliance Profile

Model Card: FLAVA

Model Details

FLAVA model was developed by the researchers at FAIR to understand if a single model can work across different modalities with a unified architecture. The model was pretrained solely using publicly available multimodal datasets containing 70M image-text pairs in total and thus fully reproducible. Unimodal datasets ImageNet and BookCorpus + CCNews were also used to provide unimodal data to the model. The model (i) similar to CLIP can be used for arbitrary image classification tasks in a zero-shot manner (ii) used for image or text retrieval in a zero-shot manner (iii) can also be fine-tuned for natural language understanding (NLU) tasks such as GLUE and vision-and-language reasoning tasks such as VQA v2. The model is able to use the data available as images, text corpus and image-text pairs. In the original paper, the authors evaluate FLAVA on 32 tasks from computer vision, NLU and vision-and-language domains and show impressive performance across the board scoring higher micro-average than CLIP while being open.

Model Date

Model was originally released in November 2021.

Model Type

The FLAVA model uses a ViT-B/32 transformer for both image encoder and text encoder. FLAVA also employs a multimodal encoder on top for multimodal tasks such as vision-and-language tasks (VQA) which is a 6-layer encoder. Each component of FLAVA model can be loaded individually from facebook/flava-full checkpoint. If you need complete heads used for pretraining, please use FlavaForPreTraining model class otherwise FlavaModel should suffice for most use case. This repository also contains code to pretrain the FLAVA model from scratch.

Documents

• FLAVA Paper

Using with Transformers

FlavaModel

FLAVA model supports vision, language and multimodal inputs. You can pass inputs corresponding to the domain you are concer

Your obligations as a flava-full deployer

Track your progress

Sign up to track obligations, receive deadline reminders, and generate audit packages.

Tags

Library