videoprism-base-f16r288 EU AI Act Compliance Profile

VideoPrism Model Card

Paper: https://huggingface.co/papers/2402.13217

arXiv: https://arxiv.org/pdf/2402.13217

GitHub: https://github.com/google-deepmind/videoprism

Blog: https://research.google/blog/videoprism-a-foundational-visual-encoder-for-video-understanding/

VideoPrism is a foundational video encoder that enables state-of-the-art performance on a large variety of video understanding tasks. It takes video frames as input and outputs compact embeddings of the frames, which one can conveniently feed into classifiers, LLMs, retrieval models, etc. When tested on 33 public video understanding benchmarks over four task categories, a single frozen VideoPrism checkpoint outperforms previous best-performing foundation models on 31 of them, with no fine-tuning on target task datasets.

Model details

We release the following model variants:

| Model Name | Configuration Name | Model Type | Backbone | #Params | File Size | Checkpoint | | -------- | -------- | ------- | :-------: | :-------: | :-------: | :-------: | | VideoPrism-B | videoprism_public_v1_base | Video encoder | ViT-B | 114M | 458MB | link | | VideoPrism-L | videoprism_public_v1_large | Video encoder | ViT-L | 354M | 1.42GB | link | | VideoPrism-LvT-B | videoprism_lvt_public_v1_base | Video-text encoders | ViT-B | 248M | 991MB | link | | VideoPrism-LvT-L | videoprism_lvt_public_v1_large | Video-text encoders | ViT-L | 580M | 2.30GB | link |

Model description

VideoPrism-B/L are the composition of a Vision Transformer image encoder and four temporal-attention Transformer layers. The image encoder and text encoder are initialized from CoCa, which is trained on WebLI following the CoCa recipes. VideoPri

Your obligations as a videoprism-base-f16r288 deployer

Track your progress

Sign up to track obligations, receive deadline reminders, and generate audit packages.

Tags

Library