Video LLaVA 7B hf EU AI Act Compliance Profile

Model Card for Video-LLaVa

Model Details

Model type: Video-LLaVA is an open-source multomodal model trained by fine-tuning LLM on multimodal instruction-following data. It is an auto-regressive language model, based on the transformer architecture. Base LLM: lmsys/vicuna-13b-v1.5

Model Description: The model can generate interleaving images and videos, despite the absence of image-video pairs in the dataset. Video-LLaVa is uses an encoder trained for unified visual representation through alignment prior to projection. Extensive experiments demonstrate the complementarity of modalities, showcasing significant superiority when compared to models specifically designed for either images or videos.

<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/videollava_example.png" alt="drawing" width="600"/>

<small> VideoLLaVa example. Taken from the <a href="https://arxiv.org/abs/2311.10122">original paper.</a> </small>

Paper or resources for more information: https://github.com/PKU-YuanGroup/Video-LLaVA

🗝️ Training Dataset

• The images pretraining dataset is from LLaVA.
• The images tuning dataset is from LLaVA.
• The videos pretraining dataset is from Valley.
• The videos tuning dataset is from Video-ChatGPT.

How to Get Started with the Model

Use the code below to get started with the model.

from PIL import Image
import requests
import numpy as np
import av
from huggingface_hub import hf_hub_download
from transformers import VideoLlavaProcessor, VideoLlavaForConditionalGeneration

def read_video_pyav(container, indices):
    '''
    Decode the video with PyAV decoder.

    Args:
        container (av.container.input.InputContainer): PyAV container.
        ind

Your obligations as a Video LLaVA 7B hf deployer

Track your progress

Sign up to track obligations, receive deadline reminders, and generate audit packages.

Tags

Library