AutoGaze EU AI Act Compliance Profile

AutoGaze

Project Page | Paper | GitHub | Models & Data & Benchmark | Demo

AutoGaze is a ultra light-weight model that automatically removes redundant patches in a video before passing it to any Vision Transformer (ViT) or Multi-modal Large Language Model (MLLM).

Speficially, AutoGaze perceives each frame and autoregressively selects ("gazing") a minimal set of patches that can reconstruct the original video (i.e., non-redundant patches) up to a reconstruction loss threhold provided by the user. AutoGaze can self-decide when to stop gazing for each frame based on user's request on the acceptable maximum reconstruction loss.<br>

Empircally, AutoGaze can reduce #tokens in ViTs/MLLMs by up to 100x, reducing their latency by up to 19x/10x. This enables efficiently scaling MLLMs to 4K-resolution, 1K-frame videos, improving performance on benchmarks such as VideoMME. Especially, it improves performance by 14% on HLVid, a high-resolution long-form video benchmark proposed in this work as well.

This model is for research and development only. <br>

Quick Start:

See our GitHub repo for instructions on how to use AutoGaze.

License/Terms of Use:

NVIDIA license (see LICENSE.md). The reference to the NVIDIA License means the attached custom NSCLv1 license, under which users may use for purposes of conducting non-commercial research activities and non-commercial research publications.

Deployment Geography:

Global <br>

Use Case: <br>

The model is used to remove redundancy in videos and accelerate video encoders and MLLMs. <br>

References(s):

GitHub: https://github.com/NVlabs/AutoGaze <br>

Model Architecture:

Architecture Type: CNN and Transformer. <br>

**Network Arc

Your obligations as a AutoGaze deployer

Track your progress

Sign up to track obligations, receive deadline reminders, and generate audit packages.

Tags