Vinci: A Real-time Embodied Smart Assistant based on Egocentric Vision-Language Model

Abstract: We introduce Vinci, a real-time embodied smart assistant built upon an egocentric vision-LLM. Designed for deployment on portable devices such as smartphones and wearable cameras, Vinci operates in an "always on" mode, continuously observing the environment to deliver seamless interaction and assistance. Users can wake up the system and engage in natural conversations to ask questions or seek assistance, with responses delivered through audio for hands-free convenience. With its ability to process long video streams in real-time, Vinci can answer user queries about current observations and historical context while also providing task planning based on past interactions. To further enhance usability, Vinci integrates a video generation module that creates step-by-step visual demonstrations for tasks that require detailed guidance. We hope that Vinci can establish a robust framework for portable, real-time egocentric AI systems, empowering users with contextual and actionable insights. We release the complete implementation for the development of the device in conjunction with a demo web platform to test uploaded videos at https://github.com/OpenGVLab/vinci.

• The paper introduces Vinci, a real-time embodied smart assistant utilizing an egocentric vision-language model and a memory module for context-aware interaction from the user's perspective.
• Vinci integrates modules for real-time multimedia input processing, a fine-tuned vision-language model, and a temporal memory system to achieve detailed scene understanding and task planning.
• This system demonstrates practical capabilities in scene understanding, temporal grounding, and future planning, suggesting strong potential for integration into AR/VR, robotics, and future portable AI systems.

Vinci: A Real-time Embodied Smart Assistant based on Egocentric Vision-LLM

The paper introduces Vinci, an innovative smart assistant that operates in real-time, utilizing an embodied egocentric vision-LLM (VLM). Vinci embodies advancements in real-time video processing, natural language interaction, and context-aware AI systems, situating itself as an essential development for wearable devices and applications requiring constant environmental interaction.

Egocentric vision, which captures the world from the user's perspective, is central to this work. This approach is particularly effective for portable devices, providing users with an intuitive and immersive experience. However, existing systems often fail to maintain real-time performance on resource-constrained devices or to retain contextual historical information for user interactions, which Vinci addresses adeptly.

System Architecture and Components

Vinci is built on a sophisticated infrastructure comprising several modules:

1. Input Processing Module: This module is responsible for managing live streaming video and audio inputs from user devices, forming the basis for the Vinci’s interaction capabilities. By converting audio to text and ensuring video and textual data alignment, it sets the stage for effective downstream processing.
2. EgoVideo-VL Model: At the heart of Vinci, this model is designed from the EgoVideo foundation and enriched by a two-stage fine-tuning with a tailored dataset drawn from Ego4D, EgoExoLearn, and Ego4D-Goalstep. It integrates visual inputs with language queries, underpinned by large-scale LLMs like InternLM-7B, to deliver flexible and robust real-time interaction.
3. Memory Module: Critical for contextual grounding, this module captures and organizes historical video snapshots to support temporal reasoning. It enables Vinci to anchor responses not just in the present but in the nuanced context of past interactions, thus facilitating intricate task planning and user assistance.
4. Generation and Retrieval Modules: These modules augment Vinci’s user assistance capabilities. The Generation Module, trained via SEINE, offers visual demonstrations essential for tasks requiring explicit guidance, while the Retrieval Module provides relevant third-person instructional videos, demonstrating Vinci's multifaceted adaptability.

Practical Capabilities and Performance

Vinci exhibits a plethora of capabilities crucial for advanced AI systems. These include precise scene understanding, capable of identifying current actions and environments; temporal grounding, allowing for detailed event tracking and summary generation; and future planning, predicting subsequent steps in a user's workflow. Each of these abilities underscores Vinci's utility in dynamic, real-world applications.

The utility of Vinci is further enhanced by its integration capabilities with augmented reality (AR) and virtual reality (VR) environments, alongside real-world scenarios requiring embodied AI, like robotics. Its modular approach makes it versatile, allowing for application across domains demanding context-aware, real-time interaction.

Concluding Remarks and Future Directions

Vinci emerges as a pivotal advancement in real-time smart assistant technology. By marrying egocentric vision with advanced VLMs, it opens new vistas for portable AI systems. Additionally, the released open-source codebase lays a foundation for further exploration, encouraging advancements within the community.

Future developments could leverage Vinci’s framework to enhance AR/VR systems, delivering unprecedented assistance in environments necessitating detailed procedural guidance and contextual interaction. Moreover, its potential extends into fields like smart homes and educational technologies, further demonstrating its relevance as a prototype for future AI applications requiring immersive and interactive solutions.