Metadata-Version: 2.4
Name: vlm-monitor
Version: 0.0.3
Summary: A second set of eyes.
Author-email: Salman Naqvi <s1148093@s.eduhk.hk>
License: Apache-2.0
Project-URL: Repository, https://github.com/ForBo7/vlm-monitor
Project-URL: Documentation, https://ForBo7.github.io/vlm-monitor/
Keywords: nbdev,vlms,llms,computer vision,natural language processing
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: fastcore==2.2.13
Requires-Dist: fastlite
Requires-Dist: python-fastllm==0.0.36
Requires-Dist: aidialog==0.0.6
Requires-Dist: fastprogress
Requires-Dist: tiktoken
Requires-Dist: pycachy
Dynamic: license-file

# vlm-monitor


<!-- WARNING: THIS FILE WAS AUTOGENERATED! DO NOT EDIT! -->

This is a general purpose library that allows you to use a VLM (Vision Language Model) to thoroughly describe the contents of a video, with subtitles included for additional context.

In essence, this library provides a second set of eyes.

The output is a database object containing the video description.

If you want to directly run this notebook, you want to have an `OPENROUTER_API_KEY` set.

### Installation

Install latest from the GitHub [repository](https://github.com/ForBo7/vlm-monitor):

``` sh
$ pip install git+https://github.com/ForBo7/vlm-monitor.git
```

or from [pypi](https://pypi.org/project/vlm-monitor/)

``` sh
$ pip install vlm_monitor
```

### Documentation

Documentation can be found hosted on this GitHub [repository](https://github.com/ForBo7/vlm-monitor)’s [pages](https://ForBo7.github.io/vlm-monitor/). Additionally you can find package manager specific guidelines on [pypi](https://pypi.org/project/vlm-monitor/).

## Preface

This is a library that allows you to thoroughly describe what occurs in a video.

**Concisely:**

- A VLM describes the video frame by frame. Each time, it is provided with an empty history together with the frame, the subtitle, and instructions about what exactly to do to describe the frame. **If one is processing a 5 minute video with a sample rate of 1 frame per second, the output is 300 individual, isolated frame descriptions.**
- An LLM then takes all the isolated frame descriptions and pieces them together to form a summary/description of the video. **The user could, for instance, piece together a video description whos summary windows consist of 60 frames (in which case, when the LLM produces the summary of the next window, it will keep the previous windows in its chat history). Or, the user could piece toether an overall video description consisting of a single window comprising of 300 frames.**

The biggest beneficiary of this approach is LLM context. Traditional VLM description systems keep the image in the chat history. Images are token heavy. Storing the desription of the image, rather than the image itself, saves the necessary information whilst allowing higher definition: you can describe videos at 1 frame per second, or even lower if you desire so.

**More concretely, this library works as follows:**

1.  Set up a database to store the data.
2.  Load the videos and their frames into the database.
3.  Allow a VLM to describe the frames.
4.  Allow a LLM to piece together a description.

**And at an even lower level, as follows:**

1.  Set up a database consisting of 4 tables.
    - A `video` table to store metadata about your videos
    - A `frame` table to store metadata about the frames in each of your videos
    - A `run` table to store metadata about each description process
    - A `runframe` table to store metadata about each described frame
2.  Populate the `video` and `frame` tables
3.  Define the VLM settings
4.  Process the frames through the VLM, storing the frame descriptions in `runframe`
5.  Process the resulting descriptions through the LLM, storing the resulting summary in `video`

## Example Usage

``` python
from vlm_monitor.core import *
```

``` python
!rm -rf test_db.db
db = init_db('test_db.db'); db
```

    <Database <apsw.Connection "/app/data/vlm-monitor/nbs/test_db.db">>

``` python
?init_db
```

``` python
def init_db(
    path:str | pathlib.Path='db.db', # Path to database
)->Database:
    "Initialize a database and return it."
```

**File:** `~/vlm-monitor/vlm_monitor/core.py`; line: 35

**Type:** function

Intialize the database with the relevant tables.

``` python
dpath = Path('../../data/timss/'); dpath.ls()[:5]
```

    [Path('../../data/timss/M-CZ3'), Path('../../data/timss/M-AU2'), Path('../../data/timss/M-CZ4'), Path('../../data/timss/M-JP2'), Path('../../data/timss/S-AU4')]

``` python
dpaths = filter_paths(dpath.ls()).sorted(lambda o: (o.stem[:-1], o.stem[-1])); dpaths[:5]
```

    [Path('../../data/timss/M-AU1'), Path('../../data/timss/M-AU2'), Path('../../data/timss/M-AU3'), Path('../../data/timss/M-AU4'), Path('../../data/timss/M-CZ1')]

``` python
?filter_paths
```

``` python
def filter_paths(
    paths:list, # List of paths to filter
    chs:str='.', # Characters to check for in the component
    comp:str='stem', # Path attribute to inspect (e.g. 'stem', 'name')
    negate:bool=True, # If True, exclude paths whose `comp` contains `chs`; if False, keep only those
)->list: # Filtered list of paths
    "Filter paths by whether a path component contains specified characters."
```

**File:** `~/vlm-monitor/vlm_monitor/core.py`; line: 63

**Type:** function

``` python
populate_db(db, dpaths)
```

``` python
?populate_db
```

``` python
def populate_db(
    db:Database, # Database to populate
    paths:list, # List of video directories
    sample_rate:int=1, # Sampling rate for frames
    trans_suffix:str='txt', # Transcript file suffix
)->None:
    "Populate video and frame tables from a list of video directories."
```

**File:** `~/vlm-monitor/vlm_monitor/core.py`; line: 122

**Type:** function

``` python
len(db.t.video()), len(db.t.frame())
```

    (52, 149880)

Populate the database with your data. `populate_db` assumes your data exists in a flat directory as follows.

``` python
Path('../../data/timss').ls()[:5]
```

    [Path('../../data/timss/M-CZ3'), Path('../../data/timss/M-AU2'), Path('../../data/timss/M-CZ4'), Path('../../data/timss/M-JP2'), Path('../../data/timss/S-AU4')]

Each flder is a video containing all frames for that video, as well as that video’s transcript in SRT format, saved as a `.txt` file.

``` python
s = session(system='Reply concisely', model='bytedance-seed/seed-2.0-mini', vendor_name='openrouter', reasoning_effort='high')
```

``` python
?session
```

``` python
def session(
    msgs:list | None=None, model:str='', max_think:float=inf, usage:bool=True, display:bool=True, **kwargs
):
    "Create a stream partial with preset model/kwargs."
```

**File:** `~/vlm-monitor/vlm_monitor/core.py`; line: 204

**Type:** function

A session is an instance of a LLM.

``` python
from PIL import Image
```

``` python
Image.open('test.jpg')
```

![](index_files/figure-commonmark/cell-15-output-1.png)

``` python
r = await s([user('what do your elf eyes see?', img2b64(Path('test.jpg')))]); r
```

Elf eyes spot this crisp Honda EBR2300CX portable gas generator, nestled in sun-warmed wild grass dotted with tiny pink clover blooms. I notice the black roll-cage frame, the plugged-in power cord, the Japanese-labeled control panel with its voltage meter and circuit breaker, and the branded recoil starter on the engine side.

<details>

- model: `bytedance-seed/seed-2.0-mini`
- finish_reason: `stop`
- usage: `Usage(prompt_tokens=1336, completion_tokens=399, total_tokens=1735, cached_tokens=0, cache_creation_tokens=0, reasoning_tokens=326, raw={'prompt_tokens': 1336, 'completion_tokens': 399, 'total_tokens': 1735, 'cost': 0.0002932, 'is_byok': False, 'prompt_tokens_details': {'cached_tokens': 0, 'cache_write_tokens': 0, 'audio_tokens': 0, 'video_tokens': 0}, 'cost_details': {'upstream_inference_cost': 0.0002932, 'upstream_inference_prompt_cost': 0.0001336, 'upstream_inference_completions_cost': 0.0001596}, 'completion_tokens_details': {'reasoning_tokens': 326, 'image_tokens': 0, 'audio_tokens': 0}})`

</details>

``` python
?user
```

``` python
def user(
    txt:str, img:str | None=None
)->Msg:
    "Build a user message with optional image."
```

**File:** `~/vlm-monitor/vlm_monitor/core.py`; line: 147

**Type:** function

``` python
?img2b64
```

``` python
def img2b64(
    path:Path
)->str:
    "Encode an image file as a base64 data URL."
```

**File:** `~/vlm-monitor/vlm_monitor/core.py`; line: 197

**Type:** function

I’ll now try process an entire video.

``` python
vlm_prompt = 'Tell me what your elf eyes see. In particular, pay attention to anything that looks green.'
```

``` python
vid = db.t.video[1].id; vid
```

    1

``` python
db.t.video[1].title
```

    'M-AU1'

``` python
o = await deploy_run(vid, db, s, vlm_prompt, prompt_type='default', stop=300, step=3, cache=True, n_workers=20, pause=0.1, max_retries=2)
```

``` python
?deploy_run
```

``` python
async def deploy_run(
    video_id:int, db:Database, session:Callable, prompt:str, prompt_type:str, start:int=0, stop:int | None=None,
    step:int=1, cache:bool=False, include_subs:bool=True, n_workers:int=8, pause:float=3, max_retries:int=2
)->Run:
    "Run a single prompt across a range of frames, storing results in the database."
```

**File:** `~/vlm-monitor/vlm_monitor/core.py`; line: 307

**Type:** function

Individual frame descriptions have been stored. It is now time to combine everything together to form a coherent narrative.

``` python
summary_prompt = 'You will receive a series of individual frame descriptions. You need to combine these individual descriptions together into a coherent narrative.'
```

``` python
summary = await summarize_run(db, s, sys_prompt=summary_prompt, run_id=o.id, window_sec=60, step=1, cache=True)
```

``` python
?summarize_run
```

``` python
async def summarize_run(
    db:Database, session:Callable, sys_prompt:str, run_id:int | None=None, video_id:int | None=None,
    window_sec:int=300, step:int=1, cache:bool=False
)->str:
    "Summarize a single run or all frames for a video in rolling windows."
```

**File:** `~/vlm-monitor/vlm_monitor/core.py`; line: 424

**Type:** function

``` python
print(summary[:500])
```

    [1–178s] ### Cohesive Narrative of the Classroom Footage
    What unfolds across the timed footage is a detailed tour of multiple school classroom settings, viewed through the sharp, observant lens of elven vision, with green as the recurring standout accent color.

    Opening in the first 25 seconds, the scene is a quiet, empty classroom: bright green padded plastic chair seats and backs pair with dark-topped student desks clustered across the room, with faint green trim lining the left-side windows. 

And that’s that.

## Developer Guide

This library is built using [nbdev](https://nbdev.fast.ai/), a way to create dlightful software with Jupyter Notebooks. Learn how to get started with nbdev [here](https://nbdev.fast.ai/tutorials/tutorial.html).

### Install vlm_monitor in Development mode

``` sh
# make sure vlm_monitor package is installed in development mode
$ pip install -e .

# make changes under nbs/ directory
# ...

# compile to have changes apply to vlm_monitor
$ nbdev-prepare
```

After cloning, be usre to run [`nbdev-install-hooks`](http://nbdev.fast.ai/tutorials/tutorial.html#install-hooks-for-git-friendly-notebooks) in your terminal to install Jupyter and git hooks. These hooks clean, trust, and fix merge conflicts in notebooks.

Anytime you make changes to the repo, run [`nbdev-prepare`](http://nbdev.fast.ai/tutorials/tutorial.html#prepare-your-changes).

## Credit

This library is built using [nbdev](https://nbdev.fast.ai/) on [SolveIt](https://solve.it.com/), both by \[Answer.AI\]. Other libraries used include:
- [fastcore](https://github.com/AnswerDotAI/fastcore)
- [fastlite](https://github.com/AnswerDotAI/fastlite)
- [fastllm](https://github.com/AnswerDotAI/fastllm)
- [aidialog](https://github.com/AnswerDotAI/aidialog/)
- [fastprogress](https://github.com/AnswerDotAI/fastprogress)
- [pycachy](https://github.com/AnswerDotAI/cachy)
- [tiktoken](https://github.com/openai/tiktoken)
