Metadata-Version: 2.4
Name: frame2tensor
Version: 0.6.0
Summary: A simple Python toolkit for real-time Computer Vision pipelines on Linux systems.
Keywords: cuda,opengl,interop,screen-capture,x11,gpu,tensor,computer-vision
Author: Shep L
License-Expression: MIT
License-File: LICENSE
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: GPU :: NVIDIA CUDA
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Multimedia :: Graphics :: Capture :: Screen Capture
Classifier: Topic :: Scientific/Engineering :: Image Processing
Classifier: Typing :: Typed
Requires-Dist: cuda-python>=12.0.0,<13.0.0
Requires-Dist: glfw>=2.10.0
Requires-Dist: moderngl>=5.12.0
Requires-Dist: moderngl-window>=3.1.1
Requires-Dist: pyopengl>=3.1.10
Requires-Dist: python-xlib>=0.33
Requires-Python: >=3.11
Project-URL: Documentation, https://frame2tensor.readthedocs.io
Project-URL: Homepage, https://github.com/ShepL137/frame2tensor
Project-URL: Issues, https://github.com/ShepL137/frame2tensor/issues
Description-Content-Type: text/markdown

# frame2tensor

A simple Python toolkit for real-time Computer Vision pipelines on Linux systems.
Using the CUDA runtime API, it registers GL textures with CUDA: enabling on-device interoperation between CUDA, GL, and ML frameworks.

Designed for: game playing agents/bots, neural visualization, synthetic data generation, and anything else where frames and tensors must remain on the GPU.

>Requires: X11 & a CUDA-capable GPU; Python 3.11+

## Features

- Capture a window or a custom draw function into a tensor.
- Render tensors and GL draw functions to a window or disk.
- Write tensors into a GL texture, e.g. for visualization.

## Dependencies

`frame2tensor` depends on `cuda-python`, which must match your installed NVIDIA driver. If you see `cudaErrorInsufficientDriver`, your driver is too old for the installed `cuda-python` version.
Check with `nvidia-smi`, then consult the [CUDA toolkit compatibility matrix](https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/) to find the compatible `cuda-python` range and pin it in your dependencies.

>NOTE: this project was developed on a 3060 with a 525–550 series driver. I have no idea how to support other hardware at the moment. Keep this in mind if you have any issues.

`xwininfo` is used by `window_finder.pick_window`.

`ffmpeg` is used by `VideoWriter.write`.

## Install

Install from GitHub:

```
uv add git+https://github.com/ShepL137/frame2tensor.git
```
or
```
pip install git+https://github.com/ShepL137/frame2tensor.git
```

## Quick Start

### Window capture

Capture a live window and feed it to a PyTorch model.

```python
import torch
from frame2tensor.capture import X11Window, pick_window

with X11Window(pick_window()) as win:
    buf    = win.capture()                          # CUDABuffer, 4 channel uint8
    tensor = torch.as_tensor(buf, device="cuda")    # zero-copy wrap
    tensor = tensor.float() / 255.0                 # cast + copy to float (we're working to eliminate this)
    model(tensor)                                   # feed it your CV pipeline
```

`CUDABuffer` exposes `__cuda_array_interface__`, so any compatible framework can wrap it zero-copy.

`capture()` returns a borrowed reference, the same object on every call, mutated in place; `tensor` above is a zero-copy wrap of it, so clone it before exiting the context manager if you still need the old frame.
>WARNING: on a source-window resize, `capture()` frees the old buffer and allocates a new one internally, so a zero-copy wrap that _wasn't_ cloned is now pointing to memory that's been freed.

`X11Window` takes an X11 XID (decimal or `0x`-prefixed hex):
- `pick_window()` selects one interactively by click (requires `xwininfo`)
- `get_active_window()` returns the currently focused window
- or pass an XID directly.

See [docs/quickstart.md](https://github.com/ShepL137/frame2tensor/blob/master/docs/quickstart.md) for more.

## Examples

See [`examples/`](https://github.com/ShepL137/frame2tensor/tree/master/examples) for complete, self-contained demos:

- [`motion_detection`](https://github.com/ShepL137/frame2tensor/tree/master/examples/motion_detection): live frame-differencing motion filters over a captured window (including windowed and click-through AR overlay versions).
- [`retinal_opponency`](https://github.com/ShepL137/frame2tensor/tree/master/examples/retinal_opponency): a center-surround color-opponency filter over a captured window, with four views switchable at runtime.
- [`rnn_activity`](https://github.com/ShepL137/frame2tensor/tree/master/examples/rnn_activity): an RNN's per-layer activations rendered live as heatmap tiles.

## Motivation

I wanted to make a bot with biologically motivated real-time vision to play games, exploring such things as world models; for, games and other applications supply a vast range of complex dynamics and challenges.

I wasn't sure of anything besides mss for the question of frame capture. Perhaps similar projects as this exist, but I didn't manage to discover them. Regardless, I should hope that this one might help to boost the signal for such solutions.

I only wanted to get frames as tensors without the CPU. I do hope that this might spare others the trouble; that others may, as I had wished to, care not for the tooling, but for the model architecture instead.

And, I shall endeavor to prove this project by the others I have planned.

## Future

The roadmap gist:

- better type support (rendering & direct f32 capture)
- better API
- JAX support
- desktop capture
- performance optimizations (especially for recording)
- various fixes
- more examples
- better docs
