Metadata-Version: 2.5
Name: gst-python-ml
Version: 1.2.1
Summary: An ML package for GStreamer
Project-URL: Homepage, https://github.com/collabora/gst-python-ml
Author-email: Aaron Boxer <aaron.boxer@collabora.com>
License-File: COPYING
Classifier: License :: OSI Approved :: GNU Lesser General Public License v3 (LGPLv3)
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Requires-Python: >=3.12
Requires-Dist: accelerate>=1.13.0
Requires-Dist: huggingface-hub
Requires-Dist: numpy
Requires-Dist: opencv-python-headless>=4.13.0; python_version >= '3.14'
Requires-Dist: opencv-python-headless>=4.9.0; python_version < '3.14'
Requires-Dist: pillow
Requires-Dist: protobuf
Requires-Dist: pycairo
Requires-Dist: pygobject
Requires-Dist: pyopengl
Requires-Dist: scipy
Requires-Dist: sentencepiece
Requires-Dist: torch>=2.11.0; python_version >= '3.14'
Requires-Dist: torch>=2.7.0; python_version < '3.14'
Requires-Dist: torchvision>=0.22.0; python_version < '3.14'
Requires-Dist: torchvision>=0.26.0; python_version >= '3.14'
Requires-Dist: transformers>=4.44.0; python_version < '3.14'
Requires-Dist: transformers>=5.6.2; python_version >= '3.14'
Provides-Extra: all
Requires-Dist: ai-edge-litert; extra == 'all'
Requires-Dist: bitsandbytes>=0.49.2; extra == 'all'
Requires-Dist: confluent-kafka; extra == 'all'
Requires-Dist: diffusers; extra == 'all'
Requires-Dist: faster-whisper; extra == 'all'
Requires-Dist: insightface; extra == 'all'
Requires-Dist: ipython; extra == 'all'
Requires-Dist: lap; extra == 'all'
Requires-Dist: llama-cpp-python; extra == 'all'
Requires-Dist: onnxruntime; extra == 'all'
Requires-Dist: onnxscript; extra == 'all'
Requires-Dist: openvino>=2024.0; extra == 'all'
Requires-Dist: paho-mqtt; extra == 'all'
Requires-Dist: pysilero; extra == 'all'
Requires-Dist: pysilero-vad; extra == 'all'
Requires-Dist: qwen-vl-utils[decord]>=0.0.8; extra == 'all'
Requires-Dist: skia-python; extra == 'all'
Requires-Dist: soundfile; extra == 'all'
Requires-Dist: spandrel; extra == 'all'
Requires-Dist: speechbrain; extra == 'all'
Requires-Dist: tensorflow>=2.16.0; extra == 'all'
Requires-Dist: tinygrad; extra == 'all'
Requires-Dist: torchaudio>=2.11.0; (python_version >= '3.14') and extra == 'all'
Requires-Dist: torchaudio>=2.7.0; (python_version < '3.14') and extra == 'all'
Requires-Dist: ultralytics; extra == 'all'
Requires-Dist: whisperspeech; extra == 'all'
Provides-Extra: audio
Requires-Dist: ipython; extra == 'audio'
Requires-Dist: soundfile; extra == 'audio'
Requires-Dist: speechbrain; extra == 'audio'
Requires-Dist: torchaudio>=2.11.0; (python_version >= '3.14') and extra == 'audio'
Requires-Dist: torchaudio>=2.7.0; (python_version < '3.14') and extra == 'audio'
Requires-Dist: whisperspeech; extra == 'audio'
Provides-Extra: awq
Requires-Dist: gptqmodel; extra == 'awq'
Provides-Extra: diffusion
Requires-Dist: diffusers; extra == 'diffusion'
Provides-Extra: executorch
Requires-Dist: executorch; (python_version < '3.14') and extra == 'executorch'
Provides-Extra: face
Requires-Dist: insightface; extra == 'face'
Provides-Extra: iree
Requires-Dist: iree-base-compiler; extra == 'iree'
Requires-Dist: iree-base-runtime; extra == 'iree'
Requires-Dist: onnx; extra == 'iree'
Provides-Extra: jax-cpu
Requires-Dist: flax; extra == 'jax-cpu'
Requires-Dist: grain; extra == 'jax-cpu'
Requires-Dist: jax[cpu]; extra == 'jax-cpu'
Requires-Dist: keras-hub>=0.32; extra == 'jax-cpu'
Provides-Extra: jax-gpu
Requires-Dist: flax; extra == 'jax-gpu'
Requires-Dist: grain; extra == 'jax-gpu'
Requires-Dist: jax[cuda12]; extra == 'jax-gpu'
Requires-Dist: keras-hub>=0.32; extra == 'jax-gpu'
Provides-Extra: jax-tpu
Requires-Dist: flax; extra == 'jax-tpu'
Requires-Dist: grain; extra == 'jax-tpu'
Requires-Dist: jax[tpu]; extra == 'jax-tpu'
Requires-Dist: keras-hub>=0.32; extra == 'jax-tpu'
Provides-Extra: kafka
Requires-Dist: confluent-kafka; extra == 'kafka'
Provides-Extra: litert
Requires-Dist: ai-edge-litert; extra == 'litert'
Provides-Extra: llamacpp
Requires-Dist: llama-cpp-python; extra == 'llamacpp'
Provides-Extra: llm
Requires-Dist: bitsandbytes>=0.49.2; extra == 'llm'
Requires-Dist: qwen-vl-utils[decord]>=0.0.8; extra == 'llm'
Provides-Extra: mcp
Requires-Dist: mcp>=2.2.0; extra == 'mcp'
Provides-Extra: mlx
Requires-Dist: mlx-lm; extra == 'mlx'
Requires-Dist: mlx>=0.20.0; extra == 'mlx'
Provides-Extra: mlx-cpu
Requires-Dist: mlx[cpu]>=0.20.0; extra == 'mlx-cpu'
Provides-Extra: mqtt
Requires-Dist: paho-mqtt; extra == 'mqtt'
Provides-Extra: ncnn
Requires-Dist: ncnn; extra == 'ncnn'
Provides-Extra: onnx
Requires-Dist: onnxruntime; extra == 'onnx'
Requires-Dist: onnxscript; extra == 'onnx'
Provides-Extra: onnx-gpu
Requires-Dist: onnxruntime-gpu; extra == 'onnx-gpu'
Requires-Dist: onnxscript; extra == 'onnx-gpu'
Provides-Extra: openvino
Requires-Dist: openvino>=2024.0; extra == 'openvino'
Provides-Extra: rknn
Requires-Dist: rknn-toolkit-lite2>=2.3.2; (platform_machine == 'aarch64' and python_version == '3.12') and extra == 'rknn'
Provides-Extra: skia
Requires-Dist: skia-python; extra == 'skia'
Provides-Extra: superres
Requires-Dist: spandrel; extra == 'superres'
Provides-Extra: tensorflow
Requires-Dist: tensorflow>=2.16.0; extra == 'tensorflow'
Provides-Extra: tinygrad
Requires-Dist: tinygrad; extra == 'tinygrad'
Provides-Extra: tvm
Requires-Dist: apache-tvm; extra == 'tvm'
Provides-Extra: vad
Requires-Dist: faster-whisper; extra == 'vad'
Requires-Dist: pysilero; extra == 'vad'
Requires-Dist: pysilero-vad; extra == 'vad'
Provides-Extra: yolo
Requires-Dist: lap; extra == 'yolo'
Requires-Dist: ultralytics; extra == 'yolo'
Description-Content-Type: text/markdown

# GStreamer Python ML

[![CI](https://github.com/collabora/gst-python-ml/actions/workflows/ci.yml/badge.svg)](https://github.com/collabora/gst-python-ml/actions/workflows/ci.yml)

Run any model with ease inside any GStreamer pipeline.

Pure Python ML elements for upstream GStreamer 1.24 or later: one element per task, or bring your own
model and run it with the base classes and matching engine. 

Write the pipeline in `gst-launch` format and run it with `pyml-launch`, with media backend set to either GStreamer or [glass2glass](https://gitlab.collabora.com/glass2glass/glass2glass).

## Features

- Video
  - object detection and zero-shot detection
  - tracking
  - pose estimation
  - depth estimation
  - zero-shot classification with CLIP or SigLIP
  - segmentation with SAM2
  - OCR
  - face detection and recognition
  - optical flow
  - super-resolution
  - action recognition
  - anomaly detection
  - captioning and vision-language models
  - embeddings and video search
- Audio
  - voice activity detection
  - transcription and translation
  - speech separation
  - text to speech
  - audio classification with CLAP
- Text
  - local and remote LLMs
  - incident digests
  - text to image
- Around them
  - alerts over webhooks and MQTT
  - clip recording
  - metadata sink and replay
  - Kafka sink
  - [MCP server](PIPELINES.md#mcp-server) for agents
- Engines
  - PyTorch by default
  - ONNX Runtime, TensorRT through ONNX Runtime, OpenVINO, LiteRT, TensorFlow
  - Apache TVM, tinygrad, Apple MLX, ExecuTorch, llama.cpp, JAX
  - MiGraphX, IREE, NCNN, Renesas DRP-AI
  - CI runs every engine that installs from PyPI on the CPU and checks its
    output against PyTorch

## Install

Ubuntu 24.04 or later:

```
sudo apt install -y python3-pip python3-venv \
    gstreamer1.0-plugins-base gstreamer1.0-plugins-base-apps \
    gstreamer1.0-plugins-good gstreamer1.0-plugins-bad \
    gir1.2-gst-plugins-bad-1.0 python3-gst-1.0 gstreamer1.0-python3-plugin-loader \
    libcairo2 libcairo2-dev git
```

Then a venv on the system Python, which is the one GStreamer's plugin loader
embeds. `uv sync` installs PyTorch, with CUDA on Linux, and the `yolo` extra
adds the YOLO models the quick start uses.

```
curl -LsSf https://astral.sh/uv/install.sh | sh
git clone https://github.com/collabora/gst-python-ml.git && cd gst-python-ml
uv venv --python /usr/bin/python3 --system-site-packages
uv sync --extra yolo
export GST_PLUGIN_PATH=$PWD/plugins:$GST_PLUGIN_PATH
gst-inspect-1.0 python
```

The last line lists the `pyml_*` elements. [INSTALL.md](INSTALL.md) covers
Fedora, Windows, Docker, the feature extras, the other engines and custom
plugins.

## Quick start

Paths are relative to the checkout directory. Change `device=cuda` to `device=cpu` on a
machine without a GPU.

Track people with YOLO:

```
python pyml-launch.py filesrc location=data/soccer_tracking.mp4 ! decodebin ! videoconvertscale ! video/x-raw,width=640,height=480 ! pyml_yolo model-name=yolo11m device=cuda track=True ! pyml_overlay ! videoconvert ! autovideosink
```

Estimate depth with Depth Anything V2:

```
python pyml-launch.py filesrc location=data/people.mp4 ! decodebin ! videoconvert ! videoscale ! video/x-raw,width=640,height=480 ! pyml_depth model-name=depth-anything/Depth-Anything-V2-Small-hf device=cuda ! videoconvert ! autovideosink sync=false
```

Transcribe Korean speech and translate it to English. Transcripts are logged at
GStreamer info level:

```
GST_DEBUG=python:4 python pyml-launch.py filesrc location=data/air_traffic_korean_with_english.wav ! decodebin ! audioconvert ! pyml_whispertranscribe device=cuda language=ko translate=yes ! fakesink
```

Record a clip around every person detection:

```
python pyml-launch.py filesrc location=data/people.mp4 ! decodebin ! videoconvert ! videoscale ! video/x-raw,width=640,height=480 ! pyml_yolo model-name=yolo11m device=cuda ! pyml_alert rules='{"class":"person","min_score":0.7}' cooldown=8 draw-alert=false ! pyml_alertrecorder location=alert-%s.webm seconds-before=2 seconds-after=3 ! pyml_overlay ! videoconvert ! autovideosink sync=false
```

Replay a recorded detection stream through the tracker and overlay, with no
model and no GPU:

```
python pyml-launch.py filesrc location=data/people.mp4 ! decodebin ! videoconvert ! videoscale ! video/x-raw,width=640,height=480 ! pyml_metareplay location=data/people.jsonl ! pyml_tracker tracker-type=sort ! pyml_overlay ! videoconvert ! autovideosink sync=false
```

[PIPELINES.md](PIPELINES.md) describes every pipeline, one or two per element, how to
run them under glass2glass, and how to run the MCP server.

## Blog post

[![Unleashing gst-python-ml](https://www.collabora.com/assets/images/blog/Collabora-GStPython.jpg)](https://www.collabora.com/news-and-blog/blog/2025/05/12/unleashing-gst-python-ml-analytics-gstreamer-pipelines/)

**[Unleashing gst-python-ml: Python-powered ML analytics for GStreamer pipelines](https://www.collabora.com/news-and-blog/blog/2025/05/12/unleashing-gst-python-ml-analytics-gstreamer-pipelines/)**

Combining GStreamer with machine learning frameworks into video analytics
pipelines, from object detection to video captioning.
