Metadata-Version: 2.4
Name: rasptorch
Version: 2.1.0
Summary: Deep learning library with NumPy and GPU backends (Vulkan, OpenCL, CUDA).
Author: Joshua Ludolf
License-Expression: MIT
Project-URL: Homepage, https://github.com/Joshua-Ludolf/rasptorch
Project-URL: Repository, https://github.com/Joshua-Ludolf/rasptorch
Project-URL: Issues, https://github.com/Joshua-Ludolf/rasptorch/issues
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: click>=8.0
Requires-Dist: numpy>=2.4.4
Requires-Dist: plotly>=7.1.0
Requires-Dist: prompt-toolkit>=3.0
Requires-Dist: pytest>=9.0.3
Requires-Dist: streamlit>=1.56.0
Requires-Dist: torch>=2.11.0
Requires-Dist: twine>=7.0.0
Requires-Dist: vulkan>=1.3.275.1
Provides-Extra: optional
Requires-Dist: requests>=2.33.1; extra == "optional"
Requires-Dist: pygments>=2.20.0; extra == "optional"
Requires-Dist: mako>=1.3.11; extra == "optional"
Provides-Extra: dev
Requires-Dist: build>=1.4.4; extra == "dev"
Requires-Dist: cupy-cuda13x[ctk]>=14.0.1; extra == "dev"
Requires-Dist: pytest>=9.0.3; extra == "dev"
Requires-Dist: twine>=6.2.0; extra == "dev"
Requires-Dist: requests>=2.33.1; extra == "dev"
Requires-Dist: pygments>=2.20.0; extra == "dev"
Requires-Dist: mako>=1.3.11; extra == "dev"
Requires-Dist: vulkan<1.5,>=1.3; extra == "dev"
Requires-Dist: pyopencl>=2026.1.2; extra == "dev"
Requires-Dist: torch>=2.11.0; extra == "dev"
Provides-Extra: vulkan
Requires-Dist: vulkan<1.5,>=1.3; extra == "vulkan"
Provides-Extra: cuda
Requires-Dist: cupy-cuda13x[ctk]>=14.0.1; extra == "cuda"
Provides-Extra: opencl
Requires-Dist: pyopencl>=2026.1.2; extra == "opencl"
Provides-Extra: torch
Requires-Dist: torch>=2.11.0; extra == "torch"
Provides-Extra: gpu
Requires-Dist: vulkan<1.5,>=1.3; extra == "gpu"
Requires-Dist: pyopencl>=2026.1.2; extra == "gpu"
Dynamic: license-file

# 🍓🎇 rasptorch

rasptorch is an experimental deep learning library inspired by PyTorch, built with a singular focus: **making complex neural networks practical and efficient to run on resource-constrained hardware like the Raspberry Pi 5, by leveraging its GPU capabilities via Vulkan.**

---

### ✨ Core Concepts

The library operates on a multi-layered architecture to maximize hardware utilization:

1.  **CPU Backend (Software):** Uses a pure NumPy-backed autograd engine and `nn` module for reliable computation when GPU acceleration is unavailable.
2.  **GPU Backend (Hardware):** Features an experimental **Vulkan backend** for high-speed tensor operations (elementwise math, matmul, reductions) directly on the Pi 5's GPU — and on discrete GPUs via DEVICE_LOCAL VRAM residency plus a register-blocked tiled GEMM kernel.
3.  **Interface:** Provides a streamlined CLI/Streamlit UI for interactive model building, training, persistence, and inspection.

The Vulkan path relies on real compute shaders compiled to SPIR-V, giving deep control over the underlying hardware.

### 🔌 Backend Abstraction (Connectable Backends)

rasptorch now exposes a backend abstraction API so compute backends can be registered and connected at runtime:

```python
import rasptorch

# Inspect availability
print(rasptorch.available_backends())  # {'cpu': True, 'vulkan': ..., 'opencl': ..., 'cuda': ...}

# Try to connect a backend (falls back to CPU in non-strict mode)
active = rasptorch.connect_backend("vulkan", strict=False)
print(active.name)
```

Built-in backend adapters:
- `numpy` (NumPy adapter; internal key: `cpu`) - Pure NumPy autograd
- `vulkan` (rasptorch Vulkan kernels, with optional CPU fallback) - **Optimized for Raspberry Pi 4/5** ⚡ and accelerated further on discrete GPUs (VRAM-resident buffers, `matmul_blk_tiled` GEMM)
- `opencl` (pyopencl when available, optional CPU fallback)
- `cuda` (CuPy when available, with PyTorch CUDA fallback, optional CPU fallback)

#### OpenCL on Raspberry Pi 5 (Vulkan GPU via `clvk`)

Raspberry Pi 5’s GPU stack is Vulkan-first. If you want the `opencl` backend to run on the **Vulkan GPU**, you typically need an OpenCL ICD that targets Vulkan (commonly `clvk`). Once installed, rasptorch’s OpenCL backend will prefer GPU-like devices automatically.

Useful environment variables for selecting the right OpenCL runtime/device:
- `RASPTORCH_OPENCL_PREFER`: `gpu` (default) or `clvk`/`vulkan` to strongly prefer a Vulkan-backed OpenCL platform
- `RASPTORCH_OPENCL_PLATFORM`: substring match against platform name/vendor
- `RASPTORCH_OPENCL_DEVICE`: substring match against device name/vendor
- `RASPTORCH_OPENCL_TILE`: matmul tiling (default `16`); the backend will auto-reduce if the device can’t support it

Example (force clvk if multiple OpenCL platforms exist):
```bash
RASPTORCH_OPENCL_PREFER=clvk uv run rasptorch backend connect opencl --strict
```

CLI helpers:
```bash
rasptorch backend list
rasptorch backend connect numpy
rasptorch backend connect vulkan --strict
# Benchmark with auto-tuned Vulkan kernel and submission batching
rasptorch --json backend benchmark --backends numpy,vulkan,cuda --size 2048 --iterations 100 --warmup 20 --vulkan-kernel auto --vulkan-autotune-submit --seed 42
```

> **Note:** User-facing CLI/UI labels the CPU backend as **`numpy`**.
> Vulkan benchmark mode uses resident buffers (upload once, repeated on-device matmul, download once).
> On discrete GPUs, buffers now prefer **DEVICE_LOCAL (VRAM) memory** — tile reads/writes never cross PCIe;
> unified-memory devices (Raspberry Pi 4/5, integrated GPUs) keep the zero-copy host-mapped path automatically.
> Override with `RASPTORCH_VULKAN_DEVICE_LOCAL=1` (force VRAM) or `=0` (force host-mapped).
> **Performance:** auto mode now reaches ~8,800 GFLOPS at 2048² on a discrete NVIDIA GPU via `matmul_blk_tiled`
> (2× faster than the next-best kernel), and beats the CUDA backend at every size from 256² up.
> `--vulkan-kernel auto` probes `matmul_blk_tiled`, `matmul_vec4_32x32_tiled`, `matmul_vec4_wide_tiled`,
> `matmul_vec4_tiled`, `matmul`, `matmul_vec4`, `matmul_tiled`, `matmul_a_bt`, and `matmul_a_bt_tiled`
> (when available) with one untimed warm-up dispatch each, then keeps the fastest stable path.
> For a fair GPU-vs-GPU comparison, add `--cuda-resident` so CUDA also skips per-iteration host transfers.
> If Vulkan hits `VkErrorDeviceLost`, lower `--vulkan-submit-every` (for example, `4` or `1`) or use auto-tuning.
> Optimizations: Command buffer batching, VRAM residency + staging copies, auto kernel selection.

### 📚 What's Included (Core Features)

*   **Tensor Operations:** Support for elementwise math, matrix multiplication (`matmul`), reductions, indexing, reshaping, stacking, and broadcasting.
*   **Layers:** Includes standard neural network blocks: `Linear`, `MLP`, `CNN`, `GRU`, `Transformer`, normalization layers, activations, pooling, embeddings, and attention.
*   **Training Tools:** Full suite of tools including optimizers (`SGD`), learning-rate schedulers, gradient clipping, and regularization helpers.
*   **Persistence:** Ability to save and load checkpoint weights without needing the full `torch` dependency.
*   **Interfaces:** CLI (`rasptorch chat`) and Streamlit UI (`rasptorch ui`).

### 🚀 Getting Started

#### 1. Installation

**A. Basic Install (CPU Only):**
To get the core library components running on the CPU:
```bash
pip install rasptorch
```

**B. Development Install (Full Capability):**
For local development and access to all potential backends:
```bash
pip install -e ".[dev]"
```

**C. GPU Backend Dependencies:**
To use the GPU backends, install the extra for the backend(s) you need:
```bash
# Prerequisites for Vulkan backend (Raspberry Pi 4/5 & Linux)
sudo apt update
sudo apt install -y glslc

# For Windows, install the Vulkan SDK from LunarG:
./VulkanSDK-Installer.exe --accept-licenses --default-answer --confirm-command install

# Then install rasptorch with GPU support:
pip install rasptorch[gpu]
```

You can also install only the backend(s) you need instead of the entire `gpu` extra. Examples:

```bash
# Vulkan-only (recommended for Raspberry Pi 4/5)
pip install rasptorch[vulkan]

# OpenCL-only
pip install rasptorch[opencl]

# CUDA-only for NVIDIA systems with CUDA 13
# The CuPy CUDA runtime libraries are installed by the extra when they are
# not present locally.
pip install rasptorch[cuda]

# uv equivalents
uv sync --extra opencl
uv sync --extra cuda
uv sync --extra cuda --extra opencl

# Vulkan + OpenCL
pip install "rasptorch[vulkan,opencl]"
```

The CUDA extra uses the prebuilt `cupy-cuda13x` wheel with CuPy's `ctk`
runtime extra. Do not run `uv add cupy`: the generic `cupy` package may resolve
to a source distribution and require a complete local CUDA build toolchain.

`uv lock` does not accept `--extra`; it resolves the project and declared
extras. Select extras when installing or running commands with `uv sync
--extra ...` or `uv run --extra ...`.

#### 2. Quick Run Examples

**Start the Interactive Shell:**
```bash
uv run rasptorch chat
```

**Launch the Web UI:**
```bash
uv run rasptorch ui
```
*(This will usually open at `http://localhost:8501`)*

When using `uv`, optional backend dependencies are not installed by a plain
`uv sync`. Use the corresponding `--extra` flags, for example:

```bash
uv sync --extra cuda --extra opencl
uv run --extra cuda --extra opencl rasptorch backend benchmark
```

**Streamlit Community Cloud:**

The repository includes [`packages.txt`](packages.txt), which installs Vulkan
runtime libraries and Mesa's software Vulkan driver on Streamlit Community
Cloud. This can make the Vulkan backend available through CPU-based software
rendering (lavapipe), but it does not provide Raspberry Pi VideoCore,
NVIDIA CUDA, or other hardware GPU access.

The hosted app keeps `auto` backend selection enabled:

1. Use Vulkan if a compatible software Vulkan device is available.
2. Fall back to the NumPy backend when Vulkan is unavailable.

Open the **Vulkan diagnostics** expander in the sidebar to inspect the
deployment platform, Vulkan availability, active backend, and any
initialization error. Software Vulkan may be slower than NumPy for small
workloads; use a Raspberry Pi 4/5 or another machine with a supported Vulkan
device for hardware acceleration.

**Viewing Help:**
To see all available CLI subcommands:
```bash
uv run rasptorch --help
```

---

### ⚙️ Execution Modes & Workflows

The `main.py` script controls the operational mode:

*   **`cpu`**: Pure NumPy autograd execution on the CPU.
*   **`gpu`**: Executes the training loop explicitly using the Vulkan backend kernels.
*   **`gpu-autograd`**: An experimental mode for tracing gradients across the GPU pipeline.

**Example Training Command:**
```bash
uv run main.py --device gpu --epochs 50 --batch-size 32 --lr 0.01 --seed 0
```

> **Reproducibility tip:** use the same `--seed` value when comparing CPU and GPU runs; otherwise losses can differ due to different random initialization/shuffle order.

---

### 📊 Benchmarks

rasptorch provides a built-in benchmark tool for comparing backend performance on matrix multiplication:

**Quick Benchmark (Single Size):**
```bash
# Benchmark with default settings (2048x2048 matmul, 100 iterations)
uv run rasptorch backend benchmark

# Benchmark with custom size and multiple backends
uv run rasptorch --json backend benchmark --backends numpy,vulkan,cuda --size 2048 --iterations 100 --warmup 20 --seed 42
```

On Windows, if the console uses a legacy code page and cannot display the
benchmark's checkmark/cross characters, enable UTF-8 output:

```powershell
$env:PYTHONIOENCODING = "utf-8"
uv run --extra cuda --extra opencl rasptorch backend benchmark
```

**Performance Results (Windows 11, NVIDIA GeForce RTX 3070 Ti):**

Measured backend scaling with (single pass; individual runs vary by a few percent due to GPU clock variance):
```bash
uv run rasptorch --json backend benchmark --backends numpy,opencl,vulkan,cuda --vulkan-kernel auto --vulkan-submit-every 8 --iterations 5 --warmup 1 --seed 42
```

Auto mode selects `matmul_blk_tiled` for these sizes, and buffers are VRAM-resident (DEVICE_LOCAL), so Vulkan tile traffic stays on-card.

#### 256×256 → 4096×4096 Scaling
| Backend  | 256×256 (GFLOPS) | 512×512 (GFLOPS) | 1024×1024 (GFLOPS) | 2048×2048 (GFLOPS) | 4096×4096 (GFLOPS) |
|----------|------------------|------------------|-------------------|-------------------|-------------------|
| NumPy | 174.07 | 373.64 | 770.34 | 925.75 | 1001.66 |
| OpenCL | 27.03 | 38.34 | 35.94 | 33.98 | 18.66 |
| Vulkan (`auto` → `matmul_blk_tiled`, resident) | **990.98** | **4068.44** | **7858.76** | **8101.50** | **8476.71** |
| CUDA (cuBLAS, per-iteration transfers) | 142.66 | 1433.61 | 1142.64 | 1585.06 | 2956.98 |

Vulkan leads at every size from 256 up. The CUDA column pays one host→device + download + sync per iteration; add `--cuda-resident`
(so cuBLAS also uploads once and runs N GEMMs before a single sync) for the fair GPU-side comparison:

| Size | Vulkan (resident) | CUDA (`--cuda-resident`) |
|------|-------------------|--------------------------|
| 2048×2048 | 9029.7 | 8587.1 |
| 4096×4096 | 8146.0 | 12551.9 |

Resident-vs-resident, Vulkan edges out cuBLAS at 2048 and cuBLAS leads by ~1.5× at 4096 — the remaining headroom for Vulkan is
cooperative-matrix extensions (`VK_KHR_cooperative_matrix`) and wider per-thread register tiles (see [VULKAN_OPTIMIZATION.md](VULKAN_OPTIMIZATION.md)).

#### Vulkan Kernel Comparison (2048×2048)
Measured with:
```bash
uv run rasptorch --json backend benchmark --backends vulkan --vulkan-submit-every 8 --size 2048 --iterations 5 --warmup 1 --seed 42 --vulkan-kernel <kernel>
```

| Vulkan Kernel | GFLOPS | Status |
|---------------|--------|--------|
| `matmul_blk_tiled` | **8175.7** | ok (auto-selected) |
| `matmul_vec4_wide_tiled` | 4043.3 | ok |
| `matmul_vec4_32x32_tiled` | 3264.6 | ok |
| `matmul_vec4_tiled` | 2389.8 | ok |
| `matmul` | 2385.3 | ok |
| `matmul_tiled` | 2235.6 | ok |
| `matmul_a_bt_tiled` | 2207.9 | ok |
| `matmul_vec4` | 1777.3 | ok |
| `matmul_a_bt` | 450.2 | ok |

`matmul_blk_tiled` (64×64 output tiles, 4×4 register block per thread, double-buffered shared k-tiles, ~16 FLOP/byte of global traffic)
is roughly 2× the next-best kernel and is chosen automatically for `min(m, n, k) >= 512` when the device provides ≥32 KiB of shared memory.

**Vulkan Kernel Selection:**
The `--vulkan-kernel auto` flag intelligently probes available kernels (one untimed warm-up dispatch each, then timed probes):
- `matmul` - Basic single-threaded implementation
- `matmul_tiled` - Shared-memory tiled implementation
- `matmul_vec4` - SIMD-style vec4 operations
- `matmul_vec4_tiled` - vec4 plus shared-memory tiling
- `matmul_vec4_wide_tiled` - vec4 plus wider output tiling for better reuse
- `matmul_vec4_32x32_tiled` - 32×32 tiles (2×2 outputs per thread), ~8 FLOP/byte
- `matmul_blk_tiled` - 64×64 tiles (4×4 per thread), double-buffered, ~16 FLOP/byte (fastest on discrete GPUs)
- `matmul_a_bt` - Matrix transpose optimization (for A @ B.T)
- `matmul_a_bt_tiled` - Tiled transpose optimization

**Advanced Tuning:**
```bash
# Auto-tune both kernel AND submission batching strategy
uv run rasptorch --json backend benchmark --backends vulkan --size 2048 \
  --iterations 100 --warmup 20 \
  --vulkan-kernel auto \
  --vulkan-autotune-submit \
  --seed 42

# Manual kernel selection with custom batch submission
uv run rasptorch --json backend benchmark --backends vulkan \
  --vulkan-kernel matmul_a_bt_tiled \
  --vulkan-submit-every 4 \
  --size 2048 --iterations 100

# Fair GPU-vs-GPU comparison: CUDA also runs resident (upload once, sync once)
uv run rasptorch --json backend benchmark --backends vulkan,cuda \
  --size 4096 --iterations 5 --cuda-resident
```

**Large Workload Guidance:**
- For 4096x4096 matmul, use the default `--vulkan-kernel auto` (picks `matmul_blk_tiled`); pinning `matmul_vec4_wide_tiled` is no longer needed.
- Buffers are DEVICE_LOCAL on discrete GPUs, so large sizes no longer degrade the way host-mapped buffers did.
- If a custom Vulkan kernel triggers `VkErrorDeviceLost`, reduce `--vulkan-submit-every` first.
- The benchmark path uses resident buffers, so the large-size slowdown is kernel quality rather than repeated host-device transfers.

**Output Format:**
Results are provided in JSON format (with `--json` flag) including:
- `status`: "ok" or "unavailable"
- `elapsed_seconds`: Total benchmark time
- `iterations_per_second`: Throughput metric
- `estimated_gflops`: Floating-point performance
- `checksum`: Verification result
- `kernel`: Selected kernel name (for auto mode)
- `submit_every`: Submission batch size (for Vulkan)
- `mode`: "resident" when the run uploads once and syncs at the end (Vulkan always; CUDA with `--cuda-resident`)

**Optimization Tips:**
- Use `--vulkan-autotune-submit` for best results (probes kernel + batch combinations)
- If you see `VkErrorDeviceLost`, reduce `--vulkan-submit-every` (try `4` or `1`)
- Larger problem sizes better amortize GPU setup overhead
- Command buffer batching (`--vulkan-submit-every`) balances latency and throughput
- On discrete GPUs buffers are DEVICE_LOCAL (VRAM) by default; set `RASPTORCH_VULKAN_DEVICE_LOCAL=0` to force the zero-copy host-mapped path (or `=1` to force VRAM residency when a device offers it)
- Compute+readback timing includes transfer overhead. On Raspberry Pi eGPU setups over PCIe, this overhead can outweigh compute gains, so CPU can be faster for smaller workloads.

For detailed optimization guide, see [VULKAN_OPTIMIZATION.md](VULKAN_OPTIMIZATION.md).

---

### 🧠 Advanced Topics

#### 1. Tensor Operations
Basic tensor math is performed via:
```bash
# Create tensors
uv run rasptorch tensor random --shape 2,3,4
uv run rasptorch tensor ones --shape 5,10
```
The results show the low-level tensor capabilities.

#### 2. Model Definition
Models are defined using structured commands:
```bash
# Simple MLP
uv run rasptorch model mlp --layers "64,32,16,2"
# Complex CNN
uv run rasptorch model cnn --in-channels 3 --out-channels "32,64,128"
# Computer Vision model (Conv2d -> Flatten -> Linear)
uv run rasptorch model cv --in-channels 3 --out-channels "32,64" --num-classes 10 --input-hw 32
# JEV (decision) model (MLP outputting decision vector)
uv run rasptorch model jev --input-size 10 --hidden-sizes "32,16" --num-decisions 2
```
Managing the lifecycle:
```bash
uv run rasptorch model list
uv run rasptorch model save --model-id <id> --path model.pth
```

---

### 🩹 Troubleshooting & Best Practices

1.  **Performance:** The fastest paths are those that keep the computation entirely on the GPU and minimize data transfer across the PCIe bus.
2.  **Fallback:** If GPU operations fail due to driver issues, the system gracefully falls back to the CPU NumPy path, but performance will suffer.
3.  **Advanced Use:** For understanding the deep dive into custom kernel optimization, please refer to the source code in the `rasptorch/gpu_demo.py` and `rasptorch/main.py` scripts.
