Metadata-Version: 2.5
Name: fitlab
Version: 0.1.1
Summary: Will it run on my setup? Fit-check + benchmark wizard for local LLMs on your GPU — weekly-updated model registry.
Project-URL: Homepage, https://github.com/ruslanmv/fitlab
Project-URL: Source, https://github.com/ruslanmv/fitlab
Project-URL: Issues, https://github.com/ruslanmv/fitlab/issues
Author: Ruslan Magana Vsevolodovna
License: Apache-2.0
License-File: LICENSE
Keywords: benchmark,colab,gpu,llm,local-ai,ollama,vram
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Requires-Dist: rich>=13
Provides-Extra: dev
Requires-Dist: beautifulsoup4>=4.12; extra == 'dev'
Requires-Dist: build>=1.2; extra == 'dev'
Requires-Dist: jsonschema>=4.21; extra == 'dev'
Requires-Dist: pyyaml>=6.0; extra == 'dev'
Requires-Dist: ruff>=0.6; extra == 'dev'
Requires-Dist: twine>=5.1; extra == 'dev'
Provides-Extra: maintainer
Requires-Dist: beautifulsoup4>=4.12; extra == 'maintainer'
Requires-Dist: jsonschema>=4.21; extra == 'maintainer'
Requires-Dist: pyyaml>=6.0; extra == 'maintainer'
Description-Content-Type: text/markdown

<p align="center">
  <img src="site/logo.svg" width="96" alt="FitLab logo" />
</p>

<h1 align="center">LLM FitLab</h1>

<p align="center">
  <strong>An evidence-based compatibility radar for running open LLMs on the hardware you actually have.</strong>
</p>

<p align="center">
  <a href="https://github.com/ruslanmv/fitlab/actions/workflows/weekly-sync.yml"><img alt="Weekly sync" src="https://github.com/ruslanmv/fitlab/actions/workflows/weekly-sync.yml/badge.svg" /></a>
  <a href="https://github.com/ruslanmv/fitlab/actions/workflows/weekly-benchmark.yml"><img alt="Weekly benchmark" src="https://github.com/ruslanmv/fitlab/actions/workflows/weekly-benchmark.yml/badge.svg" /></a>
  <img alt="Python" src="https://img.shields.io/badge/python-3.10%2B-blue" />
  <img alt="Code license" src="https://img.shields.io/badge/code-Apache--2.0-green" />
  <img alt="Data license" src="https://img.shields.io/badge/data-CC--BY--4.0-green" />
  <img alt="Status" src="https://img.shields.io/badge/status-beta-orange" />
</p>

---

FitLab answers the three questions every engineer asks before pulling a local model — and it
answers them with **regenerated data, not a hand-maintained list**. Every claim on the page is
produced by a script, validated against a JSON Schema, and merged through a reviewed pull request.

| Lane | Question it answers | Method | Confidence |
|---|---|---|---|
| **FITS** | Will this model fit my GPU? | Deterministic VRAM arithmetic from the model's own `config.json` — weights + KV cache + runtime overhead, per quantization and context length | Computed, reproducible |
| **RUNS** | How fast is it, really? | A real benchmark on a free T4 16 GB GPU every week, plus community-submitted runs on real consumer hardware | Measured, with raw data retained |
| **PLUGS** | Does it work with my stack? | Weekly end-to-end probes installing the **latest released** Ollama, [OllaBridge](https://github.com/ruslanmv/ollabridge), LangGraph, CrewAI, Langflow, Langfuse and DeepAgents, then issuing a genuine request through the whole chain | Verified, with exact versions recorded |
| **HOSTED** | Which hosted models can HomePilot run today, and which do people like most? | Hugging Face Hub data (published weekly; HomePilot reads it live): models a provider serves *live* through Inference Providers for chat, vision, images and video, ranked by likes, downloads, trend and freshness | Observed, with raw counts kept |

**Source of truth:** [`data/registry.json`](data/) · **Design rationale:** [`ARCHITECTURE.md`](ARCHITECTURE.md)

## Table of contents

- [Why FitLab exists](#why-fitlab-exists)
- [Installation](#installation)
- [Quick start](#quick-start)
- [CLI reference](#cli-reference)
- [Configuration](#configuration)
- [Running FitLab inside an organization](#running-fitlab-inside-an-organization)
- [Data contract](#data-contract)
- [How the numbers are produced](#how-the-numbers-are-produced)
- [Media lane](#media-lane)
- [Using FitLab from HomePilot](#using-fitlab-from-homepilot)
- [Secrets](#secrets)
- [Automation](#automation)
- [Development](#development)
- [Contributing](#contributing)
- [Security](#security)
- [Support and versioning](#support-and-versioning)
- [Related projects](#related-projects)
- [License](#license)

## Why FitLab exists

Curated "best model for X GB" lists go stale within a quarter. FitLab is built so that the
**data outlives any individual model**:

1. **Nothing is hand-ranked.** A weekly job pulls the leading models per Hugging Face pipeline tag
   through the public API, runs each one through the FITS engine, scores them with a versioned
   formula, and opens a review PR. Humans review; automation refreshes.
2. **Claims are tested, not copied.** One model per week receives a real benchmark on a free
   T4 16 GB — the same GPU class the Colab free tier allocates — so the numbers transfer directly.
3. **The community is the sensor network.** Anyone can run the
   [one-click Colab notebook](colab/benchmark_colab.ipynb) or the local benchmark script; the result
   is schema-validated and sanity-bounded by CI before it is merged, with attribution.
4. **Writes are gated.** Every JSON is validated against [`data/schema/`](data/schema/), and all
   automation writes through pull requests — never directly to the default branch.
5. **Compatibility is re-verified, not assumed.** A green PLUGS cell means the integration worked
   *this week*, at version pins recorded alongside the result.

## Installation

**Requirements:** Python 3.10 or newer. [Ollama](https://ollama.com) is required only for the
`bench` command and the wizard's benchmarking step — fit checks need no GPU and no runtime.

FitLab is published to PyPI from tagged GitHub releases via
[Trusted Publishing](https://docs.pypi.org/trusted-publishers/):

```bash
pip install fitlab
```

Until the first release lands on PyPI, install from source. The Makefile bootstraps
[uv](https://docs.astral.sh/uv/), creates `./.venv` and installs everything:

```bash
git clone https://github.com/ruslanmv/fitlab && cd fitlab
make install
make run
```

Or without make, using any tool that reads `pyproject.toml`:

```bash
pip install .                     # or: uv sync
fitlab --version
```

The model list refreshes itself at runtime: the CLI fetches the weekly `registry.json`, caches it
for 24 hours, and falls back to a bundled seed when offline. Upgrading the package is not required
to see new models.

## Quick start

Run the wizard with no arguments — it detects your hardware, shows what fits, and offers to
benchmark your selection:

```bash
fitlab
```

```text
// llm fitlab — fit-check console · v0.1.0
your rig: NVIDIA GeForce RTX 3060 · 12.0 GB VRAM · ~360 GB/s
registry 2026-08-24 · 22 models · live (weekly)

  1 homepilot-12gb — HomePilot — most liked chat models on 12 GB
  2 homepilot-16gb — HomePilot — most liked chat models on 16 GB
  3 homepilot-vision — HomePilot — most liked vision models on 16 GB
  4 local-12gb — Top 10 — local inference, low-end GPU (12 GB)
  5 colab-free — Top 10 — Colab free / Kaggle (T4 16 GB)
  6 multimodal — Top multimodal (vision-language) for 12–16 GB
  7 all        — everything, deduplicated
? Category 1-4 (1)
```

Or check a specific GPU profile without running anything:

```bash
fitlab check --gpu rtx3060-12 --category local-12gb --limit 6
```

```text
                  fit check · RTX 3060 12GB · Q4_K_M · 8K ctx
┏━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━┓
┃ # ┃ model             ┃ params ┃ est GB ┃ verdict ┃ ~tok/s ┃ ollama tag      ┃
┡━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━┩
│ 1 │ Qwen/Qwen3-8B     │ 8.2B   │ 7.2    │ FITS    │ ~44.8  │ qwen3:8b        │
│ 2 │ ibm-granite/gran… │ 8.1B   │ 7.3    │ FITS    │ ~45.4  │ granite3.3:8b   │
│ 3 │ meta-llama/Llama… │ 8.0B   │ 7.0    │ FITS    │ ~45.9  │ llama3.1:8b     │
│ 4 │ mistralai/Minist… │ 8.0B   │ 7.1    │ FITS    │ ~45.9  │ ministral-8b    │
│ 5 │ Qwen/Qwen3-14B    │ 14.8B  │ 11.5   │ TIGHT   │ ~24.8  │ qwen3:14b       │
│ 6 │ deepseek-ai/Deep… │ 14.8B  │ 11.8   │ TIGHT   │ ~24.8  │ deepseek-r1:14b │
└───┴───────────────────┴────────┴────────┴─────────┴────────┴─────────────────┘
```

After a benchmark, the CLI prints a prefilled submission link so your result can join the public
dataset. Submitting is entirely optional; nothing is transmitted automatically.

## CLI reference

| Command | Purpose |
|---|---|
| `fitlab` | Interactive wizard: detect → fit table → select → benchmark |
| `fitlab detect` | Print detected hardware (NVIDIA via `nvidia-smi`, Apple Silicon via `sysctl`, CPU fallback) |
| `fitlab check` | Fit table for a GPU profile. No model download, no GPU required |
| `fitlab bench` | Benchmark one model non-interactively; writes a schema-valid JSON result |
| `fitlab update` | Force a registry refresh, bypassing the 24-hour cache |
| `fitlab --version` | Print the installed version |

**`fitlab check`**

| Flag | Default | Values |
|---|---|---|
| `--gpu` | `auto` | `auto`, `rtx3060-12`, `t4-16`, `rtx4070-12`, `rtx4060ti-16`, `apple-m-16`, `cpu-only` |
| `--quant` | `Q4_K_M` | `Q2_K`, `Q3_K_M`, `Q4_0`, `Q4_K_M`, `Q5_K_M`, `Q6_K`, `Q8_0`, `MXFP4`, `F16` |
| `--ctx` | `8192` | `4096`, `8192`, `16384`, `32768` |
| `--category` | `all` | A registry category id, or `all` |
| `--limit` | unset | Maximum rows to print |

**`fitlab bench`**

| Flag | Default | Notes |
|---|---|---|
| `--model` | *required* | Ollama tag (`qwen3:8b`) or registry slug (`qwen3-8b`) |
| `--reps` | `3` | Repetitions; `gen_tps` is reported as the median |
| `--context` | `4096` | Context length for the run |
| `--out` | `fitlab-results` | Output directory for the result JSON |
| `--source` | `community-local` | `ci-kaggle`, `ci-actions-cpu`, `community-colab`, `community-local`, `maintainer` |
| `--submitter` | empty | GitHub handle recorded for attribution |
| `--dry-run` | off | Print the fit verdict and execution plan without running |

Benchmarks are seeded and fixed-length (`fitlab-v1-256` prompt, `num_predict 256`, `seed 42`) so
runs are comparable across machines and across time.

## Configuration

All configuration is by environment variable — no config file, no implicit state beyond the cache.

| Variable | Default | Purpose |
|---|---|---|
| `FITLAB_REGISTRY_URL` | `registry-latest`, then `master` | Point the CLI at a different registry — an internal mirror, a pinned commit, or a `file://` path. Set, it is tried first and the defaults still follow. |
| `FITLAB_CACHE` | `~/.cache/fitlab` | Cache directory for the fetched registry (24-hour TTL) |
| `FITLAB_REPO` | `ruslanmv/fitlab` | Repository used to build benchmark submission links |
| `OLLABRIDGE_URL` | `http://127.0.0.1:11435/v1` | Gateway endpoint used by the PLUGS compatibility probes |
| `KAGGLE_USERNAME`, `KAGGLE_KEY` | unset | Required only by the weekly GPU benchmark workflow |

Network behaviour is deliberately narrow and predictable:

- `check`, `detect` and the wizard's fit stage perform one HTTPS GET per published ref until one
  answers (normally the first).
- Nothing is uploaded. Benchmark results are written to local disk; sharing is a manual step.
- With no network, the CLI degrades in a defined order: fresh cache → stale cache → bundled seed,
  and the active source is printed on every run. Until 2026-09 the default URL was an unreplaced
  placeholder (`OWNER/llm-fitlab@main`) that 404s, so every installed copy quietly served the
  bundled seed and reported `bundled-seed`; if you see that on a networked machine, upgrade.

## Running FitLab inside an organization

**Internal registry mirror.** Fit verdicts and rankings are just data. To decouple from the public
repository — for change control, or to publish an approved-model list — host your own
`registry.json` and point clients at it:

```bash
export FITLAB_REGISTRY_URL="https://models.internal.example.com/fitlab/registry.json"
fitlab check --gpu rtx4060ti-16
```

**Pinned, reproducible data.** Because every registry carries `schema_version`, `ranking_version`
and `generated_at`, you can pin clients to a specific commit and get byte-identical results:

```bash
export FITLAB_REGISTRY_URL="https://raw.githubusercontent.com/ruslanmv/fitlab/<commit-sha>/data/registry.json"
```

**Air-gapped use.** Ship a registry file with your image and reference it directly; no egress is
required for fit checks:

```bash
export FITLAB_REGISTRY_URL="file:///opt/fitlab/registry.json"
```

**Custom hardware fleet.** Add your GPUs to [`data/hardware.yaml`](data/hardware.yaml) — including
capability flags such as `bf16`, `flash_attn2` and `marlin` — and regenerate the registry to get a
fit matrix for your own fleet rather than the public reference profiles.

**Extending the compatibility matrix.** Add a component to [`data/stack.yaml`](data/stack.yaml)
with a `probe` that exercises it end to end, and it will be verified on the same weekly cadence as
the rest of the stack.

## Data contract

Everything downstream depends on `data/`, and only on `data/`. The site is a renderer; the CLI is a
client. Both read the same documents.

```
data/
  registry.json              generated — the published source of truth (do not hand-edit)
  ollama_catalog.json        generated — the Ollama library index, every model and tag
  models.seed.yaml           human-curated seeds and editorial notes
  media.seed.yaml            image + video models: declared VRAM (with source), workflow cost, install ids
  media_registry.json        generated — the media lane's published document (do not hand-edit)
  hardware.yaml              GPU profiles: VRAM, bandwidth, architecture capability flags
  stack.yaml                 ecosystem components and version-discovery rules
  benchmarks/                append-only measured results, one JSON per run
  schema/
    model.schema.json        registry model entries
    benchmark.schema.json    benchmark result submissions
    media.schema.json        the media registry document
```

`registry.json` top-level keys: `schema_version`, `generated_at`, `ranking_version`, `models`,
`categories`, `benchmarks_latest`, `compat`, `calibration`.

`calibration` records how the bandwidth estimate compares with the measurements collected so
far, rather than tuning the constant to fit them. As of 2026-09-12 it reads **2.45×** over two
pairs (`qwen3:0.6b` on an Actions CPU runner, `qwen3:8b` on a Kaggle P100) — the estimate is a
consistently optimistic ceiling, and the leaderboard says so. Two points is not a refit; it is
a warning label that gets sharper each week.

### Where the published registry lives

| Ref | Written by | Use it for |
|---|---|---|
| [`registry-latest`](https://github.com/ruslanmv/fitlab/tree/registry-latest) | `weekly-sync`, every run, no review needed | The freshest data. What the CLI and the leaderboard read first. |
| `master` | a merged review PR | The reviewed copy, and the fallback when `registry-latest` is missing. |
| bundled seed | a release | Offline fallback inside the installed package. |

Publishing to a branch is deliberate: a review gate nobody can open is a freeze. Every `weekly-sync`
run between 2026-08-31 and 2026-09-09 rebuilt the registry correctly and then failed at PR creation
with *"GitHub Actions is not permitted to create or approve pull requests"*, so the published data
sat at its seed date while the job reported failure. Data refresh is not code — the branch publishes
itself, and the PR still carries it into `master` for review. To enable the PR too, turn on **Allow
GitHub Actions to create and approve pull requests** in Settings → Actions → General.

Stability guarantees:

- Every document carries `schema_version`, so historical data stays parseable indefinitely.
- `data/benchmarks/` is append-only; results are never rewritten or deleted.
- Models withdrawn from Hugging Face are marked `stale` and hidden after 60 days — never removed.
- Fit values are labeled `estimated` until a real benchmark upgrades them to `measured`.
- `ollama_tag` is only ever a tag that exists in `ollama_catalog.json`. `make validate` fails the
  build on a tag that has left the Ollama library, so the page never tells you to run something
  that is not there.
- Every benchmark result must attach to a registry model by `model_id` or `ollama_tag`; an
  unattributable result fails `make validate` rather than being silently dropped on join.
- One Ollama tag belongs to one model. A speech or quantised variant that resolves onto a text
  model's tag has the tag cleared, with a warning in the sync log.
- `license` and `capabilities` on an auto-discovered entry are derived: `license:*` and
  capability tags come from the Hugging Face model card, the rest from repo naming
  (`-Coder-`, `-VL-`, `R1`, `-A3B`). Curated seed values always win.

## How the numbers are produced

**FITS** is arithmetic, not opinion:

```
est_vram = weights(params × bpw / 8 × 1.03)
         + kv_cache(2 × layers × kv_heads × head_dim × context × 2 bytes)
         + runtime_overhead
```

Verdicts are thresholds against the profile's VRAM: `fits` at ≤ 92%, `tight` at ≤ 100%,
`offload` at ≤ 160% (partial GPU offload viable), otherwise `no`. Mixture-of-experts models are
sized by *total* parameters for memory and by *active* parameters for speed; the registry stores
both.

**RUNS** measures load time, time-to-first-token, generation tokens/second and peak VRAM through
Ollama's `/api/generate` timing fields — chosen because Ollama is what the target audience actually
runs. The weekly rotation is `week_number % queue_length`, so every model is measured and
periodically re-measured.

**Ranking** is a transparent, versioned sort (currently `ranking_version` 1.2, as computed
in [`scripts/sync_hf.py`](scripts/sync_hf.py)). The published order is Hugging Face likes among
models that fit the reference GPU. The score breaks ties:

```
score = 0.30·fit + 0.20·capability + 0.35·popularity + 0.10·speed + 0.05·plugs
popularity = 0.75·log likes + 0.25·log downloads
```

Likes are people's votes. Downloads only corroborate a model that has not been liked yet. `capability`
is log-scaled parameter count, and `speed` uses a measured value when one exists and a bandwidth-model
estimate discounted by 0.5× otherwise. HomePilot lists keep only models it can install. Changing the
weights or the order requires a version bump, making every rank shift visible in the pull request diff.

**MEDIA** (image and video generation) has its own lane — see [Media lane](#media-lane).

## Secrets

All secrets live in **GitHub** — Settings → Secrets and variables → Actions → New repository
secret. Nothing is configured on the Hugging Face side: a *static* Space has no secrets and
needs none, because the page is a static file that fetches a public `registry.json`. The
Space never calls Kaggle or Hugging Face with a token; the GitHub workflow holds the
credentials, builds the data, and pushes the finished page to the Space.

| Secret | Needed by | Where to get it | Without it |
|---|---|---|---|
| `HF_TOKEN` | `sync-hf-space` | huggingface.co → Settings → Access Tokens → **Write** | The Space is never updated |
| `KAGGLE_USERNAME` | `weekly-benchmark` GPU lane | your Kaggle username | GPU lane skips, CPU lane still runs |
| `KAGGLE_API_TOKEN` | `weekly-benchmark` GPU lane | kaggle.com → Settings → API → **Generate New Token** | GPU lane skips, CPU lane still runs |

`KAGGLE_API_TOKEN` is the variable the current Kaggle CLI reads and the only one that accepts
a `KGAT_…` token; `KAGGLE_KEY` is interpreted as a legacy API key and fails against one. The
workflow passes both, so an older `KAGGLE_KEY` still works if that is what you have.

Also enable **Allow GitHub Actions to create and approve pull requests** (Settings → Actions →
General → Workflow permissions). Without it the weekly review PRs cannot be opened; data still
publishes to `registry-latest`, so the leaderboard stays current either way.

```
GitHub repo secrets ──> weekly-sync ────────> registry-latest branch ──┐
                   │                                                   ├──> leaderboard + CLI
                   ├──> weekly-benchmark ──> Kaggle T4 ──> data/ ──────┘
                   └──> sync-hf-space ─────> Hugging Face Space (static, no secrets)
```

## Automation

| Workflow | Schedule | What it does |
|---|---|---|
| [`weekly-sync`](.github/workflows/weekly-sync.yml) | Wed 04:41 UTC | Refreshes the Ollama library catalogue, pulls the leading models per Hugging Face pipeline tag by trend, downloads **and** last-modified, re-runs the FITS engine, resolves every entry to a verified Ollama tag, rebuilds the media lane, validates, publishes to `registry-latest`, and opens a review PR |
| [`weekly-benchmark`](.github/workflows/weekly-benchmark.yml) | Mon 03:17 UTC | Benchmarks the rotation model on a free T4 via the Kaggle Kernels API, with a CPU lane as fallback, and runs the PLUGS probes. The rotation passes the registry id through as `--model-id`, so the result attaches to the model it measured |
| [`sync-hf-space`](.github/workflows/sync-hf-space.yml) | On push + Thu 06:20 UTC | Deploys the leaderboard to the Hugging Face Space, committing on top of the Space's history |
| [`community-submission`](.github/workflows/community-submission.yml) | On labeled issues | Extracts, schema-validates and sanity-bounds a submitted result, then merges it via PR with attribution |
| [`publish-pypi`](.github/workflows/publish-pypi.yml) | On release | Builds, smoke-tests and publishes the package using PyPI Trusted Publishing (OIDC, no long-lived token) |

Operational properties: workflows use `concurrency` groups to prevent overlapping writes, every
script is idempotent and safe to re-run, and degradation is graceful — if Hugging Face is
unreachable the sync PR is simply not opened and the last-good registry stands; if Kaggle is
unavailable the CPU lane still lands a datapoint.

## Development

The repository is driven by a Makefile backed by [uv](https://docs.astral.sh/uv/). From a clean
clone, two commands give you a working environment and a running CLI — uv is bootstrapped
automatically if it is not already installed:

```bash
git clone https://github.com/ruslanmv/fitlab && cd fitlab
make install          # creates ./.venv and installs fitlab + all tooling
make run              # launches the fit-check wizard
```

`make run` depends on `make install`, so `make run` alone is enough on a fresh clone. Run
`make help` to list every target.

| Target | What it does |
|---|---|
| `make install` | Create `./.venv` and install the package plus the `dev` extra (`uv sync --extra dev`) |
| `make run` | Launch the interactive wizard |
| `make detect` / `make check` / `make bench` / `make update` | The corresponding `fitlab` subcommands |
| `make registry` | Rebuild `data/registry.json` from Hugging Face (FITS + sync) |
| `python scripts/ollama_catalog.py` | Refresh `data/ollama_catalog.json` from the Ollama library |
| `python scripts/ollama_catalog.py --check <hf_id>…` | Show the Ollama tag a Hugging Face repo resolves to |
| `make estimate` | Full FITS verdict matrix across seed models × reference GPUs × quantizations |
| `make probe` | Run the PLUGS end-to-end stack probes |
| `make validate` | Schema-validate every document in `data/` |
| `make lint` / `make format` | Ruff check / safe auto-fix |
| `make smoke` | Exercise the CLI end to end without downloading a model |
| `make test` | `lint` + `validate` + `smoke` |
| `make build` | Build sdist and wheel into `dist/`, then `twine check` them |
| `make site` | Serve the static site on `http://127.0.0.1:8000` |
| `make clean` / `make distclean` | Remove artifacts / also remove `./.venv` |

Targets accept overrides on the command line:

```bash
make check GPU=t4-16 CATEGORY=colab-free LIMIT=5
make bench MODEL=qwen3:14b REPS=5
```

Available variables: `GPU`, `QUANT`, `CTX`, `CATEGORY`, `LIMIT`, `MODEL`, `REPS`, `OUT`.

In a checkout, the Makefile points `FITLAB_REGISTRY_URL` at the repository's own
`data/registry.json`, so every target reads the registry you are editing and works offline.
Override it to exercise a remote fetch:

```bash
make run FITLAB_REGISTRY_URL=https://example.com/fitlab/registry.json
```

Dependency groups live in `pyproject.toml`: the `maintainer` extra carries what the `scripts/`
engines need (`pyyaml`, `jsonschema`), and `dev` adds `ruff`, `build` and `twine` on top. `uv.lock`
is committed, so `make install` resolves to identical versions on every machine.

The engines can also be invoked directly, without make:

```bash
# FITS — verdict for one model on one GPU
python scripts/estimate_vram.py --model Qwen/Qwen3-14B --gpu rtx3060-12 --quant Q4_K_M

# Registry refresh from Hugging Face (writes data/registry.json locally)
python scripts/sync_hf.py

# RUNS — benchmark on whatever hardware you are on (local, Colab, or Kaggle)
python scripts/bench_gguf.py --model qwen3:8b --out data/benchmarks/

# PLUGS — end-to-end stack probes (CPU is sufficient)
python scripts/compat_probe.py --report data/compat_report.json

# Schema validation
python scripts/validate_data.py
```

`compat_probe.py` and `validate_data.py` both exit with the number of failures, so CI fails
honestly rather than reporting a green matrix built on skipped probes.

There is no unit-test suite yet; `make test` runs linting, schema validation and a CLI smoke test.

The published site is static and reads `data/*.json` directly; serve `site/` through GitHub Pages
or any static host.

## Contributing

Benchmark results are the most valuable contribution and take about three minutes on free
infrastructure:

1. Open [`colab/benchmark_colab.ipynb`](colab/benchmark_colab.ipynb) in Colab, select a T4 runtime,
   and run all cells — or run `python scripts/bench_gguf.py --model <tag> --source community-local`
   on your own machine.
2. The final cell prints your result and a link that opens a prefilled GitHub issue.
3. CI validates the JSON and merges it; your GPU and handle appear on the page.

Model suggestions belong in `data/models.seed.yaml` only — never in the generated `registry.json`.
Ranks are computed rather than negotiated: to change a rank, change the formula in
`scripts/sync_hf.py` and bump `ranking_version`. Full guidelines are in
[`CONTRIBUTING.md`](CONTRIBUTING.md).

## Security

- **Community submissions are data, never code.** Submitted JSON is parsed and validated; it is
  never executed.
- **Untrusted code paths are isolated.** The weekly stack probes install unpinned latest releases
  and run inside ephemeral CI runners, optionally under
  [MatrixLab](https://github.com/agent-matrix/matrixlab) Docker isolation for local maintainer runs
  (see [`docker-compose.matrixlab.yml`](docker-compose.matrixlab.yml)).
- **Secrets never reach fork-triggered workflows.** `pull_request_target` is deliberately avoided.
- **Publishing uses OIDC.** PyPI Trusted Publishing means no long-lived API token exists to leak.

Please report suspected vulnerabilities privately through
[GitHub Security Advisories](https://github.com/ruslanmv/fitlab/security/advisories/new) rather
than in a public issue.

## Support and versioning

This is a community open-source project maintained on a best-effort basis; no service level is
implied. Questions and bug reports belong in
[GitHub Issues](https://github.com/ruslanmv/fitlab/issues).

Three version numbers move independently, and only the first is a package concern:

- **Package version** (`fitlab`) follows semantic versioning.
- **`schema_version`** changes only for breaking data-format changes; consumers should pin it.
- **`ranking_version`** changes whenever scoring weights change, so ranking shifts are auditable.

Benchmark results are point-in-time measurements on specific hardware and driver versions. They are
a comparative signal, not a guarantee of what your machine will do.

## Related projects

- [OllaBridge](https://github.com/ruslanmv/ollabridge) — one OpenAI-compatible URL for all your local models
- [HomePilot](https://github.com/ruslanmv/HomePilot) — local-first chat, image and video studio
- [GitPilot](https://github.com/ruslanmv/gitpilot) — Git workflow automation
- [MatrixLab](https://github.com/agent-matrix/matrixlab) — Docker isolation sandbox used for untrusted probes
- [Best-of-the-Best](https://github.com/ruslanmv/Best-of-the-Best) — the auto-curation pattern this repository generalizes

## License

**Code** is licensed under [Apache-2.0](LICENSE). **Data** in `data/` and all published registries
are licensed under [CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/).

Benchmark results are point-in-time measurements. Models referenced remain the property of their
respective authors and are subject to their own licenses.

## Media lane

Image and video generation models have no single `config.json` to size from, so FITS arithmetic
does not apply. The media lane keeps FitLab's rules — every number has a recorded source, and
nothing is hand-ranked — with a different input:

- [`data/media.seed.yaml`](data/media.seed.yaml) declares each base checkpoint's VRAM need
  (`min` = runs with offload / low-VRAM mode, `recommended` = native resolution without offload),
  where each number came from (`homepilot-presets`, `editorial`, later `measured`), and the work its
  default workflow does (steps × megapixels × frames).
- [`scripts/build_media.py`](scripts/build_media.py) adds live Hugging Face likes and downloads,
  derives a verdict per GPU and ranks under `media_ranking_version` 1.1. Tops are ordered by likes
  and only include checkpoints HomePilot can install:

```
score = 0.40·fit + 0.25·capability + 0.25·momentum + 0.10·speed
momentum: 0.75·log likes + 0.25·log downloads
fit:  fits ≥ recommended · tight ≥ min · offload ≥ 0.7·min · otherwise no
```

The output, [`data/media_registry.json`](data/media_registry.json), publishes Top-10 lists for
12, 16 and 24 GB GPUs (image and video), and — so a client can rank for **any** VRAM without
re-implementing the formula — each model's GPU-independent `components` (capability, momentum,
speed) alongside the `scoring` weights. A client computes only the fit verdict for its own GPU:
`score = Σ weight · component`, with `fit = fit_score[verdict]`.

```bash
make media                         # refresh HF stats and rebuild
python scripts/build_media.py --offline   # rebuild from the seed, keeping previous stats
```

The LLM `registry.json` is unchanged by this lane: its schema, keys and consumers stay as they were.

## Using FitLab from HomePilot

[HomePilot](https://github.com/ruslanmv/HomePilot) reads FitLab to suggest the top models for the
GPU it is running on — chat and vision from `registry.json`, image and video from
`media_registry.json` — under **Models → Suggested for your GPU**. Suggestions are the models
HomePilot can install that fit the machine, ordered by Hugging Face likes, so the first row is the
one people have ranked highest. The contract it relies on:

| Field | Used for |
|---|---|
| `models.*.arch`, `params_b`, `active_params_b` | FITS verdict and speed for the detected GPU (same arithmetic as `fitlab check`) |
| `models.*.ollama_tag` | One-click install through HomePilot's Ollama pull |
| `media_registry.json` `models.*.homepilot` | `{provider, model_type, model_id}` for HomePilot's ComfyUI install |
| `inference_registry.json` `kinds.<type>.lists.<order>` | **Models → Hugging Face**: hosted models to pick for chat, vision, images and video |

### Inference lane — hosted models, ranked by people

`scripts/sync_inference.py` publishes `data/inference_registry.json`: for each HomePilot model type
(`chat`, `vision`, `image`, `video`) the models Hugging Face Inference Providers serve **live** for the
task HomePilot calls (adapters and GGUF repos excluded), in five orders:

| Order | Meaning |
|---|---|
| `popular` | `0.55·likes + 0.15·downloads + 0.20·trending + 0.10·recency` (each log-scaled 0..1; recency has a one-year half-life). The default suggestion. |
| `likes` | Most liked first — ranked by the people |
| `trending` | The Hub's trending score |
| `newest` | Most recently created |
| `rising` | Created in the last 120 days, by trending — newer models a likes order would bury |

Every row keeps its raw `likes`, `downloads` and `trending_score` beside the `score`, plus the
`live_providers` serving it. Each type also reports `diversity.top_publisher_share`, to watch one
publisher crowding the list. Popularity guides; it does not measure quality on your task.
The ranking is the pure module `fitlab.inference`; HomePilot ranks live Hub data with the same rules
and falls back to this feed, then to the copy it ships with.

| `media_registry.json` `scoring` + `models.*.components` | Ranking for any VRAM, identical to the published lists |
| `generated_at`, `ranking_version`, `media_ranking_version` | Shown with every suggestion |

HomePilot fetches when the user presses **Fetch definitions** — and, if the user leaves it on, in a
daily background check — reading `registry-latest` first and `master` second with `If-None-Match`,
so an unchanged registry costs a `304` and no download. Offline it falls back to its last cached
copy, then to a snapshot bundled in the release — the same order as the `fitlab` CLI.
`FITLAB_REGISTRY_URL` points it at a mirror.

## NVIDIA GPU catalog

Search 85 hardware profiles across desktop and laptop GeForce generations, or enter
a custom GPU and VRAM capacity in either web interface. The CLI shares the same
offline inventory (`fitlab gpus`, `fitlab check --gpu 'RTX 4060 Ti 8GB'`). Monthly
NVIDIA source refreshes open review PRs; unknown specs and source conflicts stay explicit.
See [GPU catalog usage and maintenance](docs/gpu-catalog.md) for sources, custom
hardware, launch guidance, validation, deployment and rollback.
