Metadata-Version: 2.5
Name: vulntrack
Version: 0.2.0b2
Summary: Assess Harbor/Trivy findings with stacked, auditable CVSS environmental rules (append-only parquet store)
Author-email: brunnelu <6707792+brunnelu@users.noreply.github.com>
License-Expression: MIT
License-File: LICENSE
Requires-Python: >=3.11
Requires-Dist: click>=8.1
Requires-Dist: cvss>=3.6
Requires-Dist: httpx>=0.27
Requires-Dist: polars>=1.17
Requires-Dist: pydantic-settings>=2.14.2
Requires-Dist: pydantic>=2.7
Requires-Dist: tenacity>=9.1.4
Requires-Dist: textual>=8.2.8
Description-Content-Type: text/markdown

# vulntrack

Assess Harbor/Trivy findings with a **stacked, auditable rules engine**.
Persistence is **append-only parquet in blob storage** (Azure container, or a local
path). Collect spec: **[vulntrack/task.md](vulntrack/task.md)**; architecture:
**[docs/index.md](docs/index.md)** (its section 7 on storage is superseded by task.md).
What changed and what is still open: **[WORKLOG.md](WORKLOG.md)**.

## Quickstart

```bash
uv sync

uv run vulntrack setup            # print the sample config, with comments
uv run vulntrack setup --write    # create vulntrack.toml, .env, rules/ and packs/

uv run vulntrack collect --no-submit        # progress bar on a terminal; --no-progress off

# -v logs every HTTP request (httpx's own line)
uv run vulntrack -v collect --no-submit

# map extra columns onto the frame — deployments, KEV, owners, anything
uv run vulntrack packs list      # resolution order and the dependency graph
uv run vulntrack packs check     # duplicates, collisions, coverage, dead rules

# rules are tables: one row is one reason. See rules/README.md
uv run vulntrack rules list
uv run vulntrack rules add --id cve.x --when 'cve=^CVE-2024-0001$' \
  --set-metric MA=N --because 'Why this does not apply here.'

# how the estate is being rated, and which .tsv row to add next
uv run vulntrack progress

# record the current assessments so changes over time are queryable
uv run vulntrack snapshot --dry-run
uv run vulntrack snapshot

# Textual TUI — walk residue by finding PK, apply rules (recommended)
uv run vulntrack ui

# same flow without TUI
uv run vulntrack assess
uv run vulntrack assess CVE-2024-0001 --set MA=N --because "…"
```

## Data model: PK, joins, assessments

### Occurrence primary key (source row)

Every finding from Harbor is one row with this identity:

```text
project
project_version     # if you maintain one; else empty
cve
image               # repository without tag
image_tag           # part of PK as you see it in the world
digest              # part of PK (immutable content); usually hidden in UI
package
package_version
```

That whole key is the **occurrence**. Digests are long — keep them in the
data, **do not put them in the default table columns** (optional short
prefix on drill-down / copy).

Today’s names: `image_repository` ≈ `project/image`, `installed_version` ≈
`package_version`, `tags` ≈ `image_tag`.

In code it is one hard-coded constant, `engine.frame.ASSESSMENT_PK`:

```python
ASSESSMENT_PK = ("image_repository", "digest", "cve", "package", "installed_version")
```

Hard-coded because a configurable primary key is one nobody can reason about, and
everything that has to line up across runs — the rated frame, the materialised
output table, `explain` — has to agree on it without reading a config file.

Measured on the live estate (22,511 findings, 7 repositories): `digest + cve +
package + installed_version` is already exactly unique and no digest appears in
two repositories, so `image_repository` is redundant *today*. It is in the key
anyway: a digest is content-addressed, so the same image pushed to a second
repository would share it while joining to different packs and rating
differently. One redundant column beats a key that is unique by an accident of
registry layout.

Not in the key: `tags` (a `List(Struct)` in the frame, and a retag must not mint a
new assessment of content that did not change) and every joined column (packs
state facts about a row, they do not identify it). `vulntrack packs check`
verifies uniqueness against real collected data, so the constant is checked rather
than asserted in a comment.

**Rules are free to match on anything**, in the key or not. `workspace_env` is a
boolean a pack states over a subset of repositories; a rule keyed on it still
produces assessments identified by exactly the columns above.

### Augmented columns (joins — not PK)

Anything else is **joined on**, not part of the source PK. Joins come from
**packs**: one TSV per mapping in `packs/`, keyed on a column the frame already
has — including a column an earlier pack produced. See `vulntrack/templates/packs/`.

```text
image  ──join──►  packs/deployments.tsv  ──►  image_kind, in_use, exposure, …
cve    ──join──►  packs/kev.tsv          ──►  kev, …
image  ──join──►  packs/projects.tsv     ──►  project_id
                     └─join──►  packs/project-tiers.tsv  ──►  tier, data_class
```

The order is derived from the key columns, never the filenames; a key nothing
produces, a cycle, or two rows with the same key is an error that names the pack.
A pack never adds a row — a mapping row matching nothing is reported by
`vulntrack packs check`, not merged in. Every value a pack writes lands in
`mapping_trail` beside the `reasoning`, and `vulntrack explain <digest> <cve>`
walks one value back through the chain to the image it came from.

`image_kind` is **not** a Harbor field and **not** part of the PK. It is
augmented data: you maintain `image → image_kind` (and friends); we left-join
it onto occurrences before rules run. Missing join → null → class rules do
not match until the register is filled.

Same pattern later for KEV, distro status, etc.: new pack, join key, new
column, null when unknown.

### Who assessed

Every rule saved from an install records `assessed_by`, taken from
`Settings.username` at save time. It resolves highest-first:

1. `username` in `vulntrack.toml`, or `VULNTRACK_USERNAME`
2. the variable `username_env` names — `VULNTRACK_STARTING_USERNAME` by default
3. the OS account

Step 2 is the point: a shared image ships one config and still records the right
person, because the platform that starts it already knows who is in it. Nothing
about it is a credential — `[projects.*].username` is a Harbor robot the tool
authenticates **as**; this is the person it acts **for**.

### Assessment (computed, then materialised on demand)

```text
occurrence (+ joined columns)  +  reasons  →  env vector, severity, trail
```

Recomputed every time, from rules + inputs. `vulntrack snapshot` writes the
result into the store's one output table, `assessments`, change-detected against
the PK — so a row lands only when the assessment actually changed, and "which
findings moved, and when" is a plain store query. Its columns are static
(`engine.frame.ASSESSMENT_OUTPUT`):

```text
image_repository digest cve package installed_version   the PK
assessment_reason                                       every reason that fired
env_vector env_score env_severity                       what it came out as
assessed_by assessed_at                                 who last had an opinion
```

Static rather than configurable on purpose: a per-install output shape would make
two stores incomparable, which is the one thing a change-tracking table cannot
afford.

**Who is answerable.** An assessment is the product of several reasons, so the
author is picked from a set: the *most recently edited reason that applied*. Each
rule row carries `assessed_at`, and the freshest opinion carries its author. This
is the only table vulntrack writes the contents of — findings can be re-collected
and the rating recomputed, but what we concluded last month exists nowhere else.

### How people actually assess (the product loop)

In practice almost all **new** work is **per CVE**:

1. Open a CVE still in residue (Critical/High after the stack).
2. See which occurrence rows / joined context it covers (images, tags,
   packages, versions — digest hidden).
3. Write **one rule** with:
   - `when`: at least `cve = …`, plus whatever else is true  
     (e.g. only workspace via joined `image_kind`, only one package, …)
   - `set`: environmental metrics (and/or terminal severity)
   - `because`: **custom reasoning text** (the audit trail)
4. Save → recompute → matching rows leave residue; continue.

You are not maintaining a spreadsheet of assessments. You add **rules with
reasons**; the PK table only shows *where* that rule applies after the join
and stack.

Broad posture rules (`project.*`, `workspace.posture`) stay few and shared.
Residue work is mostly `cve.…` rules with a human `because`.

### What the UI should show

```text
Queue:      residue grouped or listed by CVE (work list)
Detail:     description + joined facts + occurrence rows for that CVE
            (PK columns; digest hidden)
Applied:    which rules already fired / current env result
Write:      scope (CVE ± package ± image ± image_kind) + metrics + because
```

Showing every PK row is fine at residue scale (~hundreds). Collapsing
identical env outcomes is optional sugar — not required to understand the
model.

### Rules vs PK (important)

| Prefer in `when` | Avoid in `when` |
|---|---|
| `cve`, `package`, `image`, joined `image_kind` | `digest` (breaks on rebuild) |
| other joined facts you trust | bare `image_tag` unless the claim is really tag-specific |

Tag and package_version **are** part of the occurrence PK (inventory truth).
Rules still usually **omit** them so one decision covers rebuilds; only add
them when the claim is truly version- or tag-specific.

### Mapping today

| surface | role |
|---|---|
| `vulntrack ui` | walk residue, write per-CVE rules |
| `vulntrack assess` | same in pure CLI |
| `vulntrack rate` | dump rated occurrences (includes digest) |
| `vulntrack packs list` | pack resolution order and the dependency graph |
| `vulntrack packs check` | lint the packs — non-zero on error, for CI |
| `vulntrack explain <digest> <cve>` | rules fired, their inputs, and the mapping chain |
| `vulntrack progress` | severity shift, per-rule hits, pack coverage, what to map next |
| `vulntrack snapshot` | materialise the current assessments into the store (`--dry-run`) |
| `vulntrack setup` | print (or `--write`) the config files this repo needs |

## Blob storage & Azure credentials

`store` can be a local path or `az://container/prefix`. Credentials are **injected,
never constructed inside the store** — whoever runs vulntrack owns the auth decision.
Three ways in, highest precedence first.

**1. A credential object you already hold** (the injection point for an app that has
already authenticated):

```python
from azure.identity import DefaultAzureCredential
from vulntrack.settings import Settings
from vulntrack.store import open_store

store = open_store(Settings(), credential=DefaultAzureCredential())
findings, images = store.findings(), store.images()
```

Any object with a `get_token(*scopes)` returning `.token` / `.expires_on` works —
`ManagedIdentityCredential`, `AzureCliCredential`, `ClientSecretCredential`, or your
own wrapper. It is exchanged for a bearer token that polars refreshes on expiry, so a
long `collect` does not die an hour in.

**2. A provider callable**, if you want full control of caching or a non-Azure token:

```python
store = open_store(Settings(), credential_provider=lambda: ({"bearer_token": tok}, exp))
```

The contract is polars': return `({key: value}, expiry_epoch_or_None)`.

**3. A name in `vulntrack.toml`** — the convenience path the CLI uses. Needs the
`azure-identity` package; nothing secret goes in the file:

```toml
store = "az://vulntrack/prod"
azure_credential = "default"   # default | cli | managed_identity | environment | workload_identity
```

For an account key or SAS instead, hand object_store its options directly. These *are*
secrets, so prefer the environment (`VULNTRACK_STORAGE_OPTIONS`) over the file:

```toml
[storage_options]
account_name = "myaccount"
```

`storage_options` is merged in either way, so the account name can live in the config
while the token arrives from code. A local `store` path ignores all of it.

> Untested against a real Azure account — there is no tenant available here. The
> contract is pinned by tests using a fake credential (`tests/test_store_credentials.py`),
> including the `bearer_token` key name, which was taken from polars'
> `CredentialProviderAzure` rather than guessed.

## Layout

| path | role |
|---|---|
| `vulntrack/engine/` | frame + packs, stacking rules, CVSS env score |
| `vulntrack.toml` | every setting (gitignored; written by `vulntrack setup --write`) |
| `packs/*.tsv` | mapping packs joined onto the frame (templates in `vulntrack/templates/`) |
| `rules/*.tsv` | rules tables — one row is one reason (template `vulntrack/templates/rules/`) |
| `.env` | every secret — `HARBOR_<PROJECT>_PASSWORD` |
| `vulntrack/settings.py` | env → `.env` → `vulntrack.toml` → defaults |
| `vulntrack/templates/` | what `setup` copies: config, `.env`, packs, seed ruleset |
| `vulntrack/store.py` | append-only parquet store, SCD2 views |
| `vulntrack/harbor.py` | Harbor client + collect job |
| `vulntrack/task.md` | collect-layer spec — read this before changing either |
| `vulntrack/tui.py` | Textual assess UI |
| `vt-store/` | runtime data (gitignored) |

### What code runs when

One path per command, so "which code produced this number" is answerable without
reading the whole tree. Read top to bottom; every rating goes through the same four
steps in the same order.

```text
vulntrack collect     harbor.py            Harbor API  → store.append()      (writes)
                      store.py             append-only parquet, change-detected

vulntrack rate        store.py             load findings + images
  │                   engine/frame.py      build_frame()  ─ one row per occurrence
  │                   engine/packs.py      resolve() → apply_pack()  ─ LEFT join, row count fixed
  │                   engine/rules.py      Reason.mask()  ─ one polars expr per condition
  │                   engine/rating.py     _metrics()  ─ most severe proposal wins
  │                   engine/rating.py     _verdicts() ─ terminal labels, after the metrics
  └─►                 engine/rating.py     _score()    ─ scored per distinct vector

vulntrack assess      engine/rating.py     residue() over the same rated frame
vulntrack ui          tui.py               Textual; the same rate() call
vulntrack packs …     engine/packs.py      coverage(), claimed_twice(), unproduced_rule_columns()
vulntrack explain …   engine/packs.py      explain_column() walks the trail backwards
vulntrack snapshot    store.py             append() — change-detected, PK-keyed
vulntrack progress    cli.py               the same rate() call, summarised
vulntrack setup       cli.py               copies *.example — no logic, on purpose
```

Two properties worth knowing, because they are what keeps the above auditable:

- **`Reason.mask()` is the only place a condition is evaluated.** Rating and the
  coverage preview call the same expression, so the preview cannot disagree with the
  rating. There is no second row-at-a-time implementation. `Store.pending()` is the
  same idea for writes: `snapshot --dry-run` asks the writer what it would do rather
  than re-deriving the diff.
- **Reasons are a set, and the most severe proposal wins.** Order-free, so the answer
  cannot depend on how the tables are sorted — see [docs/rules.md](docs/rules.md).
- **`apply_pack` asserts the row count is unchanged.** Every join is a left join and
  the frame is exactly the findings Harbor reported — see `plan.md`, "Invariant:
  every join is a left join".
