Metadata-Version: 2.5
Name: bspkit
Version: 0.2.0
Summary: Reproducibly fetch, prepare, patch and build software components
Project-URL: Homepage, https://gitlab.com/conpiko/bspkit
Project-URL: Repository, https://gitlab.com/conpiko/bspkit
Project-URL: Issues, https://gitlab.com/conpiko/bspkit/-/issues
License-Expression: MIT
License-File: LICENSE
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Build Tools
Requires-Python: >=3.11
Description-Content-Type: text/markdown

# bspkit

bspkit is a small, generic CLI for reproducibly fetching, preparing, patching,
and building software components.

This README documents the implemented user/developer interface. Development
workflow and coding-agent rules are documented in [AGENTS.md](AGENTS.md).

## Implementation status

The current checkout implements source/patch management, single-component builds,
and project status summary:

- `create`
- `fetch` for Git, checksum-verified binary, and hook-managed native sources
- `fetch --no-verify` for binary checksum bootstrap
- `fetch --unshallow` for complete pinned Git ancestry
- `prepare`
- `patch export`
- `patch sync`
- `build [-f/--force] <component>`
- `status`

Dependency-aware builds, cleaning, detailed component status, install/staging,
and scheduling remain future work. Sections describing those commands below are
not implemented yet.

## Requirements

- Python 3.11+
- Git
- Linux, macOS, or Windows through WSL2

Native Windows is not supported. The Git profiles and developer tooling use
POSIX facilities such as `/dev/null` and a shell.

## Install

From this checkout:

```sh
pip install .
```

Or run directly:

```sh
PYTHONPATH=src python3 -m bspkit
```

## Quick start

Create a project:

```sh
bspkit create my-bsp
cd my-bsp
```

A new project is valid before any components are added:

```toml
schema = 1
```

Add a component in `bspkit.toml`:

```toml
schema = 1

[components.u-boot]
type = "git"
url = "https://source.denx.de/u-boot/u-boot.git"
revision = "<full-commit-sha>"
```

Fetch and prepare it:

```sh
bspkit fetch u-boot
bspkit prepare u-boot
```

Project state:

- cached Git mirrors: `.bspkit/repos/`
- editable source worktrees: `work/<component>/`
- project-owned patch stacks/hooks: `<component>/`
- private bspkit state: `.bspkit/state/`

## Components and dependencies

A component can depend on other components:

```toml
[components.tf-a]
type = "git"
url = "https://git.trustedfirmware.org/TF-A/trusted-firmware-a.git"
revision = "<full-commit-sha>"

[components.u-boot]
type = "git"
url = "https://source.denx.de/u-boot/u-boot.git"
revision = "<full-commit-sha>"
depends = ["tf-a"]
```

A project-owned `<component>/` directory is optional. A source-only component
needs only its `bspkit.toml` entry.

`fetch` and `prepare` expand dependencies. The current `build` command does not:
run dependency builds manually, in the required order.

## Selective and shallow Git sources

Git components can provide acquisition options alongside the immutable commit pin:

```toml
[components.linux]
type = "git"
url = "https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git"
revision = "<full-commit-sha>"
ref = "refs/tags/v6.18.56"
depth = 1
```

Replace the placeholder with the tag's full commit SHA, peeling an annotated tag.
`ref` must be a fully qualified Git reference under `refs/`, without wildcards or
refspec syntax. `depth` must be a positive integer; booleans are rejected. Both
options are Git-only. `revision` remains a required full commit SHA.

When the pin is absent locally, bspkit checks the exact remote ref against it,
then fetches only that ref, without other branches or automatic tag following.
Without `depth`, the selected ref's reachable history is fetched. With `depth`
alone, bspkit attempts a direct pinned-SHA fetch; if the server rejects it,
configure a matching `ref`. Selective failures never fall back to downloading the
entire remote. Without either option, initial acquisition retains full mirror
cloning.

Shallow fetching limits commit ancestry. The pinned checkout's trees and blobs
are still required, so `depth = 1` does not avoid downloading its source content.
Caches remain bare repositories under `.bspkit/repos/`, shared by linked
developer worktrees. Acquisition options do not affect source/build identity or
invalidate preparation. A cached pin is reused without requiring its remote ref
to remain unchanged. Existing history is never narrowed to match a new depth.

To obtain complete pinned ancestry:

```sh
bspkit fetch --unshallow linux
bspkit fetch --unshallow
```

`--unshallow` ignores configured depth, including on first acquisition, and
extends existing shallow caches in place. Existing worktrees immediately see the
new history, preserving their HEAD, local commits, dirty files, and lifecycle
state. Full caches containing the pin need no network access. Later ordinary
fetches do not re-shallow a full cache. If a cached pin's configured ref moved or
disappeared, unshallow attempts the pinned SHA instead; unsupported acquisition
fails without a full-remote fallback or replacing the usable cache.

Selection includes transitive dependencies, or all components when none are
named. Selected binary and native sources undergo ordinary fetching.
`--unshallow` and `--no-verify` are mutually exclusive.

## Binary sources

To obtain a vendor binary whose trusted checksum is not known yet, configure a
binary source:

```toml
[components.firmware-imx]
type = "bin"
url = "https://www.nxp.com/lgfiles/NMG/MAD/YOCTO/firmware-imx-8.21.bin"
sha256 = ""
```

Binary components accept exactly `type`, `url`, required `sha256`, and optional
`depends`. An empty checksum is valid for bootstrap; a non-empty checksum must
contain exactly 64 hexadecimal characters. `revision` is invalid for binary
components, and `sha256` is invalid for Git components.

Run:

```sh
bspkit fetch --no-verify firmware-imx
```

The command downloads fresh bytes on every invocation, ignoring any configured
checksum and existing cache entry. It reports the component name, computed
SHA-256, and resulting cache path. Inspect and trust the downloaded file, then
manually copy the reported digest into `bspkit.toml`; bspkit never edits that file.
The reported path points into private cache storage whose layout is not a public
interface. A failed or interrupted download preserves any existing cached binary.

For `--no-verify`, selection follows normal fetch dependency expansion. Every
selected component, including dependencies, must be binary; otherwise the command
fails before any download. With no component arguments it selects all configured
components.

After pinning a trusted checksum, use normal fetch:

```sh
bspkit fetch firmware-imx
```

Normal binary fetch requires a non-empty `sha256`. It hashes the cached file and
reuses matching bytes without network access. An absent or checksum-invalid cache
triggers a fresh download. Checksum comparison is case-insensitive. Downloads are
verified in temporary storage on the cache filesystem and published atomically
only when their SHA-256 matches. A mismatch reports the expected and actual
digests. Download, read, verification, and publication failures preserve any
previous cache; failures with no previous cache leave no normal entry.

Normal fetch may mix Git, binary, and native components, including dependencies.
Binary sources can be prepared and patched using the same editable Git workflow
as Git sources. Binary builds and source/build identities remain deferred;
`bspkit build` rejects binary components before preparing or changing build state.

## Native sources

Use `native` for source trees assembled by project-owned scripts:

```toml
[components.generated]
type = "native"
```

Only `type` and optional `depends` are accepted. All hooks are optional; a dummy
component with no directory or hooks supports fetch, prepare, build, and status.

For downloaded inputs, provide an executable `generated/fetch`:

```sh
#!/bin/sh
set -eu
# sources.lock contains a URL and a trusted sha256sum check line.
. ./fetch.d/sources.lock
curl --fail --location "$SOURCE_URL" --output "$BSPKIT_FETCH_DIR/source.tar"
(cd "$BSPKIT_FETCH_DIR" && printf '%s\n' "$SOURCE_CHECK" | sha256sum --check -)
```

Fetch runs with zero arguments in the project-owned `generated/` directory,
with the common hook environment and `BSPKIT_FETCH_DIR` set to private staging.
Put recursive fetch inputs such as lock files in `generated/fetch.d/`.
Bspkit reuses downloads only while hook/data inputs and verified cache content
match. Failed execution, validation, or publication invalidates the previous
cache; subsequent preparation cannot fall back to those old downloads. Fetch
never creates a workspace. Hooks own download and checksum policy; bspkit has no
package manager or archive-format inference.

Provide an executable `generated/prepare` to unpack or generate sources:

```sh
#!/bin/sh
set -eu
tar -xf "$BSPKIT_FETCH_DIR/source.tar"
printf 'output/\n' > .gitignore
```

Prepare starts from a deterministic empty Git root commit in `work/generated/`.
The hook reads verified downloads through `BSPKIT_FETCH_DIR` and writes generated
sources in its workspace CWD. Treat the published cache as immutable. Without a
fetch hook, this variable points to an empty read-only directory. Bspkit captures
the prepared tree as the patch base, then applies ordinary patches. Local commits,
patch export/sync, and dirty-work protection use the same Git workflow as other
sources.

Native preparation tracks fetch inputs, verified downloaded content, `prepare`,
`prepare.d/`, and remaining source files in the component directory recursively.
Patches are tracked separately. `build`, `build.d/`, `build.fingerprint`,
`build.fingerprint.d/`, `install`, `install.d/`, `clean`, and `clean.d/` are excluded
from preparation. Changing these inputs does not block patch export. Restore
changed source/prepare inputs and the verified cache before exporting local work.

Native builds use the existing build and fingerprint mechanisms. Recursive
`build.d/` inputs affect build identity even without a build hook. Add appropriate
workspace ignore rules for generated artifacts so builds do not invalidate their
own source identity. Fingerprint inputs remain build-only.

## Source preparation

Preparation follows this pipeline:

```text
verified source -> materialization base -> optional prepare hook -> patch base -> patches
```

Git materialization checks out the pinned revision. Binary materialization copies
the verified bytes to `work/<component>/source.blob` as an independent regular
file with mode `0644`, then records a deterministic internal Git root commit.
Workspace edits and deletions cannot change the private binary cache. Core does
not detect formats, unpack archives, or execute the blob.

An optional executable `<component>/prepare` hook runs after materialization and
before the patch series. It uses the same zero-argument, worktree-CWD and common
environment contract documented below for build hooks. Consumer policy may
transform, rename, remove, or execute `./source.blob` after changing its mode.
The hook is responsible for deterministic results from identical declared inputs.

Without a hook, the patch base is the materialization base. With a hook, bspkit
captures the resulting source tree in a deterministic internal commit, including
ignored output, executable modes, and symlinks. Unsupported special objects such
as FIFOs or sockets fail preparation with a diagnostic. Internal base commits are
excluded from consumer patch exports.

Source URL, pinned revision/checksum, prepare-hook contents and executable state,
and ordered patches determine canonical preparation. Unchanged preparation is a
no-op that preserves local/dirty developer work. Changed inputs only permit
replacement of a clean canonical workspace.

A failed hook leaves an inspectable workspace and failed preparation state;
patches are not applied. Restore the recorded materialization base and remove
non-ignored untracked changes before retrying. After failed patch application,
`git am --abort` restores the recorded patch base and permits a safe retry.
Manual conflict resolution followed by `git am --continue` can be exported.

Private source state now requires schema 2. Existing schema-1 state is
incompatible: status reports a clean existing workspace as stale, and prepare,
sync, and export refuse to reinterpret or migrate it. Preserve any developer
work before explicitly removing the workspace; preparation can then recreate it
with the new schema. No implicit discard occurs.

## Patch workflow

Create:

```text
<component>/patches/
  series
  my-change.patch
```

`series` contains one logical patch filename per line.

Prepare applies the series:

```sh
bspkit prepare <component>
```

For active development, edit `work/<component>/` normally and create Git
commits. Amend, rebase, reorder, and fixup/squash workflows are supported.

Export committed work back to the version-controlled patch series:

```sh
bspkit patch export <component>
git diff
```

The consumer repository's normal `git diff` is the review surface. Export starts
at the recorded patch base for all source types. If the source or prepare-hook
inputs have changed since that base was established, restore those old inputs
before exporting committed work.

After editing the version-controlled patch series directly:

```sh
bspkit patch sync <component>
```

`patch sync` is non-destructive and refuses to discard local committed or dirty
work.

## Build one component

Provide an executable `<component>/build` script. It is called directly with no
arguments, with `work/<component>/` as its working directory. Its shebang selects
the interpreter. Component-specific configuration and compilation belong in that
script.

The hook inherits the process environment and receives these values from bspkit:

| Variable | Value |
| --- | --- |
| `BSPKIT_PROJECT_ROOT` | absolute project root |
| `BSPKIT_WORK_ROOT` | absolute `work/` path |
| `BSPKIT_COMPONENT_NAME` | selected component name |
| `BSPKIT_COMPONENT_DIR` | absolute project-owned component directory |
| `BSPKIT_COMPONENT_WORKTREE` | absolute selected worktree path |

For a component whose upstream default make target is appropriate, a hook may be:

```sh
#!/bin/sh
set -eu
make
```

Then run:

```sh
chmod +x u-boot/build
bspkit build u-boot
```

Build prepares/fetches only the selected component as needed. A repeated invocation
skips the hook when source, hook, and optional fingerprint inputs are unchanged.
Hook stdout/stderr are streamed normally; a non-zero exit fails the command and
leaves no current build stamp. A missing hook is a successful no-op.

To rerun a current build, use either equivalent option:

```sh
bspkit build -f u-boot
bspkit build --force u-boot
```

Force bypasses only the current-build skip. Normal fetch/prepare prerequisites and
hook and fingerprint validation still apply; already current preparation is reused.
Before hook execution, the previous successful build stamp is invalidated. Success
atomically records the inputs captured before execution. Execution or publication
failure leaves no successful build stamp. A missing hook still records successful no-op
state. With unchanged inputs, a successful forced rebuild has the same `build_id`.
Force does not clean generated files, expand dependencies, force source recreation,
or add toolchain/environment tracking.

Build products stay in the component worktree. Git source identity uses the
recorded patch base and ordered patchset. It also includes committed local
history and effective uncommitted source changes, but
excludes ignored output. A prepare hook therefore participates through its
post-hook source tree. Previous serialized source identities are not preserved.
Ensure the source repository's `.gitignore` covers generated files.

Build state is private under `.bspkit/state/`; no shared `out/` is required.
Toolchain/environment changes, ignored `.config` edits, and dependency build
results do not automatically invalidate builds unless explicitly covered by the
optional build fingerprint described below. Builds of dependencies must be invoked
manually, for example:

```sh
bspkit build tf-a
bspkit build u-boot
```

This does not yet propagate dependency changes into the dependent build identity.

## Build fingerprints

An optional executable `<component>/build.fingerprint` supplies a SHA-256 digest
of additional build inputs, such as an ignored U-Boot `.config`. It runs directly
with zero arguments, worktree CWD, inherited environment, and the same `BSPKIT_*`
values as `build`. For example, save this as `u-boot/build.fingerprint`:

```python
#!/usr/bin/env python3
import hashlib
from pathlib import Path

print(hashlib.sha256(Path(".config").read_bytes()).hexdigest())
```

```sh
chmod +x u-boot/build.fingerprint
bspkit prepare u-boot
# Run the consumer's configuration step in work/u-boot to create .config.
bspkit build u-boot
bspkit status
```

This example requires `.config` to exist before build inspection; a missing file
fails fingerprint evaluation. Changing an ignored `.config` now makes the build
stale and triggers rebuilding, without changing source identity or preparation.
Core does not interpret `.config` or any other component-specific input.

On exit code zero, stdout must contain exactly 64 ASCII hexadecimal characters,
optionally followed by one LF. Uppercase and lowercase are equivalent. No
filenames, prefixes, spaces, CRLF, or additional logs are allowed. Capture is
bounded and oversized output fails. Stderr is forwarded for diagnostics and
does not affect identity. The returned digest is decoded, rather than rehashed
as textual output.

Optional `<component>/build.fingerprint.d/` contains project-owned hook inputs.
Bspkit recursively hashes relative paths, entry kinds, exact file contents,
file/directory permission bits, symlink targets, and empty directories. It ignores
timestamps and ownership and never follows nested symlinks. The directory root
must be a real directory; a root symlink or unsupported special object fails
inspection. Missing and present-empty directories differ. When the hook is
absent, this entire directory is ignored, including invalid entries and changes.

Build identity includes fingerprint-hook contents, its data-tree identity, and
the returned digest. Hook or data changes invalidate builds even if the returned
digest stays unchanged. Fingerprint inputs are build-only and do not participate
in source identity. `bspkit.build-id.v1`, project schema 1, and private build-state
schema 1 are retained; older successful build stamps require rebuilding.

Normal builds, forced builds, and status share fingerprint/currentness evaluation.
All inputs are captured before build execution and never recalculated to stamp
completion. If the build changes a fingerprinted input, the next inspection can
report it stale. A fingerprint validation, execution, output, or data-read failure
prevents building, preserves the previous successful stamp, and makes status exit
nonzero. Once an actual build starts, the normal build transaction still applies.

Fingerprint hooks must be read-only. They execute during status for eligible
prepared workspaces, including those without a successful build stamp. Bspkit
trusts the hook author and does not enforce a sandbox. Missing or unprepared
workspaces are not fingerprinted. This feature adds no lifecycle phase or CLI
command and does not implement dependency build-ID propagation or binary builds.

## Cleaning (planned)

Run component clean hooks and invalidate build state:

```sh
bspkit clean [COMPONENT...]
```

Remove generated/untracked worktree files:

```sh
bspkit distclean [COMPONENT...]
```

Explicitly discard developer work and recreate canonical sources:

```sh
bspkit distclean --recreate <component>
```

The last command is destructive: it removes local commits, tracked changes,
untracked files, and ignored build output.

## Status

Run:

```sh
bspkit status
```

The command's inspection is read-only and reports one summary row per configured
component. It may execute trusted `build.fingerprint` hooks under their read-only
contract; fingerprint failures produce a nonzero exit without changing saved state:

```text
COMPONENT  CACHE  WORKSPACE  BUILD
```

For Git sources, `CACHE=yes` means the pinned commit is available in the local
mirror. For binary sources, it means a non-empty configured SHA-256 matches the
cached bytes, compared case-insensitively. Missing caches, empty binary checksums,
and mismatched binary bytes report `no`. Status never downloads or repairs cache
entries; actual local inspection/read errors fail the command. Native cache is
`yes` only when fetch inputs and the downloaded tree verify, `no` when stale or
missing, and `n/a` when there is no fetch hook.

Workspace state is summarized as `missing`, `dirty`, `stale`, `local`, or
`canonical` for all source types, with precedence in that order. Build state is
`missing`, `current`, or `stale` for Git and native components. Prepared binary
components normally report `BUILD=missing`; evaluation of a usable saved binary build stamp
fails clearly because binary build currentness remains deferred.

Detailed single-component status is planned:

```sh
bspkit status <component>
```

Explicit multi-component status selection is not currently specified.

## Generated Makefile

The generated Makefile is only a thin bspkit adapter. It currently provides:

```sh
make fetch
make prepare
```

`make all` aliases `prepare`. Invoke `bspkit build <component>` explicitly.

## bspkit development

From the repository root:

```sh
just format
just format-check
just lint
just test
just run --help
just pip-package
```

Black and flake8 are development tools; bspkit has no runtime Python
dependencies.

`just test` runs the full suite by default. Pass unittest arguments to select
modules, individual tests, or discovery filters:

```sh
just test test_git_cache -v
just test test_git_cache.GitCacheTests.test_direct_sha_shallow_and_legacy_mirror
just test discover -s tests -p 'test_git_cache.py' -v
```

`just pip-package` requires the development tools in `requirements-dev.txt`.

It writes a wheel and source distribution to `dist/`; see package validation below.

To run the checkout CLI from another consumer project:

```sh
cd /path/to/consumer
just --justfile /path/to/bspkit/justfile run build u-boot
```

`just run` preserves the invocation directory and sets `PYTHONPATH` to the
implementation checkout. Quoted paths and arguments may contain spaces.

Developer recipes are checked directly, outside the Python product test suite.

## Package validation and releases

bspkit is licensed under [MIT](LICENSE). Release versions come from annotated
`vX.Y.Z` Git tags through Hatchling and hatch-vcs; `bspkit --version` reads the
installed distribution metadata. No version edit in `pyproject.toml` is needed.
Untagged Git builds use hatch-vcs development versions, for example
`1.2.4.dev1+g<commit>` after `v1.2.3`. A repository without tags also builds a
development version from the fallback base plus commit distance/hash. Without
usable VCS or sdist metadata, the fallback is `0.0.0.dev0`. A wheel rebuilt from
an extracted sdist preserves its embedded version even without `.git`.

Install the development tools, then run the same recipes used by CI:

```sh
python3 -m pip install -r requirements-dev.txt
just check
just pip-package
just package-check
```

`just check` reuses `format-check`, `lint` and `test`. Packaging runs
`python3 -m build`, producing one wheel and one sdist in `dist/`. Move any existing
`dist/` aside first; builds refuse to mix new outputs with old or recovery
artifacts. `package-check` runs Twine, checks names/metadata and allowed archive
contents, then installs each artifact in its own fresh virtual environment
outside the checkout. It checks import, both CLI entry points, version reporting
and project creation, and writes `dist/manifest.json` with exact filenames and
SHA-256 hashes. The development environment needs network access for isolated
build dependencies and Git for the creation smoke test.

MR and default-branch CI run quality and packaging checks on Python 3.11 and
3.13 without requesting PyPI credentials. After merging into the protected
default branch, an authorized maintainer releases with an annotated tag:

```sh
git tag -a v0.2.0 -m "Release v0.2.0"
git push origin v0.2.0
```

The tag pipeline rejects malformed, lightweight or unprotected tags, commits
outside default-branch history, and artifact versions that differ from the tag.
These are pipeline checks; protected-tag settings control who may push a tag.
After both Python versions pass, publication automatically consumes the validated
Python 3.13 artifacts using PyPI Trusted Publishing in the `pypi` environment.
There is no manual publish job or stored PyPI API token. One-time GitLab/PyPI
settings are tracked in the implementation MR's setup comment.

Publication compares the expected filenames and SHA-256 hashes with the public
PyPI JSON API. It uploads only missing files, skips exact matches, and fails on
conflicting hashes, unexpected files or inaccessible/invalid metadata. It then
checks all remote hashes with bounded propagation retries. A separate job verifies
a fresh install from public PyPI before creating the GitLab Release. Release
assets use verified copies in the GitLab generic package registry, under a
pipeline-specific version, plus the PyPI version link. Existing consistent
releases are completed or accepted; contradictory data fails visibly.

To recover, retry the failed downstream job in the **same tag pipeline**, using
its retained `dist/` and manifest. Both Python packaging jobs keep their artifacts
without expiration. The publish/release jobs never rebuild. If the verified
artifacts are unavailable, stop for a deliberate recovery decision; rebuilding
and blind `--skip-existing` uploads are not recovery procedures. Preserve the
generic registry packages linked by releases when configuring cleanup policies.
