Metadata-Version: 2.4
Name: compilecode
Version: 0.3.0
Summary: Compile a Python project to native modules with Nuitka; no key, no runtime decryption.
Requires-Python: >=3.10
Description-Content-Type: text/markdown
Requires-Dist: nuitka>=4.3.1

# compilecode

`compilecode` compiles a project's Python modules to native `.so` extension
modules with [Nuitka](https://nuitka.net). The original `.py` sources and
bytecode are removed afterwards. There is no key and no runtime decryption:
the compiled modules are imported directly by Python.

## Usage

Drop-in one-liner, run from the project root:

```bash
python -m compilecode .
```

Or, after installing the package (`pip install -e .`):

```bash
compilecode path/to/project
```

Options:

- `--strip-docstrings` removes docstrings from the compiled modules.
- `--jobs N` / `-j N` compiles that many modules in parallel. Each module is
  an independent Nuitka subprocess, so compiling a tree is roughly N× faster
  on an N-core box. The default is `min(4, CPU count)` — capped because every
  job is a Nuitka + C-compiler process and uses real RAM. Pass `--jobs 1` to
  force serial compilation.
- `--nuitka-jobs N` caps how many C-compiler (`cc1`) jobs Nuitka forks *per
  module*. Nuitka's default is all cores, and on large modules each `cc1` can
  use hundreds of MB — the usual cause of an out-of-memory kill. Lower this to
  cap peak memory.
- `--low-memory` is the one switch for a small build host (~4 GB): it compiles
  one module at a time, with one `cc1` job, and turns on Nuitka's own
  low-memory mode. Slower, but it keeps the build from being OOM-killed.
  Combine it with a `.compilecodeignore` (below) that skips migrations,
  settings, and tests to cut the load further.
- `--cache-dir PATH` enables an opt-in, source-hash compile cache. Each module
  is keyed by its source content, Nuitka version, Python ABI (EXT_SUFFIX),
  compile name, is-init and the `--strip-docstrings` flag (plus the cache
  format/logic version). `--jobs`, `--nuitka-jobs` and `--low-memory` do not
  affect the key. Unchanged modules are restored from `PATH` instead of being
  recompiled, and the run prints `cache: H hits, M misses`. Without the flag
  nothing is cached. Cold-build caveat: the first build (empty cache) compiles
  everything and is no faster; the speedup appears on later builds.
- `--strict` also makes a **compile failure** exit non-zero. The default is
  **best-effort**: a module that cannot be compiled (for example a file that is
  not valid Python) is left as readable source, listed at the end of the run,
  and the build still succeeds — one bad file does not sink the whole image
  build. Use `--strict` for a hard CI gate. A **leaked** source file (a module
  that compiled but whose `.py` was not removed) is a different matter: it means
  the image would ship readable source, so it is **always fatal**, with or
  without `--strict`.

> **Running out of memory?** A big Django/FastAPI project compiled on a small
> host can exhaust RAM: Nuitka forks several `cc1` C-compilers (hundreds of MB
> each) per large module, and `--jobs > 1` multiplies that across modules. If
> the build grinds for a long time and then dies, reach for `--low-memory`,
> give the builder more RAM, or skip non-proprietary files via
> `.compilecodeignore`.

By default the exit code is `0` even when some modules fail to compile
(best-effort — the failures are reported and left as source). Pass `--strict`
to exit `1` on a compile failure too. A leaked source file (compiled but not
removed) always exits `1`, regardless of `--strict`.

## Drop-in swap (Docker)

The reference `references/Dockerfile` keeps the same `compileit.sh` one-liner
against `$DOCKERHOME`, so existing setups swap in without other changes. The line now
compiles the project to `.so` modules with Nuitka instead of encrypting it.
No key is generated or shipped in the image, so there is nothing to store or
rotate. The builder stage installs `nuitka`, and the runtime stage no longer
installs `git`, `vim` or `dos2unix`.

### Persisting the cache between builds (registry-agnostic)

`--cache-dir` reads and writes a local directory; in CI that directory travels
between builds as a small standard image. The reference Dockerfile takes it as
the `CACHE_IMAGE` build arg (default `busybox`, an empty base for the first
build):

```dockerfile
ARG CACHE_IMAGE=busybox
FROM ${CACHE_IMAGE} AS cache
# ... in the builder:
RUN mkdir -p /compilecache
COPY --from=cache /cache/ /compilecache/
RUN .../compileit.sh --cache-dir /compilecache $DOCKERHOME
```

`CACHE_IMAGE` is an ordinary `<registry>/<repo>:<tag>` reference supplied as a
build arg or pipeline variable. Because persistence is only `docker pull` and
`docker push` of a standard image, it works unchanged on Docker Hub, an
internal registry, GHCR or ACR; nothing in `compilecode` or the Dockerfile
hardcodes a registry. Wrap the build with two steps:

1. Before: `docker pull <registry>/<repo>:<tag>`. Only if the pull succeeds,
   build with `--build-arg CACHE_IMAGE=<registry>/<repo>:<tag>`. If it fails
   (tag not pushed yet), build with the `busybox` default (omit `CACHE_IMAGE`
   or pass `--build-arg CACHE_IMAGE=busybox`); that first build runs cold.
   Never pass a tag that does not exist: `FROM` on a missing tag fails the
   build. The Dockerfile normalizes a cold/empty base to an empty `/cache`, so the
   build works on any builder (legacy or BuildKit).
2. After: the final image is the runtime stage, so the Dockerfile has an export
   stage, `cache-out`, holding the populated `/compilecache`. Build and push it
   as the next build's `CACHE_IMAGE`:
   `docker build --target cache-out -t <registry>/<repo>:<tag> . && docker push <registry>/<repo>:<tag>`

## Choosing what gets compiled

Both control files sit in the project root, hold one glob per line (`fnmatch`:
`*`, `?`, `[...]`; `*` spans `/`), and ignore blank lines and `#` comments.

**`.compilecodeignore` — skip-list (default).** Compile everything *except* the
listed patterns, which ship as plain Python. A line starting with `!` forces a
file to be compiled even if another rule would skip it. Best when most of the
tree is proprietary and only a few files should stay readable. This is also the
max-protection path: leave `.compilecodeinclude` out and the whole tree compiles
except what you list here.

**`.compilecodeinclude` — allowlist (opt-in).** If this file has at least one
rule, the tree runs in **include mode**: compile *only* the listed files; every
other file ships as readable source. Best when only a few files are IP and the
rest is boilerplate. Entry scripts (`manage.py`, `__main__`) are never compiled,
even if a pattern matches them. (An empty or comments-only file is inert and
falls back to skip-list mode.)

**If both files exist, `.compilecodeinclude` wins** — only its files compile and
`.compilecodeignore` is disregarded. The build prints a notice when this
happens, plus a summary line so under-protection is visible by the numbers:

```
compilecode: mode=include | compiled 4 | skipped 229 | failed 0 | leaked 0
```

## Packaging

`compileit.sh` ships inside the installed `compilecode/` package. It runs
`python -m compilecode` and honours the `PYTHON` environment variable to pick
the interpreter.

## Releasing to PyPI / TestPyPI

`release.sh` versions, builds, and publishes the package. Everything runs
**inside the docker compose `test` container** — you only need Docker on the
host; `build`, `twine`, and `setuptools_scm` are never installed locally.

The version lives in **git tags** (via `setuptools_scm`): you never edit a
version by hand. `release.sh` computes the next one, tags the commit, builds
the sdist + wheel, uploads, and pushes the commit and tag back.

```bash
./release.sh <testpypi|pypi> [version] [options]
```

**Version selection** (pick at most one; default is a patch bump):

| You pass            | Version used                 | Example         |
| ------------------- | ---------------------------- | --------------- |
| *(nothing)*         | patch bump of the latest tag | `0.1.3 → 0.1.4` |
| *(nothing)*, but HEAD is already on a tag | that tag is **reused** (so `testpypi` → `pypi` publishes the same version) | `0.1.3` |
| `--minor`           | minor bump, patch reset      | `0.1.3 → 0.2.0` |
| `--major`           | major bump, minor+patch reset | `0.1.3 → 1.0.0` |
| `--version X.Y.Z`   | exactly that                 | `2.5.1`         |

**Options**

- `--dry-run` — compute the version, build, and `twine check` only. Makes
  **no git changes**, uploads nothing, pushes nothing. Run this first.
- `--no-push` — publish but skip `git push --follow-tags`.

**Authentication** (only needed for a real upload). Either works:

1. A `.pypirc` with `[pypi]` / `[testpypi]` sections (the default when no
   token env is set). It is searched as `$PYPIRC`, then `./.pypirc`, then
   `~/.pypirc`, mounted read-only into the container, and used via
   `twine upload --repository <target>`. The `[testpypi]` section must include
   its `repository = https://test.pypi.org/legacy/` line:

   ```ini
   [pypi]
     username = __token__
     password = pypi-AgEI...
   [testpypi]
     repository = https://test.pypi.org/legacy/
     username = __token__
     password = pypi-AgEI...
   ```

2. `export TWINE_API_TOKEN="pypi-...."` — a single token that takes precedence
   over `.pypirc` (matches twine's own env-over-file rule). Handy for CI.

Create tokens at <https://pypi.org/manage/account/token/> (or test.pypi.org).

**First release**

```bash
./release.sh testpypi --version 0.1.0 --dry-run   # rehearse, no side effects
./release.sh testpypi --version 0.1.0             # tag v0.1.0, publish to TestPyPI
./release.sh pypi     --version 0.1.0             # same version → real PyPI
```

**Routine release** after merging new commits:

```bash
./release.sh testpypi --dry-run    # shows the next patch, e.g. 0.1.1
./release.sh testpypi              # tag v0.1.1, publish to TestPyPI, push
./release.sh pypi                  # HEAD on v0.1.1 → reuse → publish to PyPI
```

Verify a TestPyPI upload installs (its deps still come from real PyPI):

```bash
pip install --index-url https://test.pypi.org/simple/ \
    --extra-index-url https://pypi.org/simple/ compilecode
```

## Try it end-to-end (smoke test)

`references/examples/sample-project/` is a small app that exercises the tricky
cases (nested relative imports, a dataclass, a data file, an entry script). Its
`smoke.sh` builds the real two-stage Docker image — compile under full
`python:3.13`, run under `python:3.13-slim` — and asserts the compiled app
runs while its `.py` sources are gone:

```bash
./references/examples/sample-project/smoke.sh
```
