Metadata-Version: 2.4
Name: compilecode
Version: 0.4.0
Summary: Compile a Python project to native modules with Nuitka; no key, no runtime decryption.
Requires-Python: >=3.10
Description-Content-Type: text/markdown
Requires-Dist: nuitka>=4.3.1

# compilecode

`compilecode` compiles a project's Python modules to native `.so` extension
modules with [Nuitka](https://nuitka.net). The original `.py` sources and
bytecode are removed afterwards. There is no key and no runtime decryption:
the compiled modules are imported directly by Python.

## Usage

Drop-in one-liner, run from the project root:

```bash
python -m compilecode .
```

Or, after installing the package (`pip install -e .`):

```bash
compilecode path/to/project
```

Options:

- `--strip-docstrings` removes docstrings from the compiled modules.
- `--jobs N` / `-j N` compiles that many modules in parallel. Each module is
  an independent Nuitka subprocess, so compiling a tree is roughly N× faster
  on an N-core box. The default is `min(4, CPU count)` — capped because every
  job is a Nuitka + C-compiler process and uses real RAM. Pass `--jobs 1` to
  force serial compilation.
- `--nuitka-jobs N` caps how many C-compiler (`cc1`) jobs Nuitka forks *per
  module*. Nuitka's default is all cores, and on large modules each `cc1` can
  use hundreds of MB — the usual cause of an out-of-memory kill. Lower this to
  cap peak memory.
- `--low-memory` is the one switch for a small build host (~4 GB): it compiles
  one module at a time, with one `cc1` job, and turns on Nuitka's own
  low-memory mode. Slower, but it keeps the build from being OOM-killed.
  Combine it with a `.compilecodeignore` (below) that skips migrations,
  settings, and tests to cut the load further.
- `--cache-dir PATH` enables an opt-in, source-hash compile cache. Each module
  is keyed by its source content, Nuitka version, Python ABI (EXT_SUFFIX),
  compile name, is-init and the `--strip-docstrings` flag (plus the cache
  format/logic version). `--jobs`, `--nuitka-jobs` and `--low-memory` do not
  affect the key. Unchanged modules are restored from `PATH` instead of being
  recompiled, and the run prints `cache: H hits, M misses`. Without the flag
  nothing is cached. Cold-build caveat: the first build (empty cache) compiles
  everything and is no faster; the speedup appears on later builds.
- `--strict` also makes a **compile failure** exit non-zero. The default is
  **best-effort**: a module that cannot be compiled (for example a file that is
  not valid Python) is left as readable source, listed at the end of the run,
  and the build still succeeds — one bad file does not sink the whole image
  build. Use `--strict` for a hard CI gate. A **leaked** source file (a module
  that compiled but whose `.py` was not removed) is a different matter: it means
  the image would ship readable source, so it is **always fatal**, with or
  without `--strict`.

> **Running out of memory?** A big Django/FastAPI project compiled on a small
> host can exhaust RAM: Nuitka forks several `cc1` C-compilers (hundreds of MB
> each) per large module, and `--jobs > 1` multiplies that across modules. If
> the build grinds for a long time and then dies, reach for `--low-memory`,
> give the builder more RAM, or skip non-proprietary files via
> `.compilecodeignore`.

By default the exit code is `0` even when some modules fail to compile
(best-effort — the failures are reported and left as source). Pass `--strict`
to exit `1` on a compile failure too. A leaked source file (compiled but not
removed) always exits `1`, regardless of `--strict`.

## Drop-in swap (Docker)

The reference `references/Dockerfile` runs the `compilecode` command (installed
on `PATH` by `pip install compilecode`) against `$DOCKERHOME`, so existing setups
swap in without other changes. The line compiles the project to `.so` modules
with Nuitka instead of encrypting it.
No key is generated or shipped in the image, so there is nothing to store or
rotate. The builder stage installs `nuitka`, and the runtime stage no longer
installs `git`, `vim` or `dos2unix`.

### Persisting the cache between builds (registry-agnostic)

`--cache-dir` reads and writes a local directory; in CI that directory travels
between builds as a small standard image. The reference Dockerfile takes it as
the `CACHE_IMAGE` build arg (default `busybox`, an empty base for the first
build):

```dockerfile
ARG CACHE_IMAGE=busybox
FROM ${CACHE_IMAGE} AS cache
# ... in the builder:
RUN mkdir -p /compilecache
COPY --from=cache /cache/ /compilecache/
RUN compilecode --cache-dir /compilecache $DOCKERHOME
```

`CACHE_IMAGE` is an ordinary `<registry>/<repo>:<tag>` reference supplied as a
build arg or pipeline variable. Because persistence is only `docker pull` and
`docker push` of a standard image, it works unchanged on Docker Hub, an
internal registry, GHCR or ACR; nothing in `compilecode` or the Dockerfile
hardcodes a registry. Wrap the build with two steps:

1. Before: `docker pull <registry>/<repo>:<tag>`. Only if the pull succeeds,
   build with `--build-arg CACHE_IMAGE=<registry>/<repo>:<tag>`. If it fails
   (tag not pushed yet), build with the `busybox` default (omit `CACHE_IMAGE`
   or pass `--build-arg CACHE_IMAGE=busybox`); that first build runs cold.
   Never pass a tag that does not exist: `FROM` on a missing tag fails the
   build. The Dockerfile normalizes a cold/empty base to an empty `/cache`, so the
   build works on any builder (legacy or BuildKit).
2. After: the final image is the runtime stage, so the Dockerfile has an export
   stage, `cache-out`, holding the populated `/compilecache`. Save it as the
   next build's `CACHE_IMAGE` with the shipped one-command helper, run on the
   host/CI right after the build (same job, so Docker's layer cache is warm and
   it does not recompile):

   ```bash
   compilecode-cachepush <registry>/<repo>:<tag>
   ```

   `compilecode-cachepush` is installed on `PATH` by `pip install compilecode`
   (equivalently `python -m compilecode.cachepush`, or the bundled
   `compilecode/cachepush.sh` directly). It takes `<CACHE_IMAGE> [BUILD_CONTEXT]`,
   builds the `cache-out` stage and
   pushes it in one step. It auto-seeds from the existing cache image when it
   exists (so the cache accumulates) and falls back to `busybox` on the cold
   first build, so the same command works every run. A probe that is *not* a
   clean "image not found" (expired token, network blip, old docker) is fatal
   rather than a silent cold seed, so a hiccup never overwrites a good cache.
   Env overrides: `DOCKERFILE`, `CACHE_TARGET`, `CACHE_FROM`, `DOCKER`. `docker
   login` to the registry must already have happened. Equivalent to, by hand:
   `docker build --target cache-out --build-arg CACHE_IMAGE=<seed> -t <registry>/<repo>:<tag> . && docker push <registry>/<repo>:<tag>`

   For deterministic CI (e.g. concurrent pipelines, or a main build run cold on
   purpose), set `CACHE_FROM` to exactly the `CACHE_IMAGE` the main build used,
   so the export reuses its layers instead of recompiling. On a *persistent*
   host, add `--pull` to the main build so `FROM ${CACHE_IMAGE}` fetches the
   newest cache rather than a stale local tag.

## Choosing what gets compiled

Two optional control files decide which modules are compiled. Each holds one
glob per line; blank lines and `#` comments are ignored. Which file you use
selects the **mode**.

### The two modes

**`.compilecodeignore` — skip-list (the default).** Compile everything *except*
the listed patterns, which ship as plain Python. A line starting with `!` forces
a file to be compiled even if another rule would skip it. Best when most of the
tree is proprietary and only a few files should stay readable. Leaving
`.compilecodeinclude` out and listing nothing to skip is the **max-protection**
path: the whole tree compiles.

**`.compilecodeinclude` — allowlist (opt-in).** If this file has at least one
rule, the tree runs in **include mode**: compile *only* the listed files; every
other file ships as readable source. Best when only a few files are IP and the
rest is boilerplate. The include file is a pure allowlist — `!` has no special
meaning in it. Entry scripts (`manage.py`, `__main__`) are never compiled, even
if a pattern matches them (a compiled entry point breaks `python manage.py`). An
empty or comments-only include file is **inert** and falls back to skip-list
mode.

**If both files exist, `.compilecodeinclude` wins** — only its files compile and
`.compilecodeignore` is disregarded. The build prints a notice when this
happens, plus a summary line so under-protection is visible by the numbers:

```
compilecode: mode=include | compiled 4 | skipped 229 | failed 0 | leaked 0
```

### Where the file must sit

The file is read from **exactly the directory you pass to the command** — the
*compile root*. Discovery is a single lookup at that root:

- It is **not** hierarchical like `.gitignore`: a rules file one, two, or three
  levels below the root is **never read**, and per-folder files are not merged.
- Give the command **multiple directories** and each is its own root, so each
  needs its own file. To drive everything from **one** file instead, point the
  command at the common parent directory and let globs (below) reach down.
- There is no flag to name the file elsewhere; placement at the root is the
  whole mechanism.

### Pattern syntax

Patterns use `fnmatch` and are matched against each file's path **relative to
the compile root** (never absolute, never relative to your shell's CWD):

- `*` matches any run of characters **including `/`** (it spans directories);
  `?` matches one character; `[abc]` / `[!abc]` is a character class.
- Patterns are **anchored at the root**. `core/*.py` does **not** match
  `app/core/billing.py` — the path starts with `app/`. To match a folder by
  name at any depth, lead with `*/`: `*/migrations/*.py`, `*/tests/*.py`.
- **No leading slash, no absolute paths** — `/app/core/*.py` matches nothing.
- `#` starts a comment; blank lines are ignored.
- `!` (ignore mode only) forces a match to compile, and it is checked *first* —
  so `!app/core/*.py` force-compiles everything in `app/core/`, including a file
  you tried to skip in the same list; narrow the `!` to specific files instead.

Example — allowlist a few jewels buried in a boilerplate tree, from one file at
the root:

```
# .compilecodeinclude
app/core/*.py
app/billing/services.py
app/*/pricing/*.py
```

## Hybrid: compile the IP, encrypt the rest

In **include mode** the files you don't list ship as readable `.py` — fine for
genuine boilerplate, but if you'd rather not leave *anything* in the clear, you
can chain a second pass that scrambles the leftovers. The idea:

- **`compilecode` (Nuitka)** turns your IP into native `.so` — no key, nothing
  to decrypt, source removed. This is the real protection.
- **`encryptcode` (Fernet)** then sweeps every remaining `.py` — boilerplate,
  settings, even `manage.py` — into an encrypted stub, so a casual reader can no
  longer `cat` the file and read it.

> **Read this before you rely on it.** The `encryptcode` layer is **obfuscation,
> not cryptography.** It ships the Fernet key *inside the same image* and the
> program decrypts itself at import time, so anyone with the image can recover
> the original source in seconds (intercept the decrypt, or pull the key out of
> the shipped module). It raises the bar from "trivial" (`cat`) to "mildly
> annoying" for a normal user — nothing more. **Never put a secret, credential,
> or anything that actually matters in a file that is only `encryptcode`-d.**
> Real IP belongs in the `.so` layer; use the encrypt pass only to avoid shipping
> *plaintext boilerplate*, with eyes open about what it does and doesn't buy.

### Recipe (order matters)

Run `compilecode` **first** (it compiles the IP and removes those sources), then
`encryptcode` over the same tree (it encrypts the `.py` files that are left):

```bash
# 1. Compile only the IP to .so (allowlist the jewels). Their .py is removed;
#    boilerplate + entry scripts remain as readable .py for now.
compilecode /home/backend             # with a .compilecodeinclude at that root

# 2. Encrypt everything still readable (boilerplate, settings, manage.py, ...).
encryptit.sh /home/backend
```

What ends up in the image:

| File kind | After step 1 | After step 2 |
| --------- | ------------ | ------------ |
| Your IP (allowlisted) | native `.so` | native `.so` (untouched) |
| Boilerplate / settings | readable `.py` | encrypted stub |
| Entry scripts (`manage.py`) | readable `.py` | encrypted stub (still runs) |

Notes:

- Entry scripts stay runnable — `encryptcode` decrypts and executes them at
  import time, so `python manage.py …` still works on the encrypted stub.
- `encryptcode` only touches `.py`; the `.so` files from step 1 are left alone.
- The split is deliberate: **strong** protection (compiled, keyless) on the code
  that matters, **speed-bump** obfuscation on the rest — so nothing ships as
  plaintext, while no one is misled into trusting the weak layer with real IP.

## Packaging

`pip install compilecode` puts two commands on `PATH`:

- `compilecode <paths> [options]` — compile a tree (the main command; use this
  in a Dockerfile `RUN` line, no path gymnastics).
- `compilecode-cachepush <CACHE_IMAGE> [CONTEXT]` — build and push the compile
  cache image (host/CI).

The bundled scripts `compileit.sh` and `cachepush.sh` still ship inside the
installed `compilecode/` package for anyone who wants to call them directly;
`compileit.sh` runs `python -m compilecode` and honours the `PYTHON` environment
variable to pick the interpreter.

## Releasing to PyPI / TestPyPI

`release.sh` versions, builds, and publishes the package. Everything runs
**inside the docker compose `test` container** — you only need Docker on the
host; `build`, `twine`, and `setuptools_scm` are never installed locally.

The version lives in **git tags** (via `setuptools_scm`): you never edit a
version by hand. `release.sh` computes the next one, tags the commit, builds
the sdist + wheel, uploads, and pushes the commit and tag back.

```bash
./release.sh <testpypi|pypi> [version] [options]
```

**Version selection** (pick at most one; default is a patch bump):

| You pass            | Version used                 | Example         |
| ------------------- | ---------------------------- | --------------- |
| *(nothing)*         | patch bump of the latest tag | `0.1.3 → 0.1.4` |
| *(nothing)*, but HEAD is already on a tag | that tag is **reused** (so `testpypi` → `pypi` publishes the same version) | `0.1.3` |
| `--minor`           | minor bump, patch reset      | `0.1.3 → 0.2.0` |
| `--major`           | major bump, minor+patch reset | `0.1.3 → 1.0.0` |
| `--version X.Y.Z`   | exactly that                 | `2.5.1`         |

**Options**

- `--dry-run` — compute the version, build, and `twine check` only. Makes
  **no git changes**, uploads nothing, pushes nothing. Run this first.
- `--no-push` — publish but skip `git push --follow-tags`.

**Authentication** (only needed for a real upload). Either works:

1. A `.pypirc` with `[pypi]` / `[testpypi]` sections (the default when no
   token env is set). It is searched as `$PYPIRC`, then `./.pypirc`, then
   `~/.pypirc`, mounted read-only into the container, and used via
   `twine upload --repository <target>`. The `[testpypi]` section must include
   its `repository = https://test.pypi.org/legacy/` line:

   ```ini
   [pypi]
     username = __token__
     password = pypi-AgEI...
   [testpypi]
     repository = https://test.pypi.org/legacy/
     username = __token__
     password = pypi-AgEI...
   ```

2. `export TWINE_API_TOKEN="pypi-...."` — a single token that takes precedence
   over `.pypirc` (matches twine's own env-over-file rule). Handy for CI.

Create tokens at <https://pypi.org/manage/account/token/> (or test.pypi.org).

**First release**

```bash
./release.sh testpypi --version 0.1.0 --dry-run   # rehearse, no side effects
./release.sh testpypi --version 0.1.0             # tag v0.1.0, publish to TestPyPI
./release.sh pypi     --version 0.1.0             # same version → real PyPI
```

**Routine release** after merging new commits:

```bash
./release.sh testpypi --dry-run    # shows the next patch, e.g. 0.1.1
./release.sh testpypi              # tag v0.1.1, publish to TestPyPI, push
./release.sh pypi                  # HEAD on v0.1.1 → reuse → publish to PyPI
```

Verify a TestPyPI upload installs (its deps still come from real PyPI):

```bash
pip install --index-url https://test.pypi.org/simple/ \
    --extra-index-url https://pypi.org/simple/ compilecode
```

## Try it end-to-end (smoke test)

`references/examples/sample-project/` is a small app that exercises the tricky
cases (nested relative imports, a dataclass, a data file, an entry script). Its
`smoke.sh` builds the real two-stage Docker image — compile under full
`python:3.13`, run under `python:3.13-slim` — and asserts the compiled app
runs while its `.py` sources are gone:

```bash
./references/examples/sample-project/smoke.sh
```
