Metadata-Version: 2.4
Name: tokless
Version: 0.3.0
Summary: Lossless token compression for OKF knowledge and agent skills.
Author-email: Karthick Raja M <Karthickrajam18@gmail.com>
License-Expression: MIT
Project-URL: Homepage, https://github.com/karthyick/tokless
Keywords: okf,context,llm,agents,tokens,compression
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: tiktoken
Requires-Dist: tiktoken>=0.7; extra == "tiktoken"
Provides-Extra: dev
Requires-Dist: pytest>=7.0.0; extra == "dev"
Requires-Dist: build>=0.10.0; extra == "dev"
Requires-Dist: twine>=4.0.0; extra == "dev"
Requires-Dist: tiktoken>=0.7; extra == "dev"
Dynamic: license-file

# tokless

Lossless token compression for OKF knowledge and agent skills, plus plain markdown to OKF.

Docs and skills loaded into an LLM's context carry markup the model does not need. tokless
removes only what small models were measured to survive (one mode, see below) and proves,
per document, that no fact was lost.

```text
plain .md ──► convert ──► OKF frontmatter added ─┐
OKF / SKILL.md ──────────────────────────────────┴─► compress ──► gate ──► same facts, fewer tokens
                                                     (no model call)  │
                                       gate fails or no saving ───────┴─► file kept as it was
```

## Install

```bash
pip install tokless              # zero dependencies
pip install "tokless[tiktoken]"  # real token counts (o200k_base) in reports
```

## Use

```bash
tokless stats ~/.claude/knowledge                 # dry run: gate + savings, writes nothing
tokless compress doc.md                           # one doc to stdout (bytes and line endings kept)
tokless compress ~/.claude/knowledge -o packed/   # whole bundle into a separate folder
tokless convert notes/ --dry-run                  # plain .md -> OKF: what would change
tokless convert notes/ -o notes-okf/              # add OKF frontmatter, then compress
tokless check doc.md packed/doc.md                # gate an existing output
tokless count packed/                             # token count
```

### In place, with a backup

```bash
tokless compress ~/.claude/knowledge --in-place   # backs up first, then writes
tokless backups ~/.claude/knowledge               # list backups
tokless restore ~/.claude/knowledge               # undo every run, back to the true originals
tokless backup-delete ~/.claude/knowledge --all   # drop backups when you are happy
```

```text
plan (all files, in memory) ──► backup + sha256 manifest ──► atomic write per file
                                                                │
                     any error or Ctrl-C ──► every written file put back, backup removed
```

- Backups live in the user cache, never in the bundle or its git work tree:
  `%LOCALAPPDATA%\tokless\backups\` on Windows, `$XDG_CACHE_HOME` or `~/.cache/tokless/backups/` elsewhere.
  Override with `--backup-dir DIR` or `TOKLESS_BACKUP_DIR`. After a run tokless prints the path, the undo command and the delete command.
- The manifest records the planned result of every file before the first write, so even a killed run restores byte for byte.
- A file edited by someone else between planning and writing is skipped, never overwritten, and reported.
- One run per bundle at a time: a second run, restore or backup-delete exits 2 at once. The lock is an OS lock, so a run that dies never leaves the bundle locked.
- `restore` skips a file you edited after the run (`--force` to overwrite it), and says which.
- `-o` must be a separate folder: the source, a folder inside it, or a parent of it is refused.
- Not touched, and listed in the report: symlinks and junctions, dot folders (`.obsidian`, `.trash`),
  tooling folders, git-ignored paths, non-UTF-8 or binary (`NUL`) files, files with mixed line endings.
- File permissions are kept.

### Exit codes

| code | meaning |
|---|---|
| 0 | done; files the gate could not shrink safely were kept as they were |
| 1 | done, but a file was skipped (not UTF-8 / binary), the body saving was under `--min-saving`, or `--strict` and a file was kept safe or warned |
| 2 | nothing written: bad path, `-o` overlaps the source, a file is not writable, another run holds the bundle |
| 130 | stopped with Ctrl-C; every file already written was put back |

Use `--strict` in CI, so a gate fallback or an invalid-OKF warning fails the job, and
`--min-saving PCT` to fail a run that saves less than you expect.

```python
from tokless import compress, convert, check, expand

result = compress(text)            # one mode: identifier backticks dropped outside code
result.text, result.tokens_before, result.tokens_after
check(text, result.text)           # raises tokless.GateError on any loss
expand(result.text)                # writes back the §N legend of a 0.1.x / 0.2.x file; else a no-op
convert(plain_md, "ops/deploy.md") # OKF frontmatter + compress
```

## What it does

One mode, no options. On every document outside code, a backtick span that holds one plain
identifier loses its backticks: `` `executor.receipt` `` -> `executor.receipt`, `` `tokens_total` ``
-> `tokens_total`. Nothing else changes: every word, table, list, line break and blank line stays.

| Kept as it is | Example |
|---|---|
| Frontmatter | byte for byte |
| Fenced and indented code, `$$` math, HTML comments, templates | verbatim |
| Spans with a space, an operator, a path prefix or a dash flag | `` `== selection_hash` ``, `` `~/.claude/x` ``, `` `--out -` `` |
| Spans that would turn into markup or glue to a word | `` `__init__` ``, `` `_x` ``, `` x`foo`s `` |
| Line endings | a CRLF file stays CRLF |

The saving is small by design. An output that is not smaller than its input by the chars/4
estimate is never returned: `compress()` gives back the input as is. A second run changes nothing.

`convert` infers `type` (Index for README/index and link-heavy docs, Runbook or Playbook by
name, else Reference), `title` (first H1, else the file name), `description` (first real prose
sentence: dates, TODOs and template lines are skipped), `tags` (folder names, minus generic ones
like `docs` or numbers) and `status: draft`. Keys a file already has are kept as they are.
Files with TOML (`+++`) or JSON front matter are skipped with a reason. Small notes grow,
because the frontmatter is new; the report shows body and frontmatter separately.

## The gate

`check(source, output)` fails the document when any of these break:

- frontmatter is not byte-identical
- a code block, math block, template span or HTML comment changed
- a locked literal is missing or glued to a new neighbour: inline code, URL, link target,
  wikilink, path, file name, flag (incl. `--out -`), number, date, version, email, env var,
  `ALLCAPS`, `snake_case`, `camelCase`, quoted string
- the sequence of content words and numbers is not the same as the source's, repeats
  included (so a swapped port between two table rows fails)
- fewer negations, or fewer negation + intensifier pairs (`not just`), than the source
- fewer `is not` pairs, or fewer copulas that carry the voice (`it is used`), than the source

The gate also reads output of older versions: a `§N=` legend is expanded first (sigils the
source itself uses are never read as one), and a factored line `- head: a; b` is split back
to `- head a` / `- head b` before the word sequence is compared.

## 0.3.0: one safe mode

Measured on the 2699-token Office doc `context-routing.md`, 42-question probe on small models:

| Version | Tokens | Small-model result |
|---|---|---|
| original | 2699 | qwen3.5:2b 34.0, llama3.2:3b 34.7, qwen3.5:9b 38/38 of 38 facts |
| 0.3.0 (identifier backticks only) | 2612 (-3.2%) | qwen3.5:2b 34.0, llama3.2:3b 35.0, qwen3.5:9b 38/38, traps the same, zero flipped questions |
| 0.2.x default | -9.2% | flipped 3 questions on the 2B/3B ("A missing field is a FAIL" became "missing field a FAIL") |
| hand-built tables | -18% | flipped 6 |

Unwrapped lines, bold removal and trimmed tables each flipped questions too. So 0.3.0 drops
filler words, copula, table trimming, list factoring, the `§N` phrase legend, link collapse,
emphasis stripping, line unwrapping and blank-line collapse. Frontmatter stays byte-identical.

Migration: output of 0.1.x and 0.2.x still verifies (`check`) and restores (`expand`,
`tokless restore`). New runs only drop identifier backticks. The flags removed in 0.2.0
(`--level`, `--factor`, `--dictionary`) stay removed. To keep the old, stronger output, pin
`tokless<0.3`.

## Design boundaries

```text
context-rainbow  -> picks WHICH OKF docs load (never reads bodies)
tokless          -> shrinks WHAT is inside a doc, once, at write time
```

No LLM summarisation, no perplexity pruning: output is deterministic and reproducible.

## Author

Karthick Raja M · Karthickrajam18@gmail.com · MIT License
