Metadata-Version: 2.4
Name: tokless
Version: 0.2.1
Summary: Lossless token compression for OKF knowledge and agent skills.
Author-email: Karthick Raja M <Karthickrajam18@gmail.com>
License-Expression: MIT
Project-URL: Homepage, https://github.com/karthyick/tokless
Keywords: okf,context,llm,agents,tokens,compression
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: tiktoken
Requires-Dist: tiktoken>=0.7; extra == "tiktoken"
Provides-Extra: dev
Requires-Dist: pytest>=7.0.0; extra == "dev"
Requires-Dist: build>=0.10.0; extra == "dev"
Requires-Dist: twine>=4.0.0; extra == "dev"
Requires-Dist: tiktoken>=0.7; extra == "dev"
Dynamic: license-file

# tokless

Lossless token compression for OKF knowledge and agent skills, plus plain markdown to OKF.

Docs and skills loaded into an LLM's context are written for people: articles, filler words,
padded tables, bold markers. The model does not need any of that. tokless strips it
deterministically and proves, per document, that no fact was lost.

```text
plain .md ──► convert ──► OKF frontmatter added ─┐
OKF / SKILL.md ──────────────────────────────────┴─► compress ──► gate ──► same facts, fewer tokens
                                                     (no model call)  │
                                       gate fails or no saving ───────┴─► file kept as it was
```

## Install

```bash
pip install tokless              # zero dependencies
pip install "tokless[tiktoken]"  # real token counts (o200k_base) in reports
```

## Use

```bash
tokless stats ~/.claude/knowledge                 # dry run: gate + savings, writes nothing
tokless compress doc.md                           # one doc to stdout (bytes and line endings kept)
tokless compress ~/.claude/knowledge -o packed/   # whole bundle into a separate folder
tokless convert notes/ --dry-run                  # plain .md -> OKF: what would change
tokless convert notes/ -o notes-okf/              # add OKF frontmatter, then compress
tokless check doc.md packed/doc.md                # gate an existing output
tokless count packed/                             # token count
```

### In place, with a backup

```bash
tokless compress ~/.claude/knowledge --in-place   # backs up first, then writes
tokless backups ~/.claude/knowledge               # list backups
tokless restore ~/.claude/knowledge               # undo every run, back to the true originals
tokless backup-delete ~/.claude/knowledge --all   # drop backups when you are happy
```

```text
plan (all files, in memory) ──► backup + sha256 manifest ──► atomic write per file
                                                                │
                     any error or Ctrl-C ──► every written file put back, backup removed
```

- Backups live in the user cache, never in the bundle or its git work tree:
  `%LOCALAPPDATA%\tokless\backups\` on Windows, `$XDG_CACHE_HOME` or `~/.cache/tokless/backups/` elsewhere.
  Override with `--backup-dir DIR` or `TOKLESS_BACKUP_DIR`. After a run tokless prints the path, the undo command and the delete command.
- The manifest records the planned result of every file before the first write, so even a killed run restores byte for byte.
- A file edited by someone else between planning and writing is skipped, never overwritten, and reported.
- One run per bundle at a time: a second run, restore or backup-delete exits 2 at once. The lock is an OS lock, so a run that dies never leaves the bundle locked.
- `restore` skips a file you edited after the run (`--force` to overwrite it), and says which.
- `-o` must be a separate folder: the source, a folder inside it, or a parent of it is refused.
- Not touched, and listed in the report: symlinks and junctions, dot folders (`.obsidian`, `.trash`),
  tooling folders, git-ignored paths, non-UTF-8 or binary (`NUL`) files, files with mixed line endings.
- File permissions are kept.

### Exit codes

| code | meaning |
|---|---|
| 0 | done; files the gate could not shrink safely were kept as they were |
| 1 | done, but a file was skipped (not UTF-8 / binary), the body saving was under `--min-saving`, or `--strict` and a file was kept safe or warned |
| 2 | nothing written: bad path, `-o` overlaps the source, a file is not writable, another run holds the bundle |
| 130 | stopped with Ctrl-C; every file already written was put back |

Use `--strict` in CI, so a gate fallback or an invalid-OKF warning fails the job, and
`--min-saving PCT` to fail a run that saves less than you expect.

```python
from tokless import compress, convert, check, expand

result = compress(text)            # one mode: every lossless layer, always on
result.text, result.tokens_before, result.tokens_after, result.dictionary
check(text, result.text)           # raises tokless.GateError on any loss
expand(result.text)                # the same text with any §N legend written back out
convert(plain_md, "ops/deploy.md") # OKF frontmatter + compress
```

## What it does

One mode, no options: every layer below runs on every document, and the gate checks the
result. Factoring and the dictionary write a change only where it is safe and pays, so on
many documents they change nothing.

| Layer | Example |
|---|---|
| Frontmatter | passed through byte for byte |
| Fenced and indented code, `$$` math, inline `$math$`, HTML comments, `{% %}` `{{ }}` `<% %>` templates | kept verbatim |
| Emphasis | `**must**` -> `must` |
| Blank lines | runs of blank lines become one (block structure is kept) |
| Table padding | `\| a   \| b \|` -> `\|a\|b\|` |
| Filler words | drops `a an the is are very really just basically actually simply quite please kindly` |
| Filler phrases | `in order to` -> `to`, `is able to` -> `can` |
| Table markdown | outer pipes dropped (kept for one-column tables), separator shortened to `-\|-` |
| Code-span backticks | `` `router.py` `` -> `router.py` (only for single tokens with `/ . - _ : digits`) |
| Same-text URL links | `[https://x.io](https://x.io)` -> `https://x.io`; relative links stay links (link text and targets are never aliased either) |
| List factoring | `- Use X for a` / `- Use X for b` -> `- Use X for: a; b` (never when the shared head holds `: ` or an item holds `; `, so the line splits back exactly; never for task-list items, an item ending in a hard break, or an item whose rest holds a negation; `e.g.` keeps its period) |
| Phrase dictionary | 3+ repeats of a phrase -> `§1`, with a legend (below) only when the aliases save 16 tokens or more in all |

Never dropped, because each one changes meaning:

- negations (`not no never none nothing nor cannot without unless except n't dont isnt ...`)
- an intensifier right after a negation: `not just snippets` stays, it never becomes `not snippets`
- `is` / `are` before a negation or a verb form, or after a pronoun: `is not`, `refuses is telling`,
  `you are dispatched`, `it is used` stay (without them the sentence flips voice)
- `you` / `your` (the actor), `was` / `were` (the tense)
- any word in a heading, so `#anchor` links keep working

Also kept: one run reaches a fixed point (a second run changes nothing), line endings (a CRLF
file stays CRLF), and the CommonMark block structure of the body (a legend adds one paragraph
above it; checked with a CommonMark parser in the tests). An output that is not smaller than
its input by the chars/4 estimate is never returned: `compress()` gives back the input as is.

The phrase dictionary, in full:

- The legend sits between the frontmatter and the first line of the body, one `§N=phrase`
  line per alias, then one blank line, so it never joins the block below it.
  `expand()` removes the legend and that blank line and writes every alias back, byte for byte
  (CRLF files too).
- An alias is never written in a heading, a block quote, a table separator row, link text
  `[...]` or a link target `(...)`, and those places do not count toward the 3 repeats.
- A phrase never holds a token with `[ ] ( ) { } < > | #`, a backtick, a sigil, `/` or `\`
  (links and paths stay whole and greppable), a bare bullet, or a token ending in `, ; . : ! ?`
  (an alias never runs across a clause).
- Savings are priced at one token per word, a lower bound for tokenizers that split on spaces
  (o200k, cl100k); chars/4 over-counts plain words. No legend is written unless the aliases save
  16 tokens or more in all by that count. The decision uses only this pricing, so the files written are the
  same with or without tiktoken.
- An alias that would let a second run change the text (for example drop an `is` whose verb
  went into the alias) is refused.

`convert` infers `type` (Index for README/index and link-heavy docs, Runbook or Playbook by
name, else Reference), `title` (first H1, else the file name), `description` (first real prose
sentence: dates, TODOs and template lines are skipped), `tags` (folder names, minus generic ones
like `docs` or numbers) and `status: draft`. Keys a file already has are kept as they are.
Files with TOML (`+++`) or JSON front matter are skipped with a reason. Small notes grow,
because the frontmatter is new; the report shows body and frontmatter separately.

## The gate

`check(source, output)` fails the document when any of these break:

- frontmatter is not byte-identical
- a code block, math block, template span or HTML comment changed
- a locked literal is missing or glued to a new neighbour: inline code, URL, link target,
  wikilink, path, file name, flag (incl. `--out -`), number, date, version, email, env var,
  `ALLCAPS`, `snake_case`, `camelCase`, quoted string
- the sequence of content words and numbers is not the same as the source's, repeats
  included (so a swapped port between two table rows fails)
- fewer negations, or fewer negation + intensifier pairs (`not just`), than the source
- fewer `is not` pairs, or fewer copulas that carry the voice (`it is used`), than the source

The output's legend is expanded first; sigils the source itself uses are never read as one.
A factored list line (`- head: a; b`) writes a shared prefix once. Only when the exact word
sequence fails, the gate writes each such line back as `- head a` / `- head b` and runs every
check again on that copy. The copy splits a list line on its first `: `, then on each `; `;
the factorer never writes `: ` into a head or `; ` into an item, so that split is exact. A line
the source already wrote as `- note: a; b` is split only in that copy, never in the output. A
repeat deleted without that `: `, or two items swapped, still fails.

## 0.2.0: one mode

The modes are gone. Removed from the Python API: `LEVELS`, the `level` argument of `compress()`
and `convert()` (keyword or positional), and `factor=` / `dictionary=`. Removed from the CLI:
`--level`, `--factor`, `--dictionary` and `check --factor`. An old flag now exits 2
(`unrecognized arguments`), an old argument raises `TypeError`, and `from tokless import LEVELS`
raises `ImportError`. List factoring and the phrase dictionary always run. Output can open with
a `§N=` legend and a blank line; `expand()` undoes it. `compress --in-place`, and the pre-commit
hook that runs it, write factored lines and legends into the source files; `tokless restore`
puts the originals back. To keep 0.1.x output, pin `tokless<0.2` (or a hook `rev:` before
`v0.2.0`).

## Measured (tokless 0.1.0, filler layer only)

tiktoken `o200k_base`, tokless 0.1.0, filler layer only (factoring and dictionary were off),
every output file passes the gate:

| Corpus | Files | Whole file | Body only |
|---|---|---|---|
| OKF knowledge bundle | 708 | -4.6% | -6.5% |
| Agent skills (SKILL.md + references) | 400 | -7.0% | -7.7% |
| Plugin marketplace cache | 27,295 (2 kept as original by the gate) | -5.8% | -8.6% |

Q&A accuracy, tokless 0.1.0, filler layer only (factoring and dictionary were off): Sonnet
writes questions from the original doc, Haiku answers from the original and from the tokless
copy, Sonnet grades both blind.

| Sample | Questions | Original | tokless | Claude input tokens |
|---|---|---|---|---|
| seed 7, 24 docs (1 call error) | 155 | 97.4% | 98.1% | 101,162 -> 95,752 (-5.3%) |
| seed 11, 25 docs (2 errors: a doc moved, a bad JSON reply) | 138 | 100% | 97.8% | 99,479 -> 94,779 (-4.7%) |
| both, pooled | 293 | 98.6% | 98.0% | -5.0% |

- Frontmatter is 28% of the knowledge bundle's tokens and is never touched (it is what
  frontmatter-only routers read), so the body number is the compressor's real rate.
- The original alone swings about 2 points between identical runs; differences inside that are noise.
- Docs already written tersely (tables, short lines) save less; prose-heavy docs save more.

## 0.2.0 on the same corpora

tiktoken `o200k_base`, measured 2026-11-01 on a copy of the OKF knowledge bundle; every output
file passes the gate (742/742) and keeps its frontmatter byte for byte.

| Run | Files | Whole file | Body only | Files with a legend | Files with a factored line |
|---|---|---|---|---|---|
| tokless 0.2.0 | 742 (56 compressed, 686 kept) | 802,516 -> 800,058 (-0.3%) | 572,152 -> 569,689 (-0.4%) | 0 | 1 (one line) |
| tokless 0.1.2 default, same copy | 742 (55 compressed, 687 kept) | 802,516 -> 800,067 (-0.3%) | 572,152 -> 569,698 (-0.4%) | 0 | 0 |

| Q&A sample | Questions | Original | tokless 0.2.0 | Claude input tokens |
|---|---|---|---|---|
| the one doc that changed under 0.2.0 (it holds the factored line) | 12 | 100% | 100% | 1,408 -> 1,393 |
| public fixture `context-routing.md` | 17 | 100% | 100% | 1,955 -> 1,702 |

- That copy had already been through tokless 0.1.x (686 of 742 files come back unchanged), so
  these are the savings left on top of that, not the rate on a fresh corpus. 0.2.0 saves 9
  tokens more than 0.1.2 on it, all from the one factored line.
- No legend was written: once headings, quotes, links, paths and clause ends are off limits and
  a phrase is priced at one token per word, no repeated phrase in that bundle saves 16 tokens.
- Three more questions aimed at the factored line were all answered right from the 0.2.0 copy.
- The agent-skills and marketplace corpora were not re-run for 0.2.0.

## Research

| Technique | Paper | In tokless |
|---|---|---|
| Lossless meta-token / dictionary compression | arXiv 2506.00307 | always on, legend only when it pays for itself |
| Dictionary encoding for repetitive data | arXiv 2604.13066 | same rule: an alias must save more than its legend costs |
| Lexical, deterministic prompt compression (stop words) | arXiv 2609.13154 | adopted with a closed word list and a gate; words that change meaning are never dropped |
| Perplexity-based pruning (LLMLingua) | arXiv 2310.05736 | rejected: needs a model and is not deterministic |
| Prompt format affects accuracy | arXiv 2411.10541 | why accuracy is measured with a Q&A eval, not assumed |

## Design boundaries

```text
context-rainbow  -> picks WHICH OKF docs load (never reads bodies)
tokless          -> shrinks WHAT is inside a doc, once, at write time
```

No LLM summarisation, no perplexity pruning: output is deterministic and reproducible.

## Author

Karthick Raja M · Karthickrajam18@gmail.com · MIT License
