Metadata-Version: 2.4
Name: tokless
Version: 0.1.1
Summary: Lossless token compression for OKF knowledge and agent skills.
Author-email: Karthick Raja M <Karthickrajam18@gmail.com>
License-Expression: MIT
Project-URL: Homepage, https://github.com/karthyick/tokless
Keywords: okf,context,llm,agents,tokens,compression
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: tiktoken
Requires-Dist: tiktoken>=0.7; extra == "tiktoken"
Provides-Extra: dev
Requires-Dist: pytest>=7.0.0; extra == "dev"
Requires-Dist: build>=0.10.0; extra == "dev"
Requires-Dist: twine>=4.0.0; extra == "dev"
Requires-Dist: tiktoken>=0.7; extra == "dev"
Dynamic: license-file

# tokless

Lossless token compression for OKF knowledge and agent skills, plus plain markdown to OKF.

Docs and skills loaded into an LLM's context are written for people: articles, filler words,
padded tables, bold markers. The model does not need any of that. tokless strips it
deterministically and proves, per document, that no fact was lost.

```text
plain .md ──► convert ──► OKF frontmatter added ─┐
OKF / SKILL.md ──────────────────────────────────┴─► compress ──► gate ──► same facts, fewer tokens
                                                     (no model call)  │
                                       gate fails or no saving ───────┴─► file kept as it was
```

## Install

```bash
pip install tokless              # zero dependencies
pip install "tokless[tiktoken]"  # real token counts (o200k_base) in reports
```

## Use

```bash
tokless stats ~/.claude/knowledge                 # dry run: gate + savings, writes nothing
tokless compress doc.md                           # one doc to stdout (bytes and line endings kept)
tokless compress ~/.claude/knowledge -o packed/   # whole bundle into a separate folder
tokless convert notes/ --dry-run                  # plain .md -> OKF: what would change
tokless convert notes/ -o notes-okf/              # add OKF frontmatter, then compress
tokless check doc.md packed/doc.md                # gate an existing output
tokless count packed/                             # token count
```

### In place, with a backup

```bash
tokless compress ~/.claude/knowledge --in-place   # backs up first, then writes
tokless backups ~/.claude/knowledge               # list backups
tokless restore ~/.claude/knowledge               # undo every run, back to the true originals
tokless backup-delete ~/.claude/knowledge --all   # drop backups when you are happy
```

```text
plan (all files, in memory) ──► backup + sha256 manifest ──► atomic write per file
                                                                │
                     any error or Ctrl-C ──► every written file put back, backup removed
```

- Backups live in the user cache, never in the bundle or its git work tree:
  `%LOCALAPPDATA%\tokless\backups\` on Windows, `$XDG_CACHE_HOME` or `~/.cache/tokless/backups/` elsewhere.
  Override with `--backup-dir DIR` or `TOKLESS_BACKUP_DIR`. After a run tokless prints the path, the undo command and the delete command.
- The manifest records the planned result of every file before the first write, so even a killed run restores byte for byte.
- A file edited by someone else between planning and writing is skipped, never overwritten, and reported.
- One run per bundle at a time: a second run, restore or backup-delete exits 2 at once. The lock is an OS lock, so a run that dies never leaves the bundle locked.
- `restore` skips a file you edited after the run (`--force` to overwrite it), and says which.
- `-o` must be a separate folder: the source, a folder inside it, or a parent of it is refused.
- Not touched, and listed in the report: symlinks and junctions, dot folders (`.obsidian`, `.trash`),
  tooling folders, git-ignored paths, non-UTF-8 or binary (`NUL`) files, files with mixed line endings.
- File permissions are kept.

### Exit codes

| code | meaning |
|---|---|
| 0 | done; files the gate could not shrink safely were kept as they were |
| 1 | done, but a file was skipped (not UTF-8 / binary), the body saving was under `--min-saving`, or `--strict` and a file was kept safe or warned |
| 2 | nothing written: bad path, `-o` overlaps the source, a file is not writable, another run holds the bundle |
| 130 | stopped with Ctrl-C; every file already written was put back |

Use `--strict` in CI, so a gate fallback or an invalid-OKF warning fails the job, and
`--min-saving PCT` to fail a run that saves less than you expect.

```python
from tokless import compress, convert, check

result = compress(text)            # level="max" (default) or "safe"
result.text, result.tokens_before, result.tokens_after
check(text, result.text)           # raises tokless.GateError on any loss
convert(plain_md, "ops/deploy.md") # OKF frontmatter + compress
```

## What it does

| Layer | Level | Example |
|---|---|---|
| Frontmatter | all | passed through byte for byte |
| Fenced and indented code, `$$` math, inline `$math$`, HTML comments, `{% %}` `{{ }}` `<% %>` templates | all | kept verbatim |
| Emphasis | all | `**must**` -> `must` |
| Blank lines | all | runs of blank lines become one (block structure is kept) |
| Table padding | all | `\| a   \| b \|` -> `\|a\|b\|` |
| Filler words | max | drops `a an the is are very really just basically actually simply quite please kindly` |
| Filler phrases | max | `in order to` -> `to`, `is able to` -> `can` |
| Table markdown | max | outer pipes dropped (kept for one-column tables), separator shortened to `-\|-` |
| Code-span backticks | max | `` `router.py` `` -> `router.py` (only for single tokens with `/ . - _ : digits`) |
| Same-text URL links | max | `[https://x.io](https://x.io)` -> `https://x.io`; relative links stay links |
| List factoring | opt-in `--factor` | `- Use X for a` / `- Use X for b` -> `- Use X for: a; b` |
| Phrase dictionary | opt-in `--dictionary` | 3+ repeats of a long phrase -> `§1`, legend at top |

Never dropped, because each one changes meaning:

- negations (`not no never none nothing nor cannot without unless except n't dont isnt ...`)
- an intensifier right after a negation: `not just snippets` stays, it never becomes `not snippets`
- `is` / `are` before a negation or an -ing verb: `is not`, `refuses is telling` stay
- `you` / `your` (the actor), `was` / `were` (the tense)
- any word in a heading, so `#anchor` links keep working

Also kept: one run reaches a fixed point (a second run changes nothing), line endings (a CRLF
file stays CRLF), and the CommonMark block structure (checked with a parser on both real corpora).

`convert` infers `type` (Index for README/index and link-heavy docs, Runbook or Playbook by
name, else Reference), `title` (first H1, else the file name), `description` (first real prose
sentence: dates, TODOs and template lines are skipped), `tags` (folder names, minus generic ones
like `docs` or numbers) and `status: draft`. Keys a file already has are kept as they are.
Files with TOML (`+++`) or JSON front matter are skipped with a reason. Small notes grow,
because the frontmatter is new; the report shows body and frontmatter separately.

## The gate

`check(source, output)` fails the document when any of these break:

- frontmatter is not byte-identical
- a code block, math block, template span or HTML comment changed
- a locked literal is missing or glued to a new neighbour: inline code, URL, link target,
  wikilink, path, file name, flag (incl. `--out -`), number, date, version, email, env var,
  `ALLCAPS`, `snake_case`, `camelCase`, quoted string
- the sequence of content words and numbers is not the same as the source's, repeats
  included (so a swapped port between two table rows fails)
- fewer negations, or fewer negation + intensifier pairs (`not just`), than the source
- fewer `is not` or `is telling` pairs than the source

`tokless check SRC OUT --factor` checks output made with `--factor`, where a shared prefix
appears once.

## Measured

tiktoken `o200k_base`, tokless 0.1.0, every output file passes the gate:

| Corpus | Files | Whole file | Body only |
|---|---|---|---|
| OKF knowledge bundle | 708 | -4.6% | -6.5% |
| Agent skills (SKILL.md + references) | 400 | -7.0% | -7.7% |
| Plugin marketplace cache | 27,295 (2 kept as original by the gate) | -5.8% | -8.6% |

Q&A accuracy: Sonnet writes questions from the original doc, Haiku answers from the original
and from the tokless copy, Sonnet grades both blind.

| Sample | Questions | Original | tokless | Claude input tokens |
|---|---|---|---|---|
| seed 7, 24 docs (1 call error) | 155 | 97.4% | 98.1% | 101,162 -> 95,752 (-5.3%) |
| seed 11, 25 docs (2 errors: a doc moved, a bad JSON reply) | 138 | 100% | 97.8% | 99,479 -> 94,779 (-4.7%) |
| both, pooled | 293 | 98.6% | 98.0% | -5.0% |

- Frontmatter is 28% of the knowledge bundle's tokens and is never touched (it is what
  frontmatter-only routers read), so the body number is the compressor's real rate.
- The original alone swings about 2 points between identical runs; differences inside that are noise.
- Docs already written tersely (tables, short lines) save less; prose-heavy docs save more.

## Research

| Technique | Paper | In tokless |
|---|---|---|
| Lossless meta-token / dictionary compression | arXiv 2506.00307 | opt-in `--dictionary`, legend only when it pays for itself |
| Dictionary encoding for repetitive data | arXiv 2604.13066 | same rule: an alias must save more than its legend costs |
| Lexical, deterministic prompt compression (stop words) | arXiv 2609.13154 | adopted with a closed word list and a gate; words that change meaning are never dropped |
| Perplexity-based pruning (LLMLingua) | arXiv 2310.05736 | rejected: needs a model and is not deterministic |
| Prompt format affects accuracy | arXiv 2411.10541 | why accuracy is measured with a Q&A eval, not assumed |

## Design boundaries

```text
context-rainbow  -> picks WHICH OKF docs load (never reads bodies)
tokless          -> shrinks WHAT is inside a doc, once, at write time
```

No LLM summarisation, no perplexity pruning: output is deterministic and reproducible.

## Author

Karthick Raja M · Karthickrajam18@gmail.com · MIT License
