Metadata-Version: 2.5
Name: toklen
Version: 0.1.1
Summary: Count tokens with a model's real Hugging Face tokenizer
Project-URL: Homepage, https://github.com/T0mSIlver/toklen
License-Expression: MIT
License-File: LICENSE
Keywords: huggingface,llm,token-count,tokenizer,tokens
Requires-Python: >=3.10
Requires-Dist: tokenizers>=0.15
Description-Content-Type: text/markdown

# toklen

Count tokens with a model's real Hugging Face tokenizer. Prints an integer.

```bash
npx toklen openai-community/gpt2 "hello world"   # 2
uvx toklen openai-community/gpt2 "hello world"   # 2
```

Both commands always print the same number. Node uses
[`@huggingface/tokenizers`](https://www.npmjs.com/package/@huggingface/tokenizers),
Python uses the Rust [`tokenizers`](https://pypi.org/project/tokenizers/) bindings. One
dependency each, no `transformers`, no PyTorch.

```bash
toklen Qwen/Qwen3-8B < README.md                        # stdin
git diff | toklen openai-community/gpt2
toklen -s google-bert/bert-base-uncased "hello world"   # 4, with [CLS] and [SEP]
```

`toklen --help` covers the flags. [docs/usage.md](https://github.com/T0mSIlver/toklen/blob/main/docs/usage.md) has the library API and
the counting rules. [npm](https://www.npmjs.com/package/toklen) ·
[PyPI](https://pypi.org/project/toklen/) · MIT.
