Metadata-Version: 2.4
Name: bltzr
Version: 0.2.7
Summary: Vocabulary extraction for language learners
Author: Samiddhi
License: CC-BY-SA-3.0
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: certifi>=2024.2.2
Requires-Dist: click>=8.1
Requires-Dist: platformdirs>=4.5.1
Provides-Extra: dev
Requires-Dist: pytest>=8; extra == "dev"
Requires-Dist: bump-my-version>=1; extra == "dev"
Requires-Dist: build>=1.2; extra == "dev"
Requires-Dist: twine>=6; extra == "dev"
Provides-Extra: japanese
Requires-Dist: SudachiPy==0.6.11; extra == "japanese"
Requires-Dist: SudachiDict-core==20250515; extra == "japanese"
Dynamic: license-file

# Blitzer

A tool for vocabulary extraction intended for language learners.

## Supported Languages

Word boundaries use Unicode letters plus AI-reviewed language-specific
rules. These may not be perfect. Multiword paradigm entries are
omitted, and dictionary coverage varies. Development packs are
excluded from this list and from releases. The list describes this
checkout's next release; newly promoted packs become downloadable
after publication.

If your target language is not on this list, see [languages without a release-ready pack](language-packs/UNSUPPORTED-LANGUAGES.md)
for an explanation of what is needed to implement
it. If your language is not listed in that document either, open a
pull request, ideally with a source of inflectional data for that
language.

| Language                    | Code |
|-----------------------------|------|
| Adyghe                      | ady  |
| Afrikaans                   | afr  |
| Akan                        | aka  |
| Albanian                    | sqi  |
| Amharic                     | amh  |
| Ancient Greek               | grc  |
| Anglo-Norman                | xno  |
| Armenian                    | hye  |
| Ashaninka                   | cni  |
| Assamese                    | asm  |
| Asturian                    | ast  |
| Aymara                      | aym  |
| Azerbaijani                 | aze  |
| Bashkir                     | bak  |
| Basic                       | base |
| Basque                      | eus  |
| Belarusian                  | bel  |
| Bengali                     | ben  |
| Braj                        | bra  |
| Breton                      | bre  |
| Bulgarian                   | bul  |
| Catalan                     | cat  |
| Cebuano                     | ceb  |
| Central Kurdish             | ckb  |
| Central Pame                | pbs  |
| Chewa                       | nya  |
| Chichicapan Zapotec         | zpv  |
| Classical Armenian          | xcl  |
| Classical Syriac            | syc  |
| Congo Swahili               | swc  |
| Cornish                     | cor  |
| Cree                        | cre  |
| Crimean Tatar               | crh  |
| Czech                       | ces  |
| Dakota                      | dak  |
| Danish                      | dan  |
| Dutch                       | nld  |
| Eastern Highland Chatino    | cly  |
| Egyptian Arabic             | arz  |
| Eibela                      | ail  |
| English                     | eng  |
| Estonian                    | est  |
| Faroese                     | fao  |
| French                      | fra  |
| Friulian                    | fur  |
| Gã                          | gaa  |
| Galician                    | gal  |
| Georgian                    | kat  |
| German                      | deu  |
| Gothic                      | got  |
| Greenlandic                 | kal  |
| Gujarati                    | guj  |
| Gulf Arabic                 | afb  |
| Haida                       | hai  |
| Hebrew                      | heb  |
| Hiligaynon                  | hil  |
| Hindi                       | hin  |
| Hungarian                   | hun  |
| Icelandic                   | isl  |
| Indonesian                  | ind  |
| Ingrian                     | izh  |
| Irish                       | gle  |
| Italian                     | ita  |
| Kabardian                   | kbd  |
| Kannada                     | kan  |
| Karelian                    | krl  |
| Kashubian                   | csb  |
| Kazakh                      | kaz  |
| Ket                         | ket  |
| Khakas                      | kjh  |
| Khaling                     | klr  |
| Khalkha Mongolian           | khk  |
| Kholosi                     | hsi  |
| Kodi                        | kod  |
| Kongo                       | kon  |
| Kunwinjku                   | gup  |
| Ladin                       | lld  |
| Lamaholot                   | slp  |
| Latin                       | lat  |
| Latvian                     | lav  |
| Lingala                     | lin  |
| Lithuanian                  | lit  |
| Livonian                    | liv  |
| Livvi-Karelian              | olo  |
| Low German                  | nds  |
| Lower Sorbian               | dsb  |
| Ludian                      | lud  |
| Luganda                     | lug  |
| Macedonian                  | mkd  |
| Magahi                      | mag  |
| Malagasy                    | mlg  |
| Maltese                     | mlt  |
| Manx                        | glv  |
| Māori                       | mao  |
| Mapudungun                  | arn  |
| Mezquital Otomi             | ote  |
| Middle French               | frm  |
| Middle High German          | gmh  |
| Middle Low German           | gml  |
| Modern Greek                | ell  |
| Modern Standard Arabic      | ara  |
| Murrinhpatha                | mwf  |
| Navajo                      | nav  |
| Neapolitan                  | nap  |
| North Frisian               | frr  |
| Northern Kurdish (Kurmanji) | kmr  |
| Northern Sami               | sme  |
| Norwegian Bokmål            | nob  |
| Norwegian Nynorsk           | nno  |
| O'odham                     | ood  |
| Occitan                     | oci  |
| Old Church Slavic           | chu  |
| Old English                 | ang  |
| Old French                  | fro  |
| Old High German             | goh  |
| Old Irish                   | sga  |
| Old Norse                   | non  |
| Old Saxon                   | osx  |
| Oromo                       | orm  |
| Pali                        | pli  |
| Pashto                      | pus  |
| Persian                     | fas  |
| Polish                      | pol  |
| Portuguese                  | por  |
| Quechua                     | que  |
| Romanian                    | ron  |
| Russian                     | rus  |
| Sakha                       | sah  |
| San Pedro Amuzgos Amuzgo    | azg  |
| Scottish Gaelic             | gla  |
| Seneca                      | see  |
| Serbo-Croatian              | hbs  |
| Shipibo-Konibo              | shp  |
| Shona                       | sna  |
| Slovak                      | slk  |
| Slovenian                   | slv  |
| Southern Kurdish            | sdh  |
| Southern Sotho (Sesotho)    | sot  |
| Spanish                     | spa  |
| Swedish                     | swe  |
| Swiss German                | gsw  |
| Tagalog                     | tgl  |
| Tatar                       | tat  |
| Telugu                      | tel  |
| Turkish                     | tur  |
| Turkmen                     | tuk  |
| Tuvan                       | tyv  |
| Ukrainian                   | ukr  |
| Upper Sorbian               | hsb  |
| Uyghur                      | uig  |
| Uzbek                       | uzb  |
| Venetian                    | vec  |
| Veps                        | vep  |
| Võro                        | vro  |
| Votic                       | vot  |
| Walmajarri                  | wmt  |
| Welsh                       | cym  |
| Western Frisian             | fry  |
| Xibe                        | sjo  |
| Yaitepec Chatino            | ctp  |
| Yanesha                     | ame  |
| Yiddish                     | yid  |
| Yoloxóchitl Mixtec          | xty  |
| Zarma                       | dje  |
| Zenzontepec Chatino         | czn  |
| Zulu                        | zul  |

## Install

Requires Python 3.11 or newer.

```sh
pip install bltzr
bltzr --help
```

## Extract vocabulary

```sh
bltzr blitz -l eng -t "The dogs are running." --lemmatize
bltzr blitz -l eng --file chapter.txt --freq --context
cat chapter.txt | bltzr blitz -l eng --format tsv > vocabulary.tsv
```

`-l` chooses the language. Use `-t` for text, `--file` for a UTF-8 file,
or pipe text into the command. Words are shown as written by default;
`--lemmatize` shows dictionary forms, such as `dogs → dog`.

## Language packs

Basic (`base`) and English (`eng`) work immediately, without a download.
Basic extracts words without a dictionary and cannot lemmatize them.
Other languages need a pack:

```sh
bltzr list-languages
bltzr install-plugin slv
bltzr blitz -l slv -t "Nisem tako prepričan."
```

To update an installed pack, use `bltzr install-plugin slv --replace`.
Processing works offline once the pack is installed.

The next release supports both pack configuration formats 1 and 2. Format 2
adds reviewed rules for tone marks, joiners and word-internal punctuation;
update the application before installing a format-2 pack. Dictionaries vary
in coverage, and support for a script does not imply coverage of every
orthography or dialect. See [project status and remaining development work](maintenance/reports/project-status-20261008.md).

Japanese remains a development pack. The optional analyzer can be tested
from this checkout with `.venv/bin/python -m pip install -e '.[dev,japanese]'`
and `--plugins-dir ./language-packs/dev`. Its dictionary must already be
installed; processing never downloads models automatically. Sanskrit,
Tibetan and Urdu still require additional lexical-segmentation work.

You can also install a local pack with `bltzr install-plugin ./my-pack/slv`.
To use this checkout's packs directly, add
`--plugins-dir /path/to/blitzer-py/language-packs` to the command.

## Create your config

Blitzer automatically reads `~/.config/bltzr/bltzr.toml`. No environment
variable or command-line option is needed. Create the file, then open it
in your editor:

```sh
mkdir -p ~/.config/bltzr
touch ~/.config/bltzr/bltzr.toml
```

Here is an arbitrary default:

```toml
[defaults]
freq = true
context = true
sort = "alphabetical"
bold = "markdown"
save_context = "flag-only"

[languages.eng]
skip_exact_words_file = "./eng-exact-words.txt"
skip_word_families_file = "./eng-word-families.txt"
```

`[defaults]` applies to every language. `[languages.eng]` overrides
settings for English; use the language code visible with `bltzr list-languages` if you are not sure which to use.

Save your settings and run normally:

```sh
bltzr blitz -l eng -t "He, she, it, swims!"
```

Use `-C PATH` to load another config, or `-n` to ignore configuration.
An existing config under `$XDG_CONFIG_HOME/bltzr/` takes priority if that
environment variable is set. `BLITZER_CONFIG` can explicitly select a
different file, but you do not need it for the default location.
Command-line options override your config; for example, `--no-context`
hides example sentences even when `context = true`.

See [config.example.toml](config.example.toml) for the full set of settings.
The complete command reference is in [bltzr(1)](docs/bltzr.1).

## Choose which words to skip

Keep two optional files. Each has one job, and you can use either or both.
Skipped words do not appear in the vocabulary and do not contribute to counts
or example sentences.

### Skip these exact words (word form)

Use this list when you know some spellings of a word but still want to study
others. For the verb **to be**, you might know **be** and **am** but still
want to study **is**, **are**, **was**, **were**, **being**, and **been**.

Create `eng-exact-words.txt` beside your config:

```text
be
am
cat
```

Then select it:

```toml
[languages.eng]
# Skip only these exact words (word form).
# "be" and "am" leave "is", "are", "was", "were", "being", and "been" counted.
# "cat" leaves "cats" counted.
skip_exact_words_file = "./eng-exact-words.txt"
```

Even though **be** is the basic word, putting it in this file skips only
**be**. It does not skip its other forms. Likewise, **cat** does not skip **cats**.
For Slovenian, listing **sem** and **biti** here leaves **sva** counted.

### Skip these words and their other forms: word families (lexeme/lemma)

Use this list when you want to skip a whole word family. The program uses its
dictionary to connect the different forms of a word. For **to be**, put
**be** in the file, without “to”:

```text
be
cat
```

```toml
[languages.eng]
# Skip these word families (lexeme/lemma), using the dictionary.
# "be" skips "be", "am", "is", "are", "was", "were", "being", and "been".
# "cat" skips both "cat" and "cats".
skip_word_families_file = "./eng-word-families.txt"
```

A word family means the grammatical forms of the same word, such as **cat**
and **cats**. It does not include related words such as **catlike**.
Write the basic word in this file: **be**, rather than **am**; **cat**, rather
than **cats**. The program does not guess a whole family from an inflected
word you list.

| What you put in a file | Which file | What gets skipped | What can still be counted |
|---|---|---|---|
| `be`, `am` | Exact words (word form) | be, am | is, are, was, were, being, been |
| `be` | Word families (lexeme/lemma) | be, am, is, are, was, were, being, been | Other words |
| `cat` | Exact words (word form) | cat | cats |
| `cat` | Word families (lexeme/lemma) | cat, cats | Other words |

### Use both lists and choose what to display

```toml
[languages.eng]
skip_exact_words_file = "./eng-exact-words.txt"
skip_word_families_file = "./eng-word-families.txt"
```

Each setting names **one file**, not a word or an array of files. Leave out
settings you do not need. Paths are relative to the config file and can use
`~` or environment variables. A missing file is treated as an empty list;
ordinary extraction does not create it.

Both files contain one word per line. Blank lines and whole-line `#` comments
are allowed. Hyphenated words such as `sally-anne` are accepted; internal spaces
are not. Words are compared using the language pack's normalization, usually
ignoring capital letters. Accepting a hyphenated entry does not change how the
language's tokenizer divides a hyphenated word in input text.

Display is a separate choice. `--lemmatize` (`-L`) shows basic words
(lexeme/lemma), such as **are → be** or **cats → cat**. Without it, the output
shows words as written (word form). This never changes what either file skips.
If **be** and **am** are in the exact-word file, **are** still contributes a
count, even when `-L` displays it as **be**.

Word-family matching depends on dictionary coverage. A spelling can belong
to more than one family: English **saw** can mean the tool or a past form of
**see**. Listing **see** removes that candidate; the **saw** tool candidate
can remain. The program does not decide the meaning from the sentence.

The bundled English dictionary currently lacks some links in the **to be**
family. The examples above describe the grammatical relationship; a pack must
contain those links to skip the whole family automatically. With the current
bundled data, put all eight spellings in the exact-word file if you want to
skip them all. Changing a skip list does not repair missing dictionary data.

### Command-line options

```sh
bltzr blitz -l eng -L --skip-exact-words-file eng-exact-words.txt \
  --skip-word-families-file eng-word-families.txt -t "I am here. They are here."
```

Each file option replaces only the corresponding configured file for that run;
it leaves the other list active. `--show-all-words` (`-N`) skips neither list
and cannot be combined with explicit skip-file options. Basic (`base`) mode
supports exact words; word families require a dictionary.

### Updating and tidying the lists

Reading a story does not mean you learned its words, so automatic additions
are off by default. To preview additions, choose which list would receive them:

```sh
bltzr blitz -l eng -i chapter.txt --update-list exact-words --test-known
bltzr blitz -l eng -i chapter.txt --update-list word-families --test-known
```

These commands use the corresponding file from your config. Previews change
neither list nor history. Replace `--test-known` with `--update-known` to write
additions. Exact-word updates add encountered spellings (word form); word-family
updates add dictionary basic words (lexeme/lemma). The choice does not depend
on `-L`. Only dictionary-matched surviving words propose additions, and only
the selected list is written. Existing files receive a `.bak` backup on rewrite.

For automatic updates configured with `auto_update_known = true`, explicitly
set `update_list = "exact-words"` or `"word-families"` in the language section.
`--show-all-words` affects skipping, not updating; use `--no-update-known` to
disable additions.

Check either list for duplicates, then apply cleanup if wanted:

```sh
bltzr cleanup-known eng ~/.config/bltzr/eng-exact-words.txt
bltzr cleanup-known eng ~/.config/bltzr/eng-exact-words.txt --apply
```

Cleanup normalizes entries, preserves comments, and saves a `.bak` copy.

### Moving from older settings

Existing configurations and scripts using `known_file`, `exclusions`,
`forms_only`, and `filter_by` continue to work with their original behavior.
The old CLI file options are kept for compatibility but hidden from normal help.

For migration:

- Files formerly read through `known_file` or `exclusions` with
  `filter_by = "forms"` belong in the exact-word list (word form).
- With `filter_by = "lemmas"`, they belong in the word-family list
  (lexeme/lemma). Review entries and use basic words in this file; move
  inflected entries that you only want skipped exactly into the exact-word file.
- Files formerly in `forms_only` belong in the exact-word list.
- If several old files belong in one new list, combine their contents into
  one file. Duplicate entries do not change the result.

Remove the old file settings and language-level `filter_by` when selecting
the new settings. Do not mix old and new file settings in one language section.
The two new lists have fixed meanings; a legacy `filter_by` in `[defaults]`
does not change them. Selecting new file options on the command line uses the
new lists for that run instead of loading configured legacy files. Combining
new file settings with explicit legacy filtering options is an error.
Your configuration and word files are never migrated automatically.

## Useful options

| Option | Effect |
|----|----|
| `--freq` | Include each word's count in your text |
| `--context` | Include example sentences |
| `--context-limit 3` | Include up to three examples per word |
| `--bold off` | Turn off highlighting; also accepts html, markdown, org |
| `--sort appearance` | Keep the order words first appeared |
| `--sort textual-frequency` | Show the most frequent words first |
| `--exclude-unknown` | Hide words missing from the dictionary |
| `--skip-exact-words-file PATH` | Skip only the listed words (word form) |
| `--skip-word-families-file PATH` | Skip listed words and their other forms (lexeme/lemma) |
| `--show-all-words` | Count words without applying either skip list |
| `--format json` | Export JSON; also accepts text, tsv, report |

Sorting also accepts `alphabetical`, `global-frequency` (requires a pack
with frequency data), and `custom` with `--custom-order PATH`.

## Save and review example sentences

```sh
bltzr blitz -l eng -t "He helps me." --lemmatize --save-context
bltzr history eng
bltzr history eng help
```

History uses the displayed word: `help` with `--lemmatize`, or `helps`
without it. Saved history is separate from your known-word list.

In config, `save_context` accepts `"flag-only"` (save when requested),
`"always"`, `"prompt"`, or `"never"`. Use `--no-save-context` to skip saving.

## Help and development

```sh
bltzr blitz --help
bltzr dev --help
man ./docs/bltzr.1
```

The distribution installs the man page under `share/man/man1` in Python's
installation prefix. If your environment includes that directory in its
man search path, use `man bltzr`. Otherwise open the file directly, for
example `man "$VIRTUAL_ENV/share/man/man1/bltzr.1"` for a virtual environment.

For a source checkout, install with
`.venv/bin/python -m pip install -e '.[dev]'` after creating a virtual
environment. Run tests with `.venv/bin/python -m pytest -q`.
See [release setup](maintenance/RELEASING.md) for one-command GitHub/PyPI publication and
[language-pack publishing](language-packs/README.md) for manual releases.

## License

[CC BY-SA 3.0](LICENSE). Dictionary packs retain their own licenses and attribution.
