Metadata-Version: 2.4
Name: icukit
Version: 0.10.0
Summary: ICU toolkit - library and CLI for Unicode and internationalization
Author-email: Kevin Lenzo <kevinlenzo@gmail.com>
License-Expression: BSD-2-Clause AND CC-BY-SA-4.0 AND Unicode-3.0
Project-URL: Homepage, https://github.com/lenzo-ka/icukit
Project-URL: Repository, https://github.com/lenzo-ka/icukit
Project-URL: Issues, https://github.com/lenzo-ka/icukit/issues
Keywords: icu,unicode,i18n,internationalization,localization,text,collation,formatting
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Operating System :: MacOS
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Software Development :: Internationalization
Classifier: Topic :: Software Development :: Localization
Classifier: Topic :: Text Processing
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: icukit/data/break_rules/en/LICENSE
License-File: icukit/data/break_rules/en/NOTICE
License-File: icukit/data/cldr_symbols/LICENSE
License-File: icukit/data/ucd_name_aliases/LICENSE
Requires-Dist: cartlet>=0.8
Requires-Dist: icukit-pyicu>=78.3.1
Provides-Extra: dev
Requires-Dist: pytest>=9.0; extra == "dev"
Requires-Dist: ruff>=0.6; extra == "dev"
Requires-Dist: lxml>=5.0; extra == "dev"
Requires-Dist: build>=1.0; extra == "dev"
Requires-Dist: tiergraph>=0.8.0; python_version >= "3.12" and extra == "dev"
Dynamic: license-file

# icukit

A comprehensive Python toolkit for Unicode and internationalization, built on ICU (International Components for Unicode).

icukit provides a Pythonic interface to ICU's powerful text processing, localization, and internationalization capabilities. It includes both a library API and a command-line interface.

## Installation

```bash
pip install icukit
```

This pulls in [`icukit-pyicu`](https://github.com/lenzo-ka/icukit-pyicu), which
bundles pre-built ICU libraries and PyICU, and `cartlet>=0.8` from PyPI. There
are no system dependencies to install on either **macOS** or **Linux** — the
wheels are self-contained.

See [Installation Guide](https://github.com/lenzo-ka/icukit/blob/main/docs/install.md) for details, including how to use a system PyICU instead.

## Features

### Text Processing

- **Transliteration**: Convert between scripts (Latin to Cyrillic, Hangul to Latin, etc.)
- **Normalization**: NFC, NFD, NFKC, NFKD Unicode normalization forms
- **Character Names**: Formal names, name aliases of every type (corrections, control names, abbreviations), and extended names of every code point, and lookup from a name or alias to its character
- **Text Segmentation**: Break text into words, sentences, lines, or grapheme clusters
- **Word shapes**: `shape` provides `coarse@1`, `cased@1`, and the approximate
  orthographic `cvletters@1`, with uncapped `cvletters_counts`. It labels vowel
  letters, not phones: the English table is curated, other alphabets remain `L`
  unless caller material supplies labels, and phone-level speakability is out of scope.
- **Unicode Regex**: Full Unicode-aware regular expressions with script and property support

### Sentence breaking

For English locales, `Breaker`, `SentenceOverride`, and `ik break sentences`
(as well as `ik break tokenize`) use ICU sentence candidates with token
integrity and the locale-fallback `break="suppress"` entries from the shipped
abbreviation lexicon. This applies to English locales with any region or script
except the `POSIX` variant. Candidate decisions follow ICU, token integrity,
caller-before rules, exceptions, an optional base, then caller-after rules.
The learned `en-tn@1` rule set, `en-tn-cart@1` synthetic-text model, and
`en-real-cart@1` real-text model remain available as opt-in bases. Pass
`base="none"` to either Python API or `--base none` to the CLI for raw ICU
sentence boundaries without the shipped list. Other locale languages keep raw
ICU as their default. `AbbreviationSentenceBreaker` remains the explicit
alternative that also deposits ambiguous lexicon boundaries. The TN bases'
measurements are agreement with a corpus splitter on synthetic text. The
real-text model's receipt reports source-record seam-break measurements and its
GUM and Hansard training sources; neither claim is a general sentence-breaking
accuracy result.

### Recognition

- **Value readings**: Find numbers, dates, times, measures, currencies, ordinals, and abbreviations in text by inverting ICU's own formatting, keeping every plausible reading of a span
- **Language-wide forms**: Read the forms every locale of a language writes (en_US text reads en_GB's "12 kilometres"), or only the locales a caller chooses

Readers come assembled in sets. `generated_detectors(locale)` holds the readers that invert ICU's canonical formatting (each date skeleton, dates with an era among them in ICU's own forms with four-digit years, each interval, compact, spelled-out, and relative form, and ranges of numbers as ICU's `NumberRangeFormatter` writes them: "3–5", "10–15%", "~3"; a set that reads currencies or measures reads their ranges too, "$3–5", "10–15 kg", through `range_detectors`); `flexible_detectors(locale)` holds the flexible readers, which read the forms text writes beyond ICU's own (negative and accounting currency, dates with eras in other layouts, mixed measures, other decimal styles, times with zone names), each reader that takes a currency or unit built for those it chooses from ICU. A unit is read where CLDR's unit preferences give it to a region of the language, and every duration and data-size unit everywhere, so German text's "°F" is not read unless you pass `units=`. The flexible set costs more than the generated one: on the order of 5 to 10 seconds to build for en_US (a few for a language of few locales), so build it once and reuse it, and each `detect` a small multiple of the generated set's (the readers share the numbers they read, kept for the 16 texts read last, about 110 bytes per character each for en_US). For every reading, use both: `generated_detectors(locale).with_(*flexible_detectors(locale).detectors)`. Pass `guarded=True` to `flexible_detectors`, or build `generated_detectors(locale, (*DEFAULT_FAMILIES, *GUARDED_FAMILIES))` (`GUARDED_FAMILIES` alone builds only the guarded readers), to add the readings the default readers refuse on purpose ("May" alone as a month; a year of under four digits beside an era in ICU's own forms, "Mar 15, 44 BC", which the generated readers refuse; a range of such years, "44–45"; or a detached quote or prime mark that ICU writes attached, such as `84 "`). Detached quote and prime readings have their own `measure:detached-unit:<unit>` type and `DETACHED_UNIT_FAMILY`; letter unit symbols retain the default measure reader's flexible spacing.

Recognition reads plain text strings. Markdown, rich text, HTML, XML, and other markup must be turned into text before icukit reads it, by the caller or on the client; icukit has no mode for them. It also has no URL or email detector, so readings can fall inside a URL ("2004" in a path).

### Localization

- **Number Formatting**: Decimal, currency, percent, scientific, spelled-out numbers
- **Date/Time Formatting**: Locale-aware date and time formatting with multiple styles
- **Duration Formatting**: Human-readable time durations ("2 hours, 30 minutes")
- **List Formatting**: Locale-aware list formatting ("A, B, and C")
- **Plural Rules**: Determine plural categories (one, few, many, other) for any locale
- **Message Formatting**: ICU MessageFormat for complex localized strings

### Internationalization Utilities

- **Collation**: Locale-aware string sorting and comparison
- **Locale Information**: Parse, validate, and query locale data
- **Script Detection**: Identify writing scripts in text
- **Bidirectional Text**: Detect and handle RTL/LTR text
- **IDNA**: Internationalized domain name encoding/decoding
- **Spoof Detection**: Detect confusable characters and homograph attacks

### Reference Data

- **Regions**: Country and region codes with containment relationships
- **Scripts**: Writing system information and properties
- **Timezones**: Timezone data with offsets and equivalents
- **Calendars**: Calendar system information (Gregorian, Hebrew, Islamic, etc.)

## Quick Start

### Library API

```python
from icukit import (
    transliterate,
    sort_strings,
    format_number,
    format_datetime,
    get_plural_category,
    break_words,
)

# Transliterate text between scripts
transliterate("Привет мир", "Russian-Latin/BGN")  # "Privet mir"
transliterate("hello", "Latin-Cyrillic")  # "хелло"

# Sort strings with locale-aware collation
sort_strings(["cafe", "café", "CAFE"], "en_US")  # ['cafe', 'CAFE', 'café']
sort_strings(["Öl", "Ol", "öl"], "de_DE")  # ['Ol', 'öl', 'Öl']

# Format numbers for different locales
format_number(1234567.89, "en_US")  # "1,234,567.89"
format_number(1234567.89, "de_DE")  # "1.234.567,89"
format_number(1234567.89, "hi_IN")  # "12,34,567.89"

# Format dates
from datetime import datetime

now = datetime.now()
format_datetime(now, "en_US", style="LONG")  # "January 19, 2026 at 4:00:00 PM PST"
format_datetime(now, "ja_JP", style="LONG")  # "2026年1月19日 16:00:00 PST"

# Determine plural category
get_plural_category(1, "en")  # "one"
get_plural_category(2, "en")  # "other"
get_plural_category(2, "ru")  # "few"
get_plural_category(5, "ru")  # "many"

# Break text into words
break_words("Hello, world!")  # ['Hello', ',', 'world', '!']
```

### Recognition

```python
from icukit import flexible_detectors, generated_detectors, icu_abbreviations

readers = generated_detectors("en_US")
for d in readers.detect("Due March 5, 2024."):
    print(d["start"], d["end"], d["type"], d["text"])
# 4 17 date:yMMMMd March 5, 2024
# 4 11 date:MMMMd March 5

d = readers.detect("Due March 5, 2024.")[0]
# The value: DateTimeValue(fields=(('y', 2024), ('M', 3), ('d', 5)), calendar='gregorian')
d["value"]
# Each part with its own span: (Capture(name='y', start=13, end=17, text='2024', ...), ...)
d["captures"]
# How to write it again: DateFormatSpec(locale='en_US', skeleton='yMMMMd', pattern='MMMM d, y', ...)
d["spec"]

# Every locale of the language by default; `locales=` chooses them, and () is the locale alone
flexible_detectors("en_US").detect("12 kilometres")  # measure:kilometer and number:decimal
flexible_detectors("en_US", locales=()).detect("12 kilometres")  # number:decimal only

# The short forms ICU writes, with their expansions
icu_abbreviations("en_US", kinds=("unit",))  # (..., IcuAbbreviation(surface='m/s²', ...), ...)
# Symbols by CLDR's names in the language, read from a snapshot of CLDR's annotations
icu_abbreviations("de_DE", kinds=("symbol",))  # (..., IcuAbbreviation(surface='&', ...), ...)
```

Offsets are code points. `date_detectors(locale, skeletons)` and `number_detectors(locale, currencies=...)` build smaller sets by hand.

### Command-Line Interface

icukit includes a full-featured CLI accessible via `icukit` or `ik`:

```bash
# Transliterate text
ik transliterate name Russian-Latin/BGN -t "Москва"
# Output: Moskva

# List transliterators
ik transliterate list --name "Russian-.*"

# Recognize typed values in running text
ik detect --currency USD -t 'Paid $1,234.50 on March 5, 2024'

# Add the flexible readers (accounting currency, mixed measures); --locales '' reads en_US alone
ik detect --flexible --locales '' -t 'Paid ($12.50) for 5 ft 3 in'
# --currency USD also reads the strict "$12.50" inside "($12.50)"
ik detect --flexible --locales '' --currency USD -t 'Paid ($12.50)'

# Format numbers
ik locale format 1234567.89 --locale de_DE
# Output: 1.234.567,89
ik compact 1234567 --locale de_DE
# Output: 1,2 Mio.

# Get locale information
ik locale info en_US

# Sort lines with locale collation
cat names.txt | ik sort --locale sv_SE

# Detect the scripts in text
ik script detect --all -t "Hello Мир 世界"

# Get Unicode character information
ik unicode info -t "Aé"

# Look up a character by its name
ik unicode lookup -t "GREEK SMALL LETTER ALPHA"
```

From Python: `char_from_name("GREEK SMALL LETTER ALPHA")` returns `"α"`, `char_from_name("NBSP")` returns `"\xa0"`, `get_char_names("Ƣ")["alias"]` returns `"LATIN CAPITAL LETTER GHA"`, and `get_char_aliases("\x07")` lists `ALERT` (a `control` alias) and `BEL` (an `abbreviation`).

Run `ik help` or `ik <command> --help` for detailed usage information.

## Supported Python Versions

- Python 3.11+
- Tested on Linux and macOS

## Documentation

- [API Reference](https://github.com/lenzo-ka/icukit/blob/main/docs/api.md)
- [CLI Reference](https://github.com/lenzo-ka/icukit/blob/main/docs/cli.md)
- [Language Availability](https://github.com/lenzo-ka/icukit/blob/main/docs/languages.md)
- [Locale Material](https://github.com/lenzo-ka/icukit/blob/main/docs/locale-material.md)
- [English Real-Text Sentence Model](https://github.com/lenzo-ka/icukit/blob/main/docs/real-text-sentence-model.md)

## License

BSD 2-Clause License

The CLDR data bundled in `icukit/data/cldr_symbols` is Unicode, Inc.'s, under the [Unicode License v3](https://github.com/lenzo-ka/icukit/blob/main/icukit/data/cldr_symbols/LICENSE).

The UCD data bundled in `icukit/data/ucd_name_aliases` is Unicode, Inc.'s, under the [Unicode License v3](https://github.com/lenzo-ka/icukit/blob/main/icukit/data/ucd_name_aliases/LICENSE).

The opt-in `en-tn@1` sentence rules, `en-tn-cart@1` synthetic-text model, and
`en-real-cart@1` real-text model in `icukit/data/break_rules/en` are distributed under
[CC BY-SA 4.0](https://creativecommons.org/licenses/by-sa/4.0/); their bundled
`NOTICE` gives attribution and describes the modifications. The TN artifacts
derive from the Google Text Normalization English corpus. The real-text model
derives from commercially compatible GUM documents and UK House of Commons
Hansard; `REAL_TEXT_RECEIPT.json` lists the documents, source URLs, licenses,
and measurements. The model is distributed under CC BY-SA 4.0 because part of
its training text is CC BY-SA; attribution is provided in the bundled `NOTICE`.
No corpus sentence or row is bundled in these artifacts.
