Metadata-Version: 2.5
Name: sentence-struct
Version: 0.6.0
Summary: Multilingual sentence structure analysis: sentences with tokens and syntactic chunks
Project-URL: Homepage, https://github.com/memshare-project/sentence-struct
Project-URL: Repository, https://github.com/memshare-project/sentence-struct
Project-URL: Issues, https://github.com/memshare-project/sentence-struct/issues
Project-URL: Changelog, https://github.com/memshare-project/sentence-struct/blob/main/CHANGELOG.md
Author: sentence-struct contributors
License-Expression: MIT
License-File: LICENSE
Keywords: chinese,chunking,english,ipa,japanese,korean,linguistics,nlp,pinyin,romanization,sentence-structure,thai,tokenization
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Text Processing :: Linguistic
Requires-Python: >=3.11
Requires-Dist: cmudict>=1.0.13
Requires-Dist: ginza>=5.2.1
Requires-Dist: ja-ginza>=5.2.0
Requires-Dist: jieba>=0.42.1
Requires-Dist: kiwipiepy>=0.23.2
Requires-Dist: koroman>=1.0.16
Requires-Dist: pypinyin>=0.55.0
Requires-Dist: pythainlp>=5.3.7
Requires-Dist: spacy<3.9.0,>=3.8.0
Requires-Dist: sudachidict-core>=20240409
Requires-Dist: sudachipy>=0.6.8
Provides-Extra: dev
Requires-Dist: pytest-cov>=6.0; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Provides-Extra: ja-full
Requires-Dist: sudachidict-full>=20240409; extra == 'ja-full'
Description-Content-Type: text/markdown

# sentence-struct

Analyze text into a learner-friendly structure:

**document → sentences[] → { tokens[], chunks[] }**

Chunks are siblings of tokens and reference them via `tokenIndices` (not nested).

Japanese (`ja`) via GiNza / Sudachi (furigana + POS coloring).

Other languages tokenize with transparent `posGroup`/`typeGroup` `"_"` (one
chunk per token). Some also fill `reading`:

| Lang | Tokenizer | `reading` |
|------|-----------|-----------|
| `en` | spaCy `en_core_web_sm` | IPA (`/…/`; POS only for homographs) |
| `ko` | Kiwi | Revised Romanization (`koroman`) |
| `th` | PyThaiNLP `newmm` | romanize |
| `zh` | jieba | pinyin (tone marks) |
| `de` `fr` `it` `es` `tr` `ru` `ar` | spaCy `blank` | — (tokenize only) |

## Install

```bash
pip install sentence-struct
# or
uv add sentence-struct
```

English IPA needs the spaCy small model:

```bash
python -m spacy download en_core_web_sm
```

(Local uv: `uv sync --group en-model`.)

Optional larger Sudachi dictionary:

```bash
pip install "sentence-struct[ja-full]"
```

Requires Python ≥ 3.11. First install pulls spaCy / GiNza / Sudachi (hundreds of MB).

## Usage

```python
from sentence_struct import analyze

doc = analyze("秋が近づくにつれ、朝晩の涼しさが心地よい。", language="ja")
print(doc["sentence_count"], doc["token_count"], doc["chunk_count"])
print(doc["sentences"][0]["tokens"][0])
print(doc["sentences"][0]["chunks"][0])

en = analyze("I refuse the refuse.", language="en")
ko = analyze("안녕하세요", language="ko")
th = analyze("สวัสดีครับ", language="th")
zh = analyze("你好，世界。", language="zh")
```

CLI:

```bash
sentence-struct "秋が近づく。"
sentence-struct -f essay.txt -o out.json
```

## Schema (abbreviated)

```json
{
  "language": "ja",
  "text": "...",
  "sentences": [
    {
      "index": 1,
      "text": "...",
      "tokens": [
        {"text": "秋", "pos": "名詞", "posGroup": "名詞", "lemma": "秋", "reading": "あき"}
      ],
      "chunks": [
        {"type": "NP", "typeGroup": "PHRASE", "tokenIndices": [0, 1], "text": "秋が", "role": "nsubj"}
      ]
    }
  ]
}
```

## Develop (uv)

```bash
uv sync
uv run pytest -m "not integration"
uv run pytest -m integration
uv run sentence-struct "今日は良い天気です。"
```

## Release

See [RELEASING.md](RELEASING.md). Tag `vX.Y.Z` → GitHub Actions builds with **uv** and publishes to PyPI via Trusted Publishing.

## License

MIT
