Metadata-Version: 2.5
Name: turkish-linguistic-features
Version: 0.2.0
Summary: A comprehensive toolkit for extracting quantitative linguistic features from Turkish and English text.
Project-URL: Repository, https://github.com/efeziya1/turkish-linguistic-features
Project-URL: Documentation, https://efeziya1.github.io/turkish-linguistic-features/
Project-URL: Issues, https://github.com/efeziya1/turkish-linguistic-features/issues
Author-email: Efe Ziya Okul <efeziya@live.com>
License: MIT
License-File: LICENSE
Keywords: computational-linguistics,corpus-linguistics,digital-humanities,english,feature-extraction,lexical-diversity,lexical-richness,morphology,natural-language-processing,quantitative-linguistics,readability,stylometry,syntactic-complexity,text-analysis,turkish
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Text Processing :: Linguistic
Requires-Python: >=3.10
Requires-Dist: nltk>=3.8
Requires-Dist: numpy>=1.21
Requires-Dist: spacy<4,>=3.8
Requires-Dist: textstat<0.8,>=0.7.10
Requires-Dist: zeyrek<0.2,>=0.1.3
Provides-Extra: all
Requires-Dist: pandas>=1.5; extra == 'all'
Requires-Dist: wordfreq>=3.0; extra == 'all'
Provides-Extra: lexical-freq
Requires-Dist: wordfreq>=3.0; extra == 'lexical-freq'
Provides-Extra: pandas
Requires-Dist: pandas>=1.5; extra == 'pandas'
Description-Content-Type: text/markdown

# turkish-linguistic-features

[![CI](https://github.com/efeziya1/turkish-linguistic-features/actions/workflows/ci.yml/badge.svg)](https://github.com/efeziya1/turkish-linguistic-features/actions/workflows/ci.yml)
[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.23009540.svg)](https://doi.org/10.5281/zenodo.23009540)

Extracts 208 quantitative linguistic features from Turkish text and 181 from
English. Every feature has its formula written out; 197 of the Turkish features
and 170 of the English ones cite a source in the literature, and the rest are
plain definitions (such as the dash's share of punctuation marks). The
[verification report](https://efeziya1.github.io/turkish-linguistic-features/verification-report/) shows which ones have been
checked against a number their source published. Documentation:
https://efeziya1.github.io/turkish-linguistic-features/

**Status: early development (0.x).** Changes: [CHANGELOG](https://github.com/efeziya1/turkish-linguistic-features/blob/main/CHANGELOG.md).

```python
import turkish_linguistic_features as tlf

oz = tlf.analyze("Renksiz yeşil fikirler öfkeyle uyur.", lang="tr")
oz["word_len_mean"]   # 6.2
oz["atesman"]         # 89.355  (Ateşman 1997 readability)
```

## Documentation

**→ [Full documentation](https://efeziya1.github.io/turkish-linguistic-features/)** — tutorial, how-to
guides, feature reference and limitations, in **Turkish and English**.

| | |
|---|---|
| Türkçe | [tr/](https://efeziya1.github.io/turkish-linguistic-features/tr/) |
| English | [en/](https://efeziya1.github.io/turkish-linguistic-features/en/) |
| Feature reference | [reference/features/](https://efeziya1.github.io/turkish-linguistic-features/reference/features/) |
| Verification report | [TR](https://efeziya1.github.io/turkish-linguistic-features/dogrulama-raporu/) · [EN](https://efeziya1.github.io/turkish-linguistic-features/verification-report/) |

## Install

```bash
pip install turkish-linguistic-features
```

### Language data

`pip install` sets up the Python dependencies but not the language data.
Install the data for the language(s) you use, once:

```bash
# Turkish — 156 MB. The model is by turkish-nlp-suite and is not in spaCy's
# registry, so install the wheel directly:
pip install https://huggingface.co/turkish-nlp-suite/tr_core_news_md/resolve/main/tr_core_news_md-1.0-py3-none-any.whl

# English — the spaCy model (12 MB) and the CMU Pronouncing Dictionary for
# syllable counts (1 MB):
python -m spacy download en_core_web_sm
python -m nltk.downloader cmudict

# Optional — for the two wordfreq_* features:
pip install 'wordfreq>=3.0'
```

If something is missing, `analyze()` stops with an error that shows the
command to run.

Verified with spaCy 3.8.16, `en_core_web_sm` 3.8.0 and `tr_core_news_md` 1.0.
The Turkish wheel says `1.0` in its filename but `3.4.2` in its metadata, and
loading it prints a `W094` warning; both are harmless
([details](https://efeziya1.github.io/turkish-linguistic-features/en/getting-started/)).

## The whole API

Eleven public names:

```python
tlf.analyze(text, lang="tr")                     # 208 features (TR) / 181 (EN)
tlf.analyze(text, groups=["readability"])        # one group only
tlf.analyze_corpus("corpus/")                    # one row per file
tlf.analyze_corpus("corpus/", segment_size=1000) # or split into chunks
tlf.save_csv(rows, "features.csv")               # write them out
tlf.segment_text(text, segment_size=1000, lang="tr")  # split a single text
tlf.analyze(text, custom_ngrams=[["ADJ", "NOUN"]])  # count your own phrases
tlf.ngram_matches(text, ["ADJ", "NOUN"])         # ...and see what they matched
tlf.describe_feature("mtld")                     # what a key measures, and its source
tlf.FeatureParams(mattr_window=100)              # metric constants

# tlf.LinguisticFeaturesError  tlf.ModelNotFoundError
# tlf.MissingDependencyWarning  tlf.ParagraphStructureWarning
```

Full signatures: [Public API](https://efeziya1.github.io/turkish-linguistic-features/reference/api/).

## Development

```bash
pip install -r requirements.txt
pytest                          # pytest -m "" also runs the slow test
ruff check .
python scripts/generate_verification_report.py  # regenerate the verification reports
python scripts/generate_feature_reference.py    # regenerate the feature reference
```

## License

MIT — see `LICENSE`.
