Metadata-Version: 2.4
Name: polyglean
Version: 0.1.0
Summary: A universal text cleaning and dataset preparation library for any language or script.
Author-email: Rasheedat Sikiru <rasheedatsikiru@gmail.com>
License-Expression: MIT
Project-URL: Source Code, https://github.com/rasheedatsikiru/polyglean
Project-URL: Bug Tracker, https://github.com/rasheedatsikiru/polyglean/issues
Keywords: nlp,text-cleaning,tts,corpus,multilingual,low-resource,african-languages,dataset,voice-data
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Text Processing :: Linguistic
Classifier: Natural Language :: English
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pandas>=2.0
Requires-Dist: openpyxl>=3.1
Requires-Dist: pyarrow>=14.0
Provides-Extra: dev
Requires-Dist: build; extra == "dev"
Requires-Dist: twine; extra == "dev"
Dynamic: license-file

# polyglean

A universal text cleaning and dataset preparation library for any language or script.

Built for TTS (text-to-speech) voice data pipelines but useful for any NLP corpus work.  
Handles Latin, Arabic, Devanagari, CJK, and any Unicode-based script correctly.

---

## Installation

```bash
pip install polyglean
```

---

## Quick start

```python
import polyglean

config = {
    "base_dir":       "/path/to/your/data/",
    "output_dir":     "/path/to/output/",
    "source_config":  [
        ("bbc_news.xlsx", "sentence", "bbc"),
        ("bible.xlsx",    "text",     "bible"),
    ],
    "separate_files": [],
    "extra_cleaners": [],
    "word_count_min": 5,
    "word_count_max": 20,
    "output_tts":   "my_language_tts.xlsx",
    "output_other": "my_language_other.xlsx",
}

polyglean.run_language("my_language", config)
```

Two output files are created:
- **`my_language_tts.xlsx`** — sentences within the word count window (TTS-ready)
- **`my_language_other.xlsx`** — sentences outside the window (for further splitting)

---

## CLI usage

Register your language config, then use the command-line tools:

```bash
# Run the full cleaning pipeline
polyglean-clean --lang yoruba

# Process multiple languages
polyglean-clean --lang yoruba hausa igbo

# Split long sentences into TTS-ready chunks
polyglean-split --lang yoruba
```

---

## Supported file formats

| Type key             | Format                        | Extra options              |
|----------------------|-------------------------------|----------------------------|
| `excel` (default)    | `.xlsx` / `.xls`              | `col` (column name)        |
| `csv`                | `.csv`                        | `col`, `sep`, `encoding`   |
| `parquet`            | `.parquet`                    | `col`                      |
| `json_list`          | JSON array of objects         | `col` (field to extract)   |
| `json_articles_body` | JSON with `articles[].body`   | —                          |
| `txt`                | Plain text, one sentence/line | —                          |

```python
source_config = [
    # Excel (shorthand tuple)
    ("corpus.xlsx", "sentence", "corpus"),

    # CSV with tab separator
    {"path": "data.tsv", "col": "text", "label": "web", "type": "csv", "sep": "\t"},

    # Parquet
    {"path": "data.parquet", "col": "sentence", "label": "hf", "type": "parquet"},

    # JSON array
    {"path": "stories.json", "col": "story_text", "label": "stories", "type": "json_list"},

    # Plain text
    {"path": "sentences.txt", "label": "misc", "type": "txt"},
]
```

---

## What the cleaning pipeline does

Applied to every sentence, in order:

1. **Unicode NFC normalization** — prevents false duplicates from diacritic encoding differences  
2. **Control character removal** — strips non-printable characters  
3. **Newline flattening** — collapses multi-line text to a single line  
4. **Bracket content removal** — removes `[footnote]`, `[210]`, etc.  
5. **Digit removal** — removes numerals (TTS-specific; see below to opt out)  
6. **Whitespace collapsing** — normalizes spacing  

Then:
- All-caps rows are dropped (headers, acronym-only rows)
- Empty sentences are dropped
- Duplicates are removed (first occurrence kept)

### Adding language-specific cleaners

```python
import re

config["extra_cleaners"] = [
    # Strip a prefix specific to your dataset
    lambda text: re.sub(r"^Chapter\s+\d+:\s*", "", text),

    # Remove URLs from web-scraped sources
    from polyglean.cleaners import remove_urls
    remove_urls,
]
```

### Keeping digits (non-TTS use)

```python
from polyglean.cleaners import BASE_CLEANERS, remove_digits

config["extra_cleaners"] = []
# Pass a custom cleaner list that excludes remove_digits:
CLEANERS_NO_DIGITS = [fn for fn in BASE_CLEANERS if fn is not remove_digits]
```

---

## Using the step-by-step API

```python
from polyglean.loaders import load_sources
from polyglean.pipeline import clean_data, split_and_save, split_other_sentences

df = load_sources(config["base_dir"], config["source_config"])
df = clean_data(df, config.get("extra_cleaners", []))
split_and_save(df, config)

# Later — chunk the long sentences
split_other_sentences("my_language", config)
```

---

## License

MIT © Rasheedat Sikiru
