Metadata-Version: 2.4
Name: pyutils-collection
Version: 0.2.0
Summary: Python utilities: 500+ type-annotated functions across 31 modules for async operations, data processing, file handling, security, and more
Author: Mykyta Forofontov
License-Expression: MIT
Project-URL: Homepage, https://github.com/MForofontov/pyutils-collection
Project-URL: Repository, https://github.com/MForofontov/pyutils-collection
Project-URL: Issues, https://github.com/MForofontov/pyutils-collection/issues
Project-URL: Documentation, https://github.com/MForofontov/pyutils-collection#readme
Project-URL: Author Profile, https://github.com/MForofontov
Keywords: utilities,utils,helpers,asyncio,database,file-handling,data-validation,compression,datetime,decorators
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.15
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Utilities
Classifier: Typing :: Typed
Requires-Python: >=3.15
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: cramjam==2.14.0
Requires-Dist: python-snappy==0.7.3
Requires-Dist: psutil==7.2.2
Requires-Dist: tqdm==4.70.1
Requires-Dist: numpy==2.5.4
Requires-Dist: matplotlib==3.11.2
Requires-Dist: aiohttp==3.14.5
Requires-Dist: pytz==2026.5
Requires-Dist: sqlalchemy==2.1.6
Requires-Dist: toml==0.10.2
Requires-Dist: pyyaml==6.0.3
Requires-Dist: pydantic==2.14.1
Requires-Dist: cerberus==1.3.8
Requires-Dist: bcrypt==5.0.0
Requires-Dist: cryptography==50.0.2
Requires-Dist: PyJWT==2.15.1
Requires-Dist: dnspython==2.9.0
Requires-Dist: requests==2.34.2
Requires-Dist: paramiko==5.1.0
Requires-Dist: beautifulsoup4==4.15.0
Requires-Dist: lxml==6.1.3
Requires-Dist: openpyxl==3.1.5
Requires-Dist: pyarrow==26.0.0
Requires-Dist: scipy==1.18.1
Requires-Dist: scikit-learn==1.9.1
Provides-Extra: dev
Requires-Dist: pytest==9.1.1; extra == "dev"
Requires-Dist: pytest-asyncio==1.4.1; extra == "dev"
Requires-Dist: pytest-cov==7.1.0; extra == "dev"
Requires-Dist: allure-pytest==2.16.2; extra == "dev"
Requires-Dist: allure-python-commons==2.16.2; extra == "dev"
Requires-Dist: mypy==2.4.0; extra == "dev"
Requires-Dist: ruff==0.17.3; extra == "dev"
Requires-Dist: pylint==4.1.3; extra == "dev"
Requires-Dist: pytest-xdist==3.8.0; extra == "dev"
Requires-Dist: bandit==1.9.4; extra == "dev"
Requires-Dist: build==1.6.1; extra == "dev"
Requires-Dist: twine==7.0.0; extra == "dev"
Requires-Dist: pip-audit==2.10.1; extra == "dev"
Requires-Dist: attrs==26.1.0; extra == "dev"
Requires-Dist: iniconfig==2.3.1; extra == "dev"
Requires-Dist: packaging==26.3; extra == "dev"
Requires-Dist: pluggy==1.7.0; extra == "dev"
Provides-Extra: playwright
Requires-Dist: playwright==1.64.1; extra == "playwright"
Dynamic: license-file

# 🛠️ Pyutils Collection

[![PyPI version](https://img.shields.io/pypi/v/pyutils-collection.svg)](https://pypi.org/project/pyutils-collection/)
[![Python versions](https://img.shields.io/pypi/pyversions/pyutils-collection.svg)](https://pypi.org/project/pyutils-collection/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)

> **Python utilities** - 500+ functions across 31 specialized modules for async operations, data processing, file handling, security, and more.

## 🎯 What is This?

A collection of **500+ utility functions** across **31 specialized modules**, designed for **copy-paste reuse** or **pip install**. Functions include type hints, docstrings, and focused tests.

**Philosophy:**
- 📋 **Copy-paste friendly** - Functions work standalone
- 🔒 **Type-safe** - Complete type hints (Python 3.15+)
- 📝 **Self-documenting** - NumPy-style docstrings with examples
- ✅ **Tested** - 5,500+ unit tests; measured coverage and skips are recorded in the maintenance plan
- 🎨 **Optional browser support** - Install the Playwright extra for browser utilities

## 📦 Quick Start

```bash
# Install from PyPI
pip install pyutils-collection

# Or clone and copy what you need
git clone https://github.com/MForofontov/pyutils-collection.git
cd pyutils-collection/pyutils_collection

# Or install for development
pip install -e ".[dev]"
```

## 📦 Modules Overview

### Companion utilities

```python
from datetime import date

from pyutils_collection.iterable_functions import unique_by, unflatten_dict, deep_delete
from pyutils_collection.datetime_functions import date_range
from pyutils_collection.url_functions import remove_query_params
from pyutils_collection.file_functions import atomic_write_file
from pyutils_collection.asyncio_functions import async_map_limited
from pyutils_collection.env_config_functions import validate_env_vars

unique_by(["Ada", "ADA", "Grace"], str.lower)  # ["Ada", "Grace"]
data = unflatten_dict({"user_name": "Ada", "user_age": 36})
deep_delete(data, "user.age")  # data is now {"user": {"name": "Ada"}}
list(date_range(date(2026, 1, 1), date(2026, 1, 4)))  # Stop is exclusive
remove_query_params("/search?q=python&utm_source=mail", ["utm_source"])
atomic_write_file("settings.json", '{"enabled": true}')
config = validate_env_vars(
    {"PORT": int, "DEBUG": bool},
    defaults={"PORT": 8080, "DEBUG": False},
)

async def double(value: int) -> int:
    return value * 2

# In an async function: await async_map_limited(double, range(100), limit=5)
```

`unflatten_dict` rejects conflicting paths and uses the same separator as
`flatten_dict`. `atomic_write_file` requires an existing parent directory and
uses temporary-file permissions. Environment defaults are already-converted
values; malformed present values raise an error instead of using a default.

### Selected utility families

| Module | Count | Description |
|--------|-------|-------------|
| 🔄 **asyncio_functions** | 17 | Async/await, connection pools, rate limiting, HTTP |
| 🗜️ **compression_functions** | 27 | GZIP, BZ2, LZMA, Snappy, Zstandard, polyline encoding |
| 🗄️ **database_functions** | 23 | SQLAlchemy utils, transactions, schema inspection |
| 📅 **datetime_functions** | 27 | Timezone conversion, business days, humanization |
| 🎨 **decorators** | 50+ | Caching, retry, timeout, type checking, profiling |
| 📁 **file_functions** | 32 | I/O, hashing, search, temp files, format conversion |
| 🌐 **http_functions** | 9 | REST operations, downloads, query strings |
| 🔄 **iterable_functions** | 55 | Chunking, filtering, grouping, flattening |
| 🧮 **mathematical_functions** | 5 | GCD, LCM, primes, factorial, fibonacci |
| 🔐 **security_functions** | 12 | Encryption (AES/RSA), hashing, JWT tokens |
| 📊 **serialization_functions** | 28 | CSV, Excel, Parquet with streaming & conversion |
| 🔌 **ssh_functions** | 12 | Remote execution, SFTP, key generation |
| 🧪 **testing_functions** | 24 | Fixtures, mocks, assertions, test data generators |
| 🌍 **network_functions** | 28 | IP utilities, DNS, port scanning, connectivity |
| 🌐 **web_scraping_functions** | 18 | HTML/CSS/XPath parsing, table extraction |
| 🎭 **playwright_functions** | 6 | Browser automation, screenshots, session management |
| 🔗 **url_functions** | 8 | Parse, build, validate, normalize URLs |
|  **regex_functions** | 5 | Email/phone/URL validation & extraction |
| ⚙️ **cli_functions** | 16 | System info, process management, environment vars |
| 📝 **logger_functions** | 7 | Logger setup, function logging, rotation |
| 🔄 **multiprocessing_functions** | 19 | Parallel processing, pool management |
| 🔧 **batch_processing_functions** | 2 | Chunked processing, streaming aggregation |
| 🌿 **env_config_functions** | 6 | Config loading (env, YAML, TOML) |
| ✅ **data_validation** | Many | Type/schema validation, Pydantic/Cerberus support |

## 🔑 Key Features

### Database-Agnostic Design
All database functions use **SQLAlchemy** for maximum portability:
- ✅ PostgreSQL
- ✅ MySQL / MariaDB
- ✅ SQLite
- ✅ Oracle
- ✅ SQL Server

### Type Safety
- Complete type hints using modern Python syntax (`list[str]`, `dict[str, Any]`)
- Runtime type checking with decorators
- mypy-compliant codebase

### Comprehensive Testing
- Package coverage reports include paths not exercised on the current platform
- 5,500+ unit tests across the utility families
- Pytest-based testing framework
- Comprehensive edge case coverage

### Documentation
- NumPy-style docstrings for all functions
- Examples in docstrings
- Time/space complexity notes for algorithms
- Comprehensive README with usage examples

## 📚 Usage Examples

### Database Operations
```python
from sqlalchemy import create_engine, text
from pyutils_collection.database_functions import atomic_transaction
from pyutils_collection.database_functions.schema_inspection import (
    get_table_info, find_duplicate_rows,
)

engine = create_engine("sqlite:///:memory:")
try:
    with engine.connect() as conn:
        with atomic_transaction(conn):
            conn.execute(text("CREATE TABLE users (id INTEGER PRIMARY KEY, email TEXT)"))
            conn.execute(
                text("INSERT INTO users (email) VALUES (:email)"),
                [{"email": "ada@example.com"}, {"email": "ada@example.com"}],
            )
        print(get_table_info(conn, "users"))
        print(find_duplicate_rows(conn, "users", ["email"]))
finally:
    engine.dispose()
```

### Async Operations
```python
import asyncio
from pyutils_collection.asyncio_functions import async_batch, async_map_limited

async def double_batch(items: list[int]) -> list[int]:
    return [item * 2 for item in items]

async def double(item: int) -> int:
    return item * 2

async def main() -> None:
    print(await async_batch(double_batch, list(range(5)), batch_size=2))
    print(await async_map_limited(double, range(5), limit=2))

asyncio.run(main())
```

### Decorators
```python
from pyutils_collection.decorators import cache, retry, timeout, enforce_types

@cache
@retry(max_retries=3, delay=0.01)
@timeout(seconds=30)
@enforce_types()
def fetch_user_data(user_id: int) -> dict:
    # Function logic here
    return {"id": user_id, "name": "John"}
```

### File Operations
```python
from pathlib import Path
from tempfile import TemporaryDirectory
from pyutils_collection.file_functions import (
    atomic_write_file, read_lines, calculate_sha256_hash, find_files_by_pattern,
)

with TemporaryDirectory() as directory:
    path = Path(directory) / "data.txt"
    atomic_write_file(str(path), "first line\nsecond line\n")
    print(read_lines(str(path)))
    print(calculate_sha256_hash(str(path)))
    print(find_files_by_pattern(directory, "*.txt"))
```

### Data Serialization
```python
from pathlib import Path
from tempfile import TemporaryDirectory
from pyutils_collection.serialization_functions import stream_csv_chunks, csv_to_parquet

with TemporaryDirectory() as directory:
    source = Path(directory) / "scores.csv"
    destination = Path(directory) / "scores.parquet"
    source.write_text("name,score\nAda,10\nGrace,12\n", encoding="utf-8")
    for chunk in stream_csv_chunks(str(source), chunk_size=1):
        print(chunk)
    print(csv_to_parquet(str(source), str(destination), compression="snappy"))
```

## 📋 Requirements

- **Python**: 3.15+
- Use a Python build that includes `compression.zstd`; official CPython builds
  provide it. Zstandard helpers now use this standard-library implementation,
  preserving support for previously compressed data.
- A package installation installs the dependencies declared in `pyproject.toml`, including scientific, database, SSH, and security libraries.
- Browser automation is optional: install with `pip install '.[playwright]'` from a checkout or `pip install 'pyutils-collection[playwright]'` from PyPI.

### Maintenance behavior changes

Paramiko SSH helpers verify server keys against `~/.ssh/known_hosts` and reject unknown or changed keys. CLI SSH helpers use the system client's host-key policy and reject option-like destinations. HTTP helpers accept only HTTP(S) URLs and redirects, stripping credentials when a redirect changes origin. Bcrypt hashing rejects passwords over 72 UTF-8 bytes, and verification returns `False` for those passwords. Parquet partitioning preserves null groups and raises `ValueError` before writing files when partition values produce colliding filenames.

Security dependency updates require modern SSH algorithms; Paramiko 5 no longer supports SHA-1 signatures/key exchange or DSA keys. Python 3.15 is now the minimum supported version. Installations on older Python versions are intentionally rejected. See the maintenance plan for migration evidence and remaining release checks.

Public function parameters remain unchanged. See [CHANGELOG.md](CHANGELOG.md) for the full list of corrections.

Async downloads preserve existing destinations on failure. Async-pool capacity
includes checked-out connections; duplicate registration and foreign or duplicate
release raise `ValueError`. ZIP extraction rejects unsafe member paths and
existing symlink escapes; TAR extraction preflights members before writing.

## 🧪 Testing

```bash
# Install development dependencies
python -m pip install -e '.[dev]'

# Unit tests and package coverage
python scripts/run_with_deadline.py --seconds 900 --log .major-check/unit.log -- python -m pytest pytest/unit --cov=pyutils_collection --cov-report=html --junitxml=.major-check/unit.xml

# Separate performance checks and optional stress tests
python -m pytest pytest/performance -m 'performance and not slow'
python -m pytest pytest/performance -m 'performance and slow'

# Isolated local HTTP integration checks
python -m pytest pytest/integration -q

# Audit inventory, Python 3.15 wheels, and repeatable async-map benchmarks
python scripts/audit_inventory.py .major-check/unit.xml
python scripts/check_python315_wheels.py
python scripts/benchmark_async_map.py
python scripts/benchmark_sqlite_migration.py

# Static checks
python -m ruff check pyutils_collection scripts
python -m ruff format --check pyutils_collection scripts
python -m mypy pyutils_collection --install-types --non-interactive
```

See [MAINTENANCE_PLAN.md](MAINTENANCE_PLAN.md) for verified results, platform coverage, and outstanding audit work. Skipped tests and unavailable integration checks are listed separately from passes.

## 🤝 Contributing

See [`.github/copilot-instructions.md`](.github/copilot-instructions.md) for detailed guidelines:
- NumPy-style docstrings with examples
- Complete type hints (Python 3.15+ syntax)
- 95%+ test coverage per function
- Self-contained, copy-paste friendly code

## 📄 License

MIT © 2026 [Mykyta Forofontov](https://github.com/MForofontov)

See [LICENSE](LICENSE) for the full license text.

## 👤 Author

**Mykyta Forofontov**
- GitHub: [@MForofontov](https://github.com/MForofontov)

## 🔗 Links

- **Repository**: https://github.com/MForofontov/pyutils-collection
- **Issues**: https://github.com/MForofontov/pyutils-collection/issues
- **Documentation**: https://github.com/MForofontov/pyutils-collection#readme

---

⭐ **Star this repository** if you find it useful!
