Metadata-Version: 2.4
Name: teadata
Version: 0.0.127
Summary: Pythonic data engine for Texas education datasets
Author-email: Alejandro Peña <adpena@gmail.com>
License-Expression: Apache-2.0
Project-URL: Homepage, https://github.com/adpena/teadata
Project-URL: Bug Tracker, https://github.com/adpena/teadata/issues
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Programming Language :: Python :: 3.15
Classifier: Operating System :: OS Independent
Classifier: Intended Audience :: Developers
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Classifier: Topic :: Scientific/Engineering :: GIS
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: python-dateutil>=2.8.2
Requires-Dist: numpy!=2.4.0,>=1.26; python_version < "3.15"
Requires-Dist: numpy>=2.5.2; python_version >= "3.15"
Requires-Dist: pandas>=2.1; python_version < "3.15"
Requires-Dist: pandas>=3.0.6; python_version >= "3.15"
Requires-Dist: shapely>=2.0; python_version < "3.15"
Requires-Dist: shapely>=2.2.0; python_version >= "3.15"
Requires-Dist: geopandas>=0.14
Requires-Dist: pyproj>=3.8.0; python_version >= "3.15"
Requires-Dist: certifi>=2026.10.19
Requires-Dist: pyogrio>=0.7; python_version < "3.15"
Requires-Dist: pyogrio>=0.13.0; python_version >= "3.15"
Requires-Dist: pyyaml>=6.0
Requires-Dist: openpyxl>=3.1
Requires-Dist: duckdb>=0.10; python_version < "3.15"
Requires-Dist: duckdb>=2.0.0; python_version >= "3.15"
Requires-Dist: pyarrow>=23.0.1; python_version < "3.15"
Requires-Dist: pyarrow>=26.0.0; python_version >= "3.15"
Requires-Dist: packaging>=23.2
Requires-Dist: scipy>=1.10; python_version < "3.15"
Requires-Dist: scipy>=1.18.1; python_version >= "3.15"
Requires-Dist: platformdirs>=3.7
Requires-Dist: tomli>=2.0
Provides-Extra: speedups
Requires-Dist: polars>=1.0; extra == "speedups"
Provides-Extra: dev
Requires-Dist: ruff>=0.5; extra == "dev"
Requires-Dist: ty>=0.0.11; extra == "dev"
Requires-Dist: pytest>=9.0.3; extra == "dev"
Requires-Dist: pytest-cov>=4.1; extra == "dev"
Requires-Dist: pip-tools>=7.4; extra == "dev"
Requires-Dist: pip-audit>=2.7; extra == "dev"
Requires-Dist: deptry>=0.16; extra == "dev"
Requires-Dist: msgpack>=1.2.3; extra == "dev"
Requires-Dist: requests>=2.33.0; extra == "dev"
Requires-Dist: idna>=3.15; extra == "dev"
Requires-Dist: urllib3>=2.8.0; extra == "dev"
Requires-Dist: filelock>=4.2.1; extra == "dev"
Requires-Dist: pip>=26.2.0; extra == "dev"
Requires-Dist: pygments>=2.20.0; extra == "dev"
Requires-Dist: setuptools>=84.1.0; extra == "dev"
Requires-Dist: wheel>=0.46.2; extra == "dev"
Provides-Extra: docs
Requires-Dist: mkdocs-material>=9.7.7; extra == "docs"
Requires-Dist: mkdocstrings[python]>=0.24; extra == "docs"
Requires-Dist: markupsafe>=3.0.4; python_version >= "3.15" and extra == "docs"
Requires-Dist: pygments>=2.20.0; extra == "docs"
Requires-Dist: pymdown-extensions>=11.0.1; extra == "docs"
Requires-Dist: requests>=2.33.0; extra == "docs"
Requires-Dist: idna>=3.15; extra == "docs"
Requires-Dist: urllib3>=2.8.0; extra == "docs"
Provides-Extra: notebooks
Requires-Dist: jupyterlab>=4.5.11; extra == "notebooks"
Requires-Dist: ipython>=8.0; extra == "notebooks"
Requires-Dist: ipywidgets>=8.0; extra == "notebooks"
Requires-Dist: ipyleaflet>=0.18; extra == "notebooks"
Requires-Dist: itables>=1.6; extra == "notebooks"
Requires-Dist: google-auth>=2.0; extra == "notebooks"
Requires-Dist: google-api-python-client>=2.0; extra == "notebooks"
Requires-Dist: cffi>=2.1.0; python_version >= "3.15" and extra == "notebooks"
Requires-Dist: markupsafe>=3.0.4; python_version >= "3.15" and extra == "notebooks"
Requires-Dist: rpds-py>=2026.5.1; python_version >= "3.15" and extra == "notebooks"
Requires-Dist: jupyter-server>=2.21.0; extra == "notebooks"
Requires-Dist: nbconvert>=7.17.2; extra == "notebooks"
Requires-Dist: mistune>=3.3.3; extra == "notebooks"
Requires-Dist: bleach>=6.4.0; extra == "notebooks"
Requires-Dist: soupsieve>=2.9.0; extra == "notebooks"
Requires-Dist: anyio>=4.14.2; extra == "notebooks"
Requires-Dist: tornado>=6.6; extra == "notebooks"
Requires-Dist: requests>=2.33.0; extra == "notebooks"
Requires-Dist: idna>=3.15; extra == "notebooks"
Requires-Dist: urllib3>=2.8.0; extra == "notebooks"
Requires-Dist: httplib2>=0.32.0; extra == "notebooks"
Requires-Dist: pyasn1>=0.6.4; extra == "notebooks"
Requires-Dist: protobuf>=6.33.5; extra == "notebooks"
Requires-Dist: pygments>=2.20.0; extra == "notebooks"
Requires-Dist: setuptools>=84.1.0; extra == "notebooks"
Provides-Extra: database
Requires-Dist: sqlalchemy>=2.0; extra == "database"
Requires-Dist: psycopg[binary]>=3.1; python_version < "3.15" and extra == "database"
Requires-Dist: psycopg[binary]>=3.3.6; python_version >= "3.15" and extra == "database"
Provides-Extra: scripts
Requires-Dist: requests>=2.33.0; extra == "scripts"
Requires-Dist: tqdm>=4.66; extra == "scripts"
Requires-Dist: geopy>=2.5.0; extra == "scripts"
Requires-Dist: aiohttp>=3.14.4; extra == "scripts"
Requires-Dist: multidict>=6.9.1; extra == "scripts"
Requires-Dist: idna>=3.15; extra == "scripts"
Requires-Dist: urllib3>=2.8.0; extra == "scripts"
Dynamic: license-file

# TEA Data Engine (`teadata`)

`teadata` is a snapshot-first Python engine for Texas education data.
It provides:

- `District` and `Campus` domain models
- a fluent query DSL using `>>`
- geospatial lookups (nearest charter, campuses in district boundaries, private-school overlap)
- config-driven enrichment from TAPR, accountability, transfers, PEIMS financials, and closure datasets
- sidecar sqlite stores for fast boundary/map/entity lookup

## Installation

Python 3.11 through 3.15 are supported. Development and local checks default to
the exact Python version in `.python-version` (currently 3.14.10).
`TEADATA_CHECK_PYTHON` overrides that default for compatibility checks. The lock
retains interpreter-specific versions where dependency compatibility requires
them. On macOS 3.15, the optional documentation tools build
Watchdog's native FSEvents extension and require the
[documented Xcode tools](https://pypi.org/project/watchdog/#dependencies).
PyYAML's supported pure Python implementation is tested on 3.15.

### PyPI

```bash
uv venv
uv pip install teadata==0.0.127
```

### Development (recommended)

```bash
GIT_LFS_SKIP_SMUDGE=1 git clone https://github.com/adpena/teadata.git
cd teadata
uv sync --extra dev --extra database --extra scripts --extra speedups
```

Source datasets are separate from the checkout; a verified release bundle is enough
for runtime use. Maintenance scripts default to `teadata/data/` beside the
installed package, independent of the current working directory. Set
`TEADATA_DATA_DIR` to use another dataset tree, or pass the scripts' explicit
input/output options. See [Configuration & Resolution](docs/config.md#maintenance-script-paths)
for examples and onboarding details.

### Dependency security

`pyproject.toml` publishes security floors for the core and each optional extra;
`uv.lock` retains the reviewed versions across supported interpreters. Validate
the selected environment with `scripts/check`; changing the lock also requires
a new producer-bound data release before publishing replacement artifacts.

The notebook extra uses Bleach 6.4 for its published fixes. Its opt-in
`parse_email=True` linkifier still has a known
[regex denial-of-service issue](https://github.com/mozilla/bleach/security/advisories/GHSA-g75f-g53v-794x).
TEA Data does not enable that path, and its nbconvert dependency uses Bleach's
HTML/CSS sanitization. The upstream limitation remains relevant to notebook
code that explicitly enables email linkification; upgrading Bleach alone does
not resolve it.

### Existing relational stores

Publisher rating categories are stored as `TEXT`, including full labels such as
`Not Rated: Minimum Size rules`. Existing PostgreSQL stores with `VARCHAR(16)`
rating columns need a caller-owned schema migration widening **both**
`districts.rating` and `campuses.rating` to `TEXT`, or a fresh destination.
`ensure_schema()` creates missing tables; it does not alter existing columns.
TEA Data does not migrate user databases automatically or truncate these labels.
Exports leave transaction commit and rollback to the caller.

### Campus identity and affiliation

`Campus.lookup_key` and `DataEngine.campus_by_key()` use nine-digit TEA campus
numbers for public schools and source-qualified keys such as
`tefa:rec1kXXvIucG7UkoA` for private schools. Private source IDs retain their exact
case and prefixes; URI-component encoding preserves other characters. The
shared identifiers helper validates the encoded ASCII key's 1024-byte limit.
`campus_number` is null for private schools. Their `source_namespace` and
`source_id` are explicit fields, and their governing `district_id` and
`district_number` are null. A directory's reported district is retained separately
as an unverified directory claim, never as governance.

Public campus UUIDs, numbers, and URLs remain unchanged. Private UUIDs derive
from their source-qualified identity. Existing entity, map, and snapshot bundles
using fabricated private TEA numbers require a source-bound rebuild; no numeric
alias is provided. Analytical schema 2 uses the exact campus lookup key as
`entity_id`, with separate nullable TEA-number and source-identity attributes.
Existing relational destinations also need the caller-owned identity-schema
migration or a fresh destination; `ensure_schema()` does not alter old tables.

Private comparisons use `private_campuses_within()`, which includes points inside
or on district boundaries. The former private district-membership operator is
retired. Private profile maps report every covering traditional public district
(`INDEPENDENT` or `COMMON`) and the boundary dataset/version; overlaps are not
collapsed to a chosen parent. Charter service areas are not attendance-boundary
comparators. Schools without coordinates remain queryable, with no geographic
comparison or guessed district.

The optional native PostgreSQL regression uses `TEADATA_TEST_POSTGRES_URL`
pointing to a fresh empty `teadata_test_*` database. It is separate from the
default SQLite checks because SQLite does not enforce declared `VARCHAR` widths.

## Quick Start

Python packages contain code and metadata; data bundles are distributed
separately. Select an immutable bundle before running the example:

```bash
export TEADATA_RELEASE_URL="https://github.com/adpena/teadata/releases/download/v0.0.127/release.json"
```

For a downloaded bundle, set `TEADATA_RELEASE` to its local `release.json`
instead. The manifest binds the selected artifacts to their content hashes.

```python
from teadata import DataEngine

# Load the configured, hash-verified release.
engine = DataEngine.load_default()

# Look up a six-digit TEA district number.
aldine = (engine >> ("district", "101902")).first()
if aldine is not None:
    print(aldine.name)

    # Campuses assigned to this district.
    for campus in aldine.campuses[:5]:
        print(campus.name, campus.campus_number)
```

The district operation returns a `Query`. `.first()` selects its first match or
returns `None`; iterate the query to inspect every match when a name or wildcard
can identify multiple districts.

`DataEngine()` creates an empty engine by default (`autoload=False`).
`load_default()` and `from_snapshot()` without a path use the configured release:
`TEADATA_RELEASE`, then `TEADATA_RELEASE_URL`, then the package's
`teadata/.cache/release.json`, unless a release is already pinned by
`using_release()`.

## Public API Surface

Primary imports:

```python
from teadata import DataEngine, District, Campus
```

Core behaviors:

- `DataEngine.load_default()` loads the snapshot and boundary store selected by the configured release manifest.
- `DataEngine.from_snapshot(path)` loads a trusted local catalog-bound snapshot; omit the path to use the configured release.
- One release manifest binds the snapshot and sidecars to a catalog; selected artifact hashes are checked as they are loaded.
- `District` and `Campus` support dynamic metadata attributes through `meta`.
- `Campus.to_dict()` always includes `percent_enrollment_change` (numeric when available, otherwise `"N/A"`).

## Snapshot and Asset Behavior

`teadata` is intentionally cache-first.

Source definitions can include dataset-specific reference periods, publisher
details, URLs, citations, and provenance record identifiers. Use
`cfg.resolve_dataset(...)` to query configured metadata; use `ProvenanceStore`
to preserve immutable acquisition and processing history with archived file
bytes. Unknown historical download dates remain unknown. See
[dataset metadata and provenance](docs/config.md#dataset-metadata-and-provenance).

Artifacts typically used at runtime:

- `snapshot_<build-id>.pkl.gz` (engine snapshot)
- `boundaries_*.sqlite` (boundary WKB sidecar)
- `maps_<build-id>.sqlite` (map payload sidecar)
- `entities_*.sqlite` (entity lookup sidecar)

The authoritative `release.json` binds the snapshot and sidecars to one catalog and verified hashes. Hydrate tracked assets with Git LFS, or configure an immutable remote release URL. Metadata, provenance, masking, estimates, and rebuild instructions are documented in [Dataset contracts and data releases](docs/data-releases.md).

### Environment Variables

- `TEADATA_RELEASE`: local release manifest.
- `TEADATA_RELEASE_URL`: immutable remote release manifest.
- `TEADATA_DATA_DIR`: source data root for scripts and configured `data/...` inputs.
- `TEADATA_CONFIG`, `TEADATA_PROVENANCE_DIR`, `TEADATA_OUTPUT_DIR`: build locations.
- `TEADATA_DISABLE_INDEXES`: disable spatial acceleration indexes.
- `TEADATA_LOG_MEMORY`: log memory snapshots.

## Query DSL

`DataEngine` and `Query` chains use `>>`.

```python
# Resolve district then expand to district-operated campuses.
q = engine >> ("district", "ALDINE ISD") >> ("campuses_in",)

# Filter, sort, and take.
top = (
    q
    >> ("filter", lambda c: (c.enrollment or 0) > 1000)
    >> ("sort", lambda c: c.enrollment or 0, True)
    >> ("take", 10)
)

rows = top.to_df(columns=["name", "campus_number", "enrollment"])
```

Supported lookup semantics include:

- case-insensitive district and campus name matching
- wildcard patterns (`*`, `?`, SQL-like `%`/`_`)
- normalized district number handling (for example `"123"` and `"'000123"`)

Spatial and transfer helpers include:

- nearest-campus/nearest-charter queries
- `nearest_charter_same_type(...)`
- transfer graph methods such as `transfers_out(...)` / `transfers_in(...)`

## Enrichment Pipeline

`teadata/enrichment` provides registered enrichers for district and campus datasets.

Included enrichers cover:

- district accountability and district TAPR profile data
- campus accountability, TAPR profile/historical enrollment, PEIMS financials
- planned closure overlays
- charter network augmentation

Required source, schema, methodology, and transfer-join failures stop publication.
The previous release remains active when a staged build fails. Geometry coverage
and pending source review remain explicit in the processing receipt.

## Data Build Pipeline

`teadata/load_data.py` builds a full `DataEngine` and updates cached artifacts.

```bash
uv run python -m teadata.load_data
```

At a high level, it:

1. resolves year-aware source paths from `teadata/teadata_sources.yaml`
2. verifies and records selected source files and schema contracts
3. builds districts/campuses from verified inputs
4. applies enrichment datasets
5. validates a staged snapshot and sidecars, then activates `teadata/.cache/release.json`

## Config and CLI (`teadata-config`)

`teadata/teadata_config.py` provides YAML/TOML config loading, year resolution, schema checks, and dataset joins.

CLI entrypoint:

```bash
uv run teadata-config --help
```

Subcommands:

- `init <out.yaml>`
- `resolve <cfg> <section> <dataset> <year>`
- `report <cfg> [--json] [--min N] [--max N]`
- `join <cfg> <year> [--datasets a,b,c] [--parquet out.parquet] [--duckdb out.duckdb --table t]`

## Testing

```bash
scripts/check
scripts/check --verify-latest
```

Local checks are the primary verification path. They run Ruff, ty, deptry, and
pytest in a locked, isolated uv environment without replacing the development
`.venv`. Logs and immutable per-run JSON receipts stay in ignored `.local/checks/`.
Verification rejects receipts after the Git revision, source, lockfile, check
policy, tool/dependency environment, or captured logs change. A commit requires
a fresh run even when it contains previously checked working-tree contents.

After rebuilding changed serialized data, also verify the retained release:

```bash
scripts/check --release teadata/.cache/release.json
scripts/check --release teadata/.cache/release.json --verify-latest
TEADATA_CHECK_PYTHON=3.11 scripts/check
```

Code-only receipts do not certify raw inputs, data quality, or data artifacts.
Release checks verify all selected artifact hashes and bindings and reject a
manifest built from different processing code or lockfile. Use the same Python
selection when verifying an earlier run. Checks make no implicit data refresh.

An optional `scripts/check --install-hook` installs a pre-push code-check guard.
It verifies a current passing receipt or runs checks and then preserves Git LFS
pre-push processing. The guard requires a clean checkout so an uncommitted fix
cannot qualify a different pushed revision. Pushed branches/tags must point to
the checked HEAD; check out other revisions before pushing them. Custom hooks or `core.hooksPath` are left untouched; compose
the check command with those hooks manually. Check subprocesses cannot consume
the Git LFS ref updates from the hook's stdin. Inherited pytest control variables
are removed so shell settings cannot silently reduce the required suite.

Hosted checks remain available through manual GitHub Actions dispatch; pushes,
pull requests, and schedules do not consume hosted minutes automatically. A
passing local receipt is command evidence, not a signed attestation or an
independent review approval. Adversarial review of the final diff remains
required before landing.

## PyPI Size Limits and Current Packaging Status

PyPI defaults currently documented at:

- per-file upload limit: `100 MB`
- total project limit: `10 GB`

Reference: <https://docs.pypi.org/project-management/storage-limits/>

Before packaging trim, `0.0.118` artifacts were above the per-file limit:

- wheel: about `448 MB`
- sdist: about `446 MB`

The Python distribution includes code, source contracts, and local methodology documents. Data artifacts are distributed separately and selected through one release manifest. Source checkouts hydrate Git LFS; HTTP deployments use a complete adjacent bundle as described in [data releases](docs/data-releases.md#runtime-and-rollout). Installed wheels require `TEADATA_RELEASE` or `TEADATA_RELEASE_URL`. Publishing code does not refresh production data.

## Release Policy

- Versioning uses thousandths-place tags (`v0.0.101`, `v0.0.102`, ...).
- Preserve published tags, releases, and assets for reproducibility; see the release policy in `AGENTS.md`.

## License

Apache License 2.0. See [LICENSE](LICENSE).
