Metadata-Version: 2.5
Name: daft-hive
Version: 1.0
Summary: Community-maintained HiveServer2 datasource for Daft
Project-URL: Homepage, https://github.com/jiangxt2/daft-hive
Project-URL: Issues, https://github.com/jiangxt2/daft-hive/issues
Project-URL: Documentation, https://jiangxt2.github.io/daft-hive/
Author: jiangxt2
License-Expression: Apache-2.0
License-File: LICENSE
License-File: NOTICE
Classifier: Development Status :: 3 - Alpha
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Typing :: Typed
Requires-Python: <3.14,>=3.12
Requires-Dist: daft<0.7.24,>=0.7.23
Requires-Dist: pyarrow==19.0.1
Requires-Dist: thrift<0.23,>=0.22
Provides-Extra: all
Requires-Dist: daft[ray]<0.7.24,>=0.7.23; extra == 'all'
Requires-Dist: gssapi<2,>=1.9; extra == 'all'
Requires-Dist: pure-sasl<0.7,>=0.6.2; extra == 'all'
Requires-Dist: thrift-sasl<0.5,>=0.4.3; extra == 'all'
Provides-Extra: http
Provides-Extra: kerberos
Requires-Dist: gssapi<2,>=1.9; extra == 'kerberos'
Requires-Dist: pure-sasl<0.7,>=0.6.2; extra == 'kerberos'
Requires-Dist: thrift-sasl<0.5,>=0.4.3; extra == 'kerberos'
Provides-Extra: ray
Requires-Dist: daft[ray]<0.7.24,>=0.7.23; extra == 'ray'
Provides-Extra: test
Requires-Dist: jsonschema<5,>=4.23; extra == 'test'
Requires-Dist: pytest-asyncio<1,>=0.25; extra == 'test'
Requires-Dist: pytest-cov<7,>=6; extra == 'test'
Requires-Dist: pytest-timeout<3,>=2.3; extra == 'test'
Requires-Dist: pytest<9,>=8.3; extra == 'test'
Description-Content-Type: text/markdown

# daft-hive

> **Non-negotiable requirement:** daft-hive reads business data directly from HiveServer2 query
> results. The client, Daft driver, and Daft workers must not access Hive Metastore, HDFS, table
> `LOCATION`, object storage, ORC/Parquet files, Iceberg manifests, or storage credentials.

`daft-hive` is an independent, community-maintained Daft `DataSource` for HiveServer2. Hive—not
the connector—interprets internal and external tables, Text/ORC/Parquet storage, transactional
tables, views, SerDes, and Iceberg tables. Results cross the HiveServer2 TCLIService protocol and
are decoded into Arrow batches before entering Daft.

## Status

The initial compatibility baseline is deliberately **Experimental**:

- Daft `0.7.23`, PyArrow `19.0.1`, Python 3.12/3.13;
- native Hive 4.2.0 TCLIService bindings and V6 columnar decoding;
- strict single-query reads by default;
- no production-complete claim until the Daft 0.7.23 Python DataSource bridge passes the
  bounded-backpressure and abandoned-generator close contracts.

Production Core status requires a released, tested Daft bridge with bounded backpressure and
receiver-drop close propagation. daft-hive does not monkey patch Daft to bypass that gate.

## Installation

```bash
pip install daft-hive
```

Install optional capabilities explicitly:

```bash
pip install "daft-hive[kerberos]"
pip install "daft-hive[ray]"
```

## Example

```python
import daft
from daft_hive import (
    HiveConnectionOptions,
    HiveReadOptions,
    HiveTableIdentifier,
    SecretRef,
    read_hive,
)

events = read_hive(
    connection=HiveConnectionOptions(
        host="hiveserver2.example.net",
        port=10000,
        database="analytics",
        user="reader",
        auth="LDAP",
        credentials=SecretRef.env(
            "HIVE_PASSWORD",
            audience="hiveserver2://hiveserver2.example.net:10000",
        ),
        tls=True,
    ),
    read=HiveReadOptions(
        table=HiveTableIdentifier(database="analytics", table="events"),
        columns=("event_id", "event_time", "score"),
    ),
    filter=daft.col("score") >= 80,
)
```

Constructing the DataSource performs HS2-only schema discovery (`GetColumns` for a table or a
schema-only query for trusted SQL). Reading remains lazy: the data operation starts when Daft
executes the DataFrame. The credential reference is resolved just in time on the driver for schema
planning and again on the execution worker for the data operation; resolved secret values are
never serialized in the DataSource or task spec.

## Semantics

- `consistency="strict"` emits one Daft task and submits one Hive query. All fetches belong to the
  same Hive operation and snapshot.
- `independent_queries` is experimental and requires explicit typed split predicates. It rejects
  global limits, including Daft `DataFrame.limit()` pushdown. Each task observes an independent Hive
  snapshot; coverage, overlap, and duplicates remain the caller's responsibility unless the
  finite-domain predicate rules can prove them.
- Predicate pushdown is all-or-nothing. Unsupported expressions remain in Daft.
- A limit is sent to Hive only when no residual filter can change which rows satisfy it.
- Count/aggregation pushdown is disabled for Daft 0.7.23 because the custom DataSource API has no
  stable public opt-in contract for it.
- HiveServer2 provides a single result stream per operation. daft-hive does not advertise file,
  stripe, row-group, bucket, or manifest parallelism.

## Supported storage and table layouts

The connector has no storage-format reader. If HiveServer2 can query a Text, ORC, Parquet, ACID,
view, SerDe, or Iceberg table and return its result through TCLIService, daft-hive processes that
result identically. Support is claimed only for combinations passing the published real-Hive
conformance matrix.

## Development

```bash
uv sync --all-extras --group dev
uv run ruff format --check .
uv run ruff check .
uv run mypy
uv run python scripts/check_license_headers.py
uv run python scripts/check_spec.py
uv run mkdocs build --strict
DAFT_RUNNER=native uv run pytest tests/unit tests/contract
./scripts/run_hive_it.sh
uv build
uv run twine check dist/*
```

See the [architecture](docs/architecture.md), [compatibility matrix](docs/compatibility.md), and
[conformance rules](docs/conformance.md) before changing protocol or runtime behavior.

## License

Apache License 2.0. Generated TCLIService bindings preserve Apache Hive provenance in
`src/daft_hive/_hs2/generated/PROVENANCE.md`.
