Metadata-Version: 2.4
Name: datasift-diff
Version: 0.8.0
Summary: Compare two datasets by key — files, remote URIs, git revisions, or databases.
Author: sarahsalary
License: MIT
Keywords: diff,data,csv,json,compare,git,s3,gcs,database
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: tui
Requires-Dist: textual>=0.60.0; extra == "tui"
Provides-Extra: datasift
Requires-Dist: datasift-py>=0.1.0; extra == "datasift"
Provides-Extra: excel
Requires-Dist: openpyxl>=3.1.0; extra == "excel"
Provides-Extra: parquet
Requires-Dist: pyarrow>=14.0.0; extra == "parquet"
Provides-Extra: yaml
Requires-Dist: PyYAML>=6.0; extra == "yaml"
Provides-Extra: toml
Requires-Dist: tomli>=2.0; python_version < "3.11" and extra == "toml"
Provides-Extra: xml
Requires-Dist: lxml>=5.0; extra == "xml"
Provides-Extra: s3
Requires-Dist: boto3>=1.34.0; extra == "s3"
Provides-Extra: gcs
Requires-Dist: google-cloud-storage>=2.14.0; extra == "gcs"
Provides-Extra: http
Provides-Extra: web
Requires-Dist: fastapi>=0.110.0; extra == "web"
Requires-Dist: uvicorn[standard]>=0.29.0; extra == "web"
Requires-Dist: python-multipart>=0.0.9; extra == "web"
Provides-Extra: postgres
Requires-Dist: psycopg[binary]>=3.1.0; extra == "postgres"
Provides-Extra: mysql
Requires-Dist: PyMySQL>=1.1.0; extra == "mysql"
Provides-Extra: mssql
Requires-Dist: pyodbc>=5.1.0; extra == "mssql"
Provides-Extra: db
Requires-Dist: datasift-diff[mssql,mysql,postgres]; extra == "db"
Provides-Extra: anomaly
Requires-Dist: numpy>=1.24.0; extra == "anomaly"
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"
Requires-Dist: pytest-cov>=4.0; extra == "dev"
Requires-Dist: pytest-asyncio>=0.21; extra == "dev"
Requires-Dist: ruff>=0.1; extra == "dev"
Requires-Dist: mypy>=1.0; extra == "dev"
Requires-Dist: build>=0.10; extra == "dev"
Provides-Extra: all
Requires-Dist: datasift-diff[anomaly,datasift,db,dev,excel,gcs,http,parquet,s3,toml,tui,web,xml,yaml]; extra == "all"
Dynamic: license-file

# datasift-diff

Compare two datasets by key — files, remote URIs, git revisions, or databases.

## Installation

```bash
pip install datasift-diff

# With all optional features
pip install datasift-diff[all]
```

## Quick Start

```bash
# Compare two local files
datasift-diff old.csv new.csv --key id

# Multi-key
datasift-diff old.csv new.csv --key user_id,date

# Git revisions
datasift-diff HEAD~1:data.csv HEAD:data.csv --key id

# Remote sources (S3/GCS)
datasift-diff s3://bucket/a.jsonl s3://bucket/b.jsonl --key id

# Databases
datasift-diff 'postgres://user:pass@host/db?table=users' \
              'postgres://user:pass@host/db?table=users_old' --key id

# HTML report
datasift-diff old.csv new.csv --key id --format html -o report.html

# Validate against a schema
datasift-diff old.csv new.csv --key id --validate --schema-file schema.yaml

# Anomaly detection
datasift-diff old.csv new.csv --key id \
    --anomaly-history history.json \
    --anomaly-threshold 3.5 \
    --fail-on-anomaly
```

## Features

- **Formats**: CSV, JSON, JSONL, YAML, TOML, XML, Excel, Parquet
- **Multi-key**: Compare by several fields
- **Nested keys**: `user.id`, `address.city`
- **Numeric tolerance**: `1.0000001` == `1.0000002`
- **Flatten**: Flatten nested dicts before comparison
- **Streaming**: Single-pass merge-join for sorted inputs
- **Remote streaming**: S3/GCS via HTTP Range requests
- **Git revisions**: `HEAD:data.csv`
- **Schema files**: YAML/TOML/JSON
- **Web UI**: `datasift-diff-serve`
- **TUI**: `datasift-diff-tui`
- **Databases**: SQLite, PostgreSQL, MySQL, MSSQL
- **Anomaly detection**: Z-score / IQR
- **Watch mode**: Re-run diff on file changes
- **Plugin system**: Add your own formats and sources

## Python API

```python
from datasift_diff import diff

result = diff("old.csv", "new.csv", key="id")
print(result.summary)
```

## Docs

- [Remote Streaming (S3/GCS)](docs/remote-streaming.md)
- [Git Revisions](docs/git.md)
- [Database Support](docs/database.md)
- [Anomaly Detection](docs/anomaly.md)
- [Web UI](docs/web-ui.md)

## License

MIT
