Metadata-Version: 2.4
Name: mini-atlas-graph-warehouse
Version: 0.2.0
Summary: DuckDB-backed merge/upsert warehouse for GraphIngestor parquet exports
License-Expression: Apache-2.0
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Database
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: duckdb>=1.1
Requires-Dist: numpy>=1.26
Requires-Dist: pyarrow>=14
Requires-Dist: pyfiglet>=1.0
Requires-Dist: tqdm>=4.66
Provides-Extra: dev
Requires-Dist: pytest>=8; extra == "dev"
Provides-Extra: igraph
Requires-Dist: igraph<2,>=1.0; extra == "igraph"
Requires-Dist: numpy>=1.26; extra == "igraph"
Dynamic: license-file

# GraphWarehouse

Local DuckDB warehouse for entity–relationship graphs: ingest GraphIngestor Parquet, query bounded subgraphs, join stored node and edge measurements, convert selected graphs to iGraph, write results back, export to Gephi, and prepare toolbox input CSVs.

It is a general-purpose storage, query, analytics, iGraph, and export library. Typical work includes neighborhood and ego analysis, shared-resource structures, temporal slices, type-scoped score policies, constrained traversal, and Gephi publication.

Analytics results are named node or edge measurements produced **outside** GraphWarehouse. They may be anomaly scores, centrality, classifications, communities, model outputs, business metrics, or other user-defined signals. GraphWarehouse stores, discovers, filters, joins, traverses, exports, and writes those results back. It does not run external scoring algorithms.

**Safety:** keep work bounded. Seeds, type allowlists, hop counts, and windows limit the **result**. They do not make warehouse scans free. Full-warehouse iGraph conversion is in-memory and not out-of-core. Do not convert an unfiltered 50–100M-edge warehouse under a typical analyst process. Windows and 50–100M calibration remain environmental validation, not certified results.

## Capabilities

| Area | What you can do |
| --- | --- |
| Ingest | Merge GraphIngestor Parquet runs into one `.duckdb` with surrogate `node_id` / `edge_id` and null-aware property merge |
| Query | Type filters, bipartite views, ego N-hop, temporal windows, CSV / Arrow / Gephi / toolbox CSVs |
| Analytics | Load, list, summarize, and delete named runs; seed, filter, and constrain traversal by stored scores |
| iGraph | Convert a **bounded** `GraphSlice` (optional extra); write vectors back; export the in-memory graph to Gephi |

Analysts: start with the [cheat sheet](docs/ANALYST_CHEAT_SHEET.md), then the [workflow guide](docs/ANALYST_WORKFLOWS.md). Building a Parquet graph from SQL is GraphETL (a separate package); GraphQuery and GraphSlice stay in this package and are not part of GraphETL. Jupyter recipes: [graph query cheat sheet](docs/GRAPH_QUERY_CHEAT_SHEET.md). Technical recipes: [graph query cookbook](docs/GRAPH_QUERY_COOKBOOK.md). Lookup: [API reference](docs/API_REFERENCE.md). Guides: [analytics](docs/ANALYTICS_QUERY.md), [iGraph](docs/IGRAPH_INTEGRATION.md). Known unfinished CLI/seed behavior: [KNOWN_NOT_FROZEN.md](docs/KNOWN_NOT_FROZEN.md).

## Install

```bash
pip install mini-atlas-graph-warehouse
```

Optional in-memory graphs (`igraph>=1.0,<2` and `numpy>=1.26`):

```bash
pip install 'mini-atlas-graph-warehouse[igraph]'
```

Contributor/editable installation from a clone is a secondary development path:

```bash
python -m venv .venv
source .venv/bin/activate
python -m pip install -e /path/to/mini-atlas-graph-warehouse
python -m pip install -e "/path/to/mini-atlas-graph-warehouse[dev]"
python -m pip install -e "/path/to/mini-atlas-graph-warehouse[igraph]"
```

From the repository root, the development extra is:

```bash
python -m pip install -e ".[dev]"
```

The `igraph` extra is not a core dependency and is not in package `__all__`.

## Example

```python
from graph_warehouse import GraphQuery, GraphWarehouse

with GraphWarehouse("graph.duckdb") as wh:
    wh.ingest_runs(["data/run_a", "data/run_b"])
    print(wh.counts())

with GraphQuery("graph.duckdb") as gq:  # read-only; does not create a missing file
    with gq.ego(uids=["user:1"], hops=2).build() as slice_:
        print(slice_.counts())
        slice_.to_csv("out/ego")
```

`GraphWarehouse` owns ingest and analytics writes. `GraphQuery` is read-only by default. Finish and close writers before opening the same file with a differently configured connection. Prefer `with` for both the query session and each slice so temporary relations are dropped.

## Ingest

GraphIngestor `export_parquet` (`local-graph-ingestor-parquet` version `1` or `2`):

```text
data/run_a/metadata.json
data/run_a/nodes/*.parquet    # uid, type, properties, event_ts
data/run_a/edges/*.parquet    # v1: from_id, to_id, type, properties, event_ts
                              # v2: uid, from_id, to_id, type, properties, event_ts
graph.duckdb
```

Parquet columns stay string identities. The warehouse assigns `node_id` / `edge_id`. File stem is used as `type` when a parquet file has no `type` column. `properties` may be STRUCT, MAP, JSON, or JSON-encoded VARCHAR.

| `metadata.json` | Behavior |
| --- | --- |
| Missing, or object with no / empty `format` | Legacy glob of `nodes/*.parquet` and `edges/*.parquet` |
| `format` `local-graph-ingestor-parquet`, integer `version` `1` | Strict v1 adapter; **manifest is authoritative**. Each edge uid is the derived `rel:` id |
| That format, integer `version` `2` | Strict v2 adapter; **manifest is authoritative**. Requires `edges.uid` |
| That format with any other version, or any other nonempty `format` | Fail the run |
| Present but not a JSON object | Fail the run |

v1 and v2 paths must be relative POSIX `nodes/<file>.parquet` or `edges/<file>.parquet`. Unlisted parquet files are skipped (`WARNING skip stray parquet`, `stray_files` in stats). Jobs run in sorted path order. Valid v1 input is assumed unique on `(uid, type)` and `(from_id, type, to_id)` inside the run. Valid v2 input is assumed unique on edge `uid`. A v2 `uid` equal to the derived `rel:` relationship id is accepted. Any other uid that uses the `rel:` prefix is rejected. Replaying a uid updates that edge and keeps its `edge_id`. Replaying it with different endpoints or type raises `ValueError` and rolls the run back. Several uids may share one `(from_id, type, to_id)`. Collisions with **already stored** rows still use `NOT EXISTS` and unique constraints. Legacy input keeps defensive identity `GROUP BY`. `parquet_batch_size` controls Arrow read size only.

`nodes_upserted` and `edges_upserted` count **rows read from Parquet**, not rows newly inserted or changed in DuckDB. Duplicate identities in a file, logical upserts, and `skip_if_ingested=False` reingestion of the same identities still report the Parquet row counts. They are not a net-delta of warehouse rows.

A uid cannot be ingested under two conflicting concrete node types. Conflicts raise `ValueError` and roll back the complete ingest. `_untyped` endpoint placeholders may still upgrade to one concrete type. The same node uid with the same type remains idempotent. Edge identity is the edge `uid`, not the endpoint triple.

Each `ingest_run` is one transaction. Metadata, manifest, path, timestamp, or SQL failure rolls **that run** back. Prior successful runs stay.

Skip is allowed only when the stored `run_id` matches, the resolved export path is the same, and `graph_id` is compatible (equal when both sides have it; missing on both sides is compatible; present-vs-absent or two different values conflict). A successful skip is reported with `skipped: true` plus a reason. That skip is an identity match, not proof that file bytes are unchanged. Rewriting an export in place still skips unless you pass `skip_if_ingested=False` or CLI `--force`. Reusing a `run_id` for a different directory or conflicting `graph_id` raises `ValueError` and the CLI exits nonzero. `--force` / `skip_if_ingested=False` means upsert/reingest. It does not delete rows previously contributed by that run that are absent from the rewritten export; Warehouse does not have safe row-level run provenance for deletion. There is no content hash and no `export_id`.

Offset-aware ISO-8601 `event_ts` (including `Z`) is stored as naive UTC DuckDB `TIMESTAMP`. Incoming null does not wipe a stored timestamp. Naive or malformed timestamps fail the run. Inside one ingest, rows for one edge uid collapse with `max(event_ts)`. The stored value is then the maximum of that incoming timestamp and the timestamp already on the row, so a later run cannot move it earlier. GraphIngestor, before export, replaces `event_ts` with the last non-null value even when that value is earlier, and it applies `incremental_keys` again on every replay. Warehouse does not apply `date_keys` or `incremental_keys`; those already ran in the export. Property merge: later non-null wins; JSON null is not SQL NULL (`json_type(value) <> 'NULL'`).

New databases are created with `edges.uid` and `UNIQUE (uid)`. Opening an older database does not migrate it. v1 Parquet still loads on that old schema. v2 Parquet raises until `GraphWarehouse.migrate()`.

`migrate()` checkpoints and closes the database, copies it to `{db_path}.pre-migrate` (and `{db_path}.pre-migrate.wal` when a WAL is still present), assigns each stored edge `relationship_uid(from_id, type, to_id)`, and keeps `edge_id`, properties, timestamps, and analytics joins. Calling it again after success does nothing and does not rewrite the backup. If migration fails, the saved files are put back, the open database is checked against the pre-migration fingerprint, and the original error is raised. The backup is kept. `migrate()` will not overwrite an existing `.pre-migrate` file. After a restored failure, confirm the open database matches the pre-migration data, move that backup aside, then call `migrate()` again. If the restore cannot be verified, `EdgeUidMigrationRecoveryError` is raised; leave the backup in place and recover from it by hand.

After the backup copy succeeds, `migrate()` writes `{db_path}.pre-migrate.incomplete`. That interrupt file is removed when migration finishes and when an exception inside `migrate()` restores the saved database. Opening the warehouse does not remove it, does not mark a rebuilt database complete, and does not replace the live file. A killed process skips the exception-time restore, so the interrupt file and the backup stay. `ingest_run` and `migrate()` then raise before they write. That includes a live `edges` table that still has the old triple key and one that already has `uid`. The error names the database state, the `{db_path}.pre-migrate` path, and `{db_path}.pre-migrate.wal` when that WAL file is present, and it says the database must be recovered before ingest or migration can proceed. A healthy old database without the interrupt file still accepts v1 Parquet. A completed migration and a new uid database still accept v1 and v2. A crash after the completion marker is written is a finished migration; a later `migrate()` removes a leftover interrupt file and leaves the backup alone.

Recover a killed migration by hand:

1. Stop every process that has the database open.
2. Move the live database to `{db_path}.interrupted`. When `{db_path}.wal` exists, move it to `{db_path}.interrupted.wal`. That live WAL has to leave the live path so DuckDB cannot replay the interrupted migration onto the restored file.
3. Copy `{db_path}.pre-migrate` to `{db_path}`. When `{db_path}.pre-migrate.wal` exists, copy it to `{db_path}.wal`. When the backup has no WAL, leave the live path without a WAL.
4. Open the restored database read-only. Confirm `edges` has no `uid` column, the unique key is `(from_id, type, to_id)`, `warehouse_meta` has no `edge_uid_migration` row, and `edges_uid_rebuild` is absent.
5. Remove `{db_path}.pre-migrate.incomplete` only after that check. Leave `{db_path}.pre-migrate` and `{db_path}.pre-migrate.wal` unchanged. Keep the `.interrupted` files until the check has succeeded.
6. v1 ingest can run again. To migrate, move the backup and its WAL to new names, then call `migrate()`.

If `{db_path}.pre-migrate` is missing, there is no backup to copy back. Leave the live files where they are. An exception raised inside `migrate()` still restores automatically and then removes the interrupt file. If that automatic restore cannot be verified, the backup and the interrupt file both stay.

Very old pre-surrogate databases still need a rebuild from parquet. Re-ingest with `--force` only upserts; it does not delete rows missing from the new export.

CLI: `python -m graph_warehouse.cli` or `graph-warehouse` after install.

```bash
python -m graph_warehouse.cli ingest --db graph.duckdb --run data/run_a --run data/run_b
python -m graph_warehouse.cli ingest --db graph.duckdb --data-dir data/
python -m graph_warehouse.cli counts --db graph.duckdb
```

Tables and merge details: [package README](graph_warehouse/README.md). Method signatures: [API reference](docs/API_REFERENCE.md).

## Query, analytics, and export

Builder, slice lifecycle, windows, seeds, and exports: [query README](graph_warehouse/query/README.md) and [API reference](docs/API_REFERENCE.md). CSV/Arrow/Gephi columns: [export README](graph_warehouse/query/export/README.md).

Callers select an **exact** analytics `run_id`. Do not use `runs[0]`. Analytics observation windows (`params["_provenance"]["window"]`) are not warehouse `edges.event_ts`. Final filters and traversal constraints produce different reachable graphs. Type-scoped edge analytics can require scores on listed types while other selected types pass through.

Analytics CSV headers with supported unusual quoting (embedded `'` / `"`) are escaped in generated SQL. A CSV or analytics-directory load that matches **zero** warehouse identities does not register an empty successful algorithm run. Partial matches still register and store the matching rows. Unmatched identities in CSV loads are skipped; iGraph writeback unmatched identities remain a hard error.

`configure_logging` configures the `graph_warehouse` package logger (not the root logger) and sets `propagate=False` so records do not also go to ancestor loggers. Library code does not configure logging by itself.

```bash
python -m graph_warehouse.cli query --db graph.duckdb --out out/ego \
  --seed-uid user:1 --hops 2 --format csv
python -m graph_warehouse.cli load-results --db graph.duckdb \
  --csv edge_results.csv --entity edge --run-id relationship_scores_2026w01 --algo external_anomaly_model
```

## Tests and diagnostics

```bash
pytest -q
python -m pip install -e ".[igraph]"   # required for tests/test_igraph_slice.py
pytest -m e2e
GRAPH_WAREHOUSE_REQUIRE_INGESTOR=1 pytest tests/test_ingestor_parquet_contract.py
GRAPH_WAREHOUSE_REQUIRE_INGESTOR=1 pytest tests/e2e/test_ingestor_boundary.py
```

GraphIngestor is not a core or development dependency. Ordinary pytest skips those contract tests when it is missing unless `GRAPH_WAREHOUSE_REQUIRE_INGESTOR=1`.

Opt-in (not CI, not Windows certification): [bench/README.md](bench/README.md), [scripts/analyst_windows_smoke.md](scripts/analyst_windows_smoke.md). Historical architecture notes are not the public entry point: `GRAPH_WAREHOUSE_REFACTOR_PLAN.md` and related plan files.
