Metadata-Version: 2.5
Name: datahub-bq-connector-session-patch
Version: 0.1.1
Summary: DataHub BigQuery sidecar source: attribute reads whose SQL does not qualify its table names
Project-URL: Homepage, https://github.com/gutro/dv-datahub/tree/master/datahub-bq-connector-session-patch
Project-URL: Repository, https://github.com/gutro/dv-datahub
Project-URL: Issues, https://github.com/gutro/dv-datahub/issues
Author: DataVantage Platform
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: bigquery,datahub,ingestion,lineage,usage
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Database
Requires-Python: >=3.9
Provides-Extra: bigquery
Requires-Dist: acryl-datahub[bigquery]; extra == 'bigquery'
Provides-Extra: test
Requires-Dist: acryl-datahub[bigquery]<1.8,>=1.6; extra == 'test'
Requires-Dist: pytest>=7; extra == 'test'
Requires-Dist: pyyaml; extra == 'test'
Description-Content-Type: text/markdown

# datahub-bq-connector-session-patch

A DataHub BigQuery **sidecar source** that attributes reads whose SQL did not qualify
its table names — the ones the stock connector sees, misfiles and silently discards.

Full diagnosis, design, verification and limits:
`datavantage/docs/datahub-unqualified-read-attribution.md`.

## The problem

A client with a BigQuery *default dataset* set can write `SELECT ... FROM Bets_Details`.
BigQuery fills in the blank at run time, but never records it: `INFORMATION_SCHEMA.JOBS`
carries the SQL text and the billing project, not the job's default dataset.

DataHub's connector substitutes `_SESSION` for the missing dataset
(`queries_extractor.py:542,558`), on the assumption that a bare name is usually a
temporary table. Its own temp-table rule then flags anything whose dataset starts with
`_` (`:309`), and the aggregator drops it — for usage
(`sql_parsing_aggregator.py:1022`) and for the query entity (`:1693`).

So the connector *sees* the table name, guesses the wrong container, and discards the
result. Nothing is reported: the sibling branch of `is_temp_table` records what it drops,
this one returns `True` in silence. Measured here: one client, 2,866 jobs in 7 days,
100% unqualified, 0 usage rows.

**A `0` in DataHub usage is therefore ambiguous** — "nobody reads this", or "read
constantly, with unqualified SQL". Decommissioning decisions depend on telling those apart.

## What this does

For upstreams that landed in `_SESSION`, it resolves the bare table name against the
tables DataHub discovered during its own schema pass. A name matching more than one
discovered table is **left untouched** — dropped exactly as before, never guessed.

It ships as a distinct source type, so the production recipe stays stock:

```yaml
source:
  type: datahub_bq_connector_session_patch
```

## Installing

Per-recipe, via **Extra Pip Libraries** (DataHub UI) or `extra_pip_requirements`. That
field is scoped to one recipe, which is what keeps the production recipe unaffected.

```
datahub-bq-connector-session-patch[bigquery]
```

**The `[bigquery]` suffix is required**, or the source will not load. A DataHub executor
builds its per-recipe venv as `acryl-datahub[<source_type>]`; because
`datahub_bq_connector_session_patch` is not a real acryl-datahub extra, pip ignores the
unknown extra and installs **core** acryl-datahub. This package imports
`datahub.ingestion.source.bigquery_v2`, so the import fails and the registry reports:

```
datahub_bq_connector_session_patch is disabled due to a missing dependency: google;
try running `pip install 'acryl-datahub[datahub_bq_connector_session_patch]'`
```

That suggested remedy is a dead end — the extra it names does not exist. The equivalent
two-line form works identically if you prefer it explicit:

```
acryl-datahub[bigquery]
datahub-bq-connector-session-patch
```

Either way the extra is unpinned, so against an already-installed acryl-datahub it adds
only the extra's dependencies and cannot move the executor's version (verified against
1.7.0.9).

The wheel declares **no mandatory dependencies** — it imports only `datahub.*`, which any
environment running a recipe already has. Declaring `acryl-datahub` would let pip
re-resolve the executor's own pinned copy while installing this. The tested range
(`>=1.6,<1.8`) is enforced at runtime instead: an out-of-range executor raises a
structured warning into the ingestion report rather than failing the run.

## The sidecar recipe

```yaml
pipeline_name: bq-session-attribution        # its OWN name — never the main recipe's
source:
  type: datahub_bq_connector_session_patch
  config:
    project_id_pattern:
      allow:
        - '^dv-prod-eu-w1(-.+)?-data$'
        - '^dv-ext-prod-eu-w1(-.+)?-data$'
        - '^dv-prod-eu-w1(-.+)?-comp\d+$'
    include_tables: false          # lightweight discovery still fills table_refs
    include_views: false
    include_table_lineage: false   # refused if true — see below
    include_usage_statistics: true
    use_queries_v2: true
    include_queries: true
    include_query_usage_statistics: true
    region_qualifiers: ['region-europe-west1']
    start_time: '-7 days'
    enable_stateful_time_window: true
    stateful_ingestion:
      enabled: true                # the time-window watermark
      remove_stale_metadata: false # refused if true — see below
    env: PROD
```

### Two settings this source refuses to start with

Both were reproduced against a live catalog. They are not style preferences, so they are
enforced in code rather than left to the YAML:

| setting | why it is refused |
|---|---|
| `stateful_ingestion.remove_stale_metadata: true` | The sidecar writes a `status` aspect on every dataset it produces usage for, enrolling them in *its* stale-removal checkpoint. Its URN set is **usage-driven**, so any dataset not read that day falls out of the checkpoint and is soft-deleted — datasets the main recipe owns. Normal operation, not an edge case. |
| `include_table_lineage: true` | Resolved names drive usage attribution only. A mis-resolved write would create a **wrong edge into a real table**, which is worse than the missing usage this package exists to fix. |

`remove_stale_metadata` defaults to `true`, so it must be set explicitly. Only the
combination bites — with `stateful_ingestion.enabled: false` there is no checkpoint and
nothing is refused.

### Scope it to the readers, not to minimise ambiguity

`dataset_pattern` should be **wide**. "Ambiguous" means the name matches more than one
entry in `discovered_tables` *within the recipe's scope*, so narrowing scope removes the
alternatives rather than resolving between them. If a reader's default dataset is
`prod_spain_*` but the index holds only `prod_malta_*`, the name resolves **confidently
and wrongly**.

Ambiguity fails safe. A too-narrow index fails silently, in the wrong direction.

Widening is otherwise harmless: `disjoint_only` means the sidecar emits only URNs it
actually recovered, so it stays quiet wherever SQL is already qualified and never
double-books usage with the main recipe.

## Reading the run report

```
rewritten=41 tables (2792 refs) ambiguous=18 names (63 refs) unknown=7 names (12 refs)
queries_passed=954 queries_skipped=3118 (index=1631 unique names, 412 ambiguous names)
```

- **distinct counters** — how much of the catalog was recovered. `rewritten` counts
  distinct resolved **target tables**; `ambiguous` and `unknown` count distinct bare
  **names**. Deliberately not distinct `_SESSION` URNs: those embed the reader project,
  so one table read from five projects would report as five.
- **`*_refs`** — how many reads that attributed. One hot table read 2,000 times is one
  rewrite and 2,000 references.
- **`queries_skipped`** — queries whose SQL was already fully qualified. The main recipe
  has those; the sidecar correctly stays out of the way.
- **`queries_mixed`** — queries carrying *both* an already-resolvable reference and a
  recovered one. Also left to the main recipe, for a subtler reason, below.

### Why mixed queries are skipped

`disjoint_only` keeps the two pipelines from double-booking **usage**, which is a
timeseries aspect. It does not, by itself, protect the **query entity**.

Query entities are keyed by `get_query_fingerprint(sql, platform, fast=True)` — a pure
function of the SQL text, so both pipelines derive the *same* `urn:li:query:<fp>`. But
`querySubjects` is a versioned aspect, not timeseries. If the sidecar emitted its
narrowed subject set for a query the main recipe also sees, whichever ran last would
win, and the entity would flip between the two subject sets on every scheduled run.

Passing the union instead would fix the subjects and double-book usage for the
already-resolvable half. Neither is acceptable, so a mixed query is left entirely to the
main recipe and counted. Measured on `dv-ext-prod-eu-w1-data` over 7 days:
`queries_mixed=0` — the guard costs nothing there, because that client's SQL is
uniformly unqualified.

A per-reader-project breakdown is logged alongside it, so a project that recovers nothing
shows up at 0.0% rather than vanishing.

### Two failures that look like success

Both were hit during development. Each leaves the run green and the output plausible:

1. **`discovered_tables` is empty at construction.** The source passes a *live reference*
   to `bq_schema_extractor.table_refs`, which only fills during the schema pass. The
   index is built lazily on first use, never in `__init__`.
2. **Refs are `projects/P/datasets/D/tables/T`, not `P.D.T`.** Naive dotted parsing
   matches nothing.

Because of these the source raises a structured **warning** if the index came out empty
or nothing was rewritten. **Never trust a negative result without checking the index
size.**

## Limits

Unique-name resolution only. Where a bare name matches several catalogued objects it is
dropped. How much that costs depends entirely on the estate:

| population | ambiguous names |
|---|---|
| views in `dv-ext-prod-eu-w1-data` | **0.0%** (162/162 unique) |
| views in `dv-prod-eu-w1-data` | **54.4%** |
| tables referenced estate-wide | 53.0% |

The external-facing views carry licence/brand suffixes and are unique by construction.
The internal estate's per-market layout duplicates names across `prod_malta_*`,
`prod_spain_*`, `prod_italy_*`…, and the cross-market `common_` layer duplicates a
further 39 names against them — so roughly half of any bare reads against it stay
unrecovered.

Resolving those needs **leaf-set matching**, not name matching: the colliding
common-vs-market views have identical names but cleanly distinct leaf sets. That is
explicitly out of scope — see §9.1 of the design doc for what it would take.

## Measuring a scope before shipping it

```bash
export BQ_SESSION_PATCH_BILLING_PROJECT=dv-ext-prod-eu-w1-data
python tools/measure_scope.py eu baseline --days 7
python tools/measure_scope.py eu patched  --days 7
python tools/diff_runs.py runs/mcps_eu_baseline_7d.json runs/mcps_eu_patched_7d.json
```

Scope flags: `--projects` for explicit ids, `--project-pattern` to override the region's
`project_id_pattern.allow` regexes, `--days` for the window.

**The billing project must be explicit** — `--billing`, or
`$BQ_SESSION_PATCH_BILLING_PROJECT`, or inferred when `--projects` names exactly one.
DataHub passes it straight to `bigquery.Client(...)`, and when it is `None` the client
inherits whatever `google.auth.default()` resolves: `GOOGLE_CLOUD_PROJECT` first, then the
ADC file's quota project, then the active gcloud config. A stray env var will therefore
bill every `INFORMATION_SCHEMA` job to an unrelated project and fail as
`bigquery.jobs.create` denied against a project you never named. The script refuses to
start rather than inherit one, and prints both the billing and the ambient project so a
mismatch is visible.

`measure_scope.py` writes MCPs to a file sink (nothing reaches GMS) and prints the
per-project recovery table; `diff_runs.py` reports datasets gaining usage, query entities
gained and readers newly attributed, and fails if any `schemaMetadata`,
`upstreamLineage` or `datasetProperties` aspect leaked into the output.

## Trying it against a local DataHub

`recipes/local/` holds a two-step pair for a quickstart instance, scoped to one project:

```bash
make venv && source .venv/bin/activate
datahub ingest -c recipes/local/01-catalog.yml       # stock `bigquery` — the catalog
datahub ingest -c recipes/local/02-usage-sidecar.yml # this package — the recovered usage
```

Step 1 runs with usage **on**, exactly as production does, and still misses the
unqualified reads — that is the premise. Step 2 shows what it missed. Measured against
`dv-ext-prod-eu-w1-data` over 7 days:

| | step 1 (catalog) | step 2 (sidecar) |
|---|---|---|
| datasets with a usage aspect | 5 | **+59** |
| query entities | 170 | **+61** |
| aspects emitted | schema, lineage, properties, usage, queries | usage and queries only |

Both use ADC, so the VPN must be up — VPC-SC blocks ADC while the `bq` CLI keeps
working, which makes the failure look like a permissions problem.

They run **stateless** (`stateful_ingestion.enabled: false`), so re-running re-emits the
same days. Timeseries aspects append rather than upsert, so repeated local runs
accumulate duplicate usage documents for a day. Harmless locally; in a real deployment
keep the stateful time window, which is what makes each day emit exactly once.

## Developing

```bash
make venv      # .venv on python 3.11, package installed editable + test extras
make test      # 49 tests against a real acryl-datahub
make gate      # the acceptance gate: must FAIL on stock, PASS patched
make release   # test, gate, bump, rebuild, verify  -> dist/
make publish
```

`make venv` exists for interactive work and for the `tools/` scripts, which need real
BigQuery credentials and so cannot run in a throwaway environment. `make test` and
`make gate` deliberately **do not** use it — they build their own environment per run, so
a stale or hand-modified `.venv` can never make them pass.

`make gate` is the one that matters on an acryl-datahub bump. This package patches
private internals, so `make test` alone would stay green even if upstream fixed the
defect out from under it — at which point this package is dead weight and should be
retired, not shipped. `gate` fails loudly in that case.

## Upstream

The defect is unfixed on `datahub-project/datahub` master as of 2026-09-14. The ask
there is modest: report the `_SESSION` discard, or offer opt-in resolution against
discovered tables. See `UPSTREAM-ISSUE.md`.
