Metadata-Version: 2.5
Name: bq-v2-field-paths
Version: 0.3.0
Summary: DataHub transformer: align BigQuery fine-grained lineage with v2 schema field paths
Project-URL: Homepage, https://github.com/gutro/dv-datahub/tree/master/bq-v2-field-paths
Project-URL: Repository, https://github.com/gutro/dv-datahub
Project-URL: Issues, https://github.com/gutro/dv-datahub/issues
Author: DataVantage Platform
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: bigquery,column-level-lineage,datahub,lineage
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Database
Requires-Python: >=3.9
Description-Content-Type: text/markdown

# bq-v2-field-paths

A DataHub ingestion transformer that makes column-level lineage resolve through nested
BigQuery RECORD columns.

## The problem

DataHub's BigQuery source spells a nested column two different ways:

- its **schema generator** writes v2 paths into `schemaMetadata` —
  `[version=2.0].[type=struct].[type=struct].payload.[type=long].amount`
- its **SQL parser** names the same column `payload.amount`, and that name goes straight
  into the `schemaField` URN of every fine-grained lineage edge

Because DataHub resolves a `schemaField` URN by exact string match against the entity's
current `schemaMetadata`, every column edge touching a RECORD column points at a field
path that does not exist, and the UI silently draws nothing. Flat scalar columns render
only because their two spellings coincide.

The mapping is not in doubt — the SQL parser derives it correctly, even through
expressions. Only the spelling is wrong. This transformer rewrites it.

For a table whose real columns all live inside one RECORD — anything landed by Debezium
CDC, for instance — that is the difference between no column lineage and complete column
lineage.

## Usage

```yaml
transformers:
  - type: "bq_v2_field_paths"
    config: {}
```

`transformers` is a top-level key in the recipe, a sibling of `source`.

Anything it cannot map with certainty is left untouched: a dataset whose schema it cannot
find, or two v2 paths that reduce to the same simplified name. It logs a one-line summary
at end of stream saying how many URNs it rewrote, how many were already valid, and how
many it skipped.

## Config

| key | default | meaning |
|---|---|---|
| `platform` | `bigquery` | only rewrite URNs on this data platform |
| `use_graph_fallback` | `true` | look schemas up in DataHub when not seen in the stream |
| `expand_struct_parents` | `true` | **creates** lineage: see below |
| `max_unmapped_logged` | `20` | cap on distinct unmapped columns named in the summary |

### `expand_struct_parents`

Every other setting only ever *rewrites* an edge the SQL parser already derived. This
one is the exception: it **emits new edges**, and it is on by default.

Where a query projects a struct wholesale, the parser emits `after_row <- after_row` and
nothing for the fields inside it. Rewriting that edge makes it resolve, but clicking a
child column still shows nothing. So for such an edge, one edge is emitted per child
whose identically-named counterpart exists on *both* sides' `schemaMetadata` — a fact
read off two schemas, never a guess. Nesting is followed all the way down, so
`after_row.addr.city` is expanded, not just `after_row.addr`.

Set it to `false` for parent-level edges only, which for a struct-shaped table amounts
to no usable column lineage.

## Requirements

Needs an executor on **acryl-datahub 1.6 or newer**, which is where `pydantic>=2.4` is
pinned. `dependencies` is deliberately empty so that installing this package can never
resolve or upgrade the executor's own `acryl-datahub` — the trade is that the floor is
documented here rather than enforced by pip.
