Metadata-Version: 2.5
Name: bq-v2-field-paths
Version: 0.2.0
Summary: DataHub transformer: align BigQuery fine-grained lineage with v2 schema field paths
Project-URL: Homepage, https://github.com/gutro/dv-datahub/tree/master/bq-v2-field-paths
Project-URL: Repository, https://github.com/gutro/dv-datahub
Project-URL: Issues, https://github.com/gutro/dv-datahub/issues
Author: DataVantage Platform
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: bigquery,column-level-lineage,datahub,lineage
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Database
Requires-Python: >=3.9
Description-Content-Type: text/markdown

# bq-v2-field-paths

A DataHub ingestion transformer that makes column-level lineage resolve through nested
BigQuery RECORD columns.

## The problem

DataHub's BigQuery source spells a nested column two different ways:

- its **schema generator** writes v2 paths into `schemaMetadata` —
  `[version=2.0].[type=struct].[type=struct].payload.[type=long].amount`
- its **SQL parser** names the same column `payload.amount`, and that name goes straight
  into the `schemaField` URN of every fine-grained lineage edge

Because DataHub resolves a `schemaField` URN by exact string match against the entity's
current `schemaMetadata`, every column edge touching a RECORD column points at a field
path that does not exist, and the UI silently draws nothing. Flat scalar columns render
only because their two spellings coincide.

The mapping is not in doubt — the SQL parser derives it correctly, even through
expressions. Only the spelling is wrong. This transformer rewrites it.

For a table whose real columns all live inside one RECORD — anything landed by Debezium
CDC, for instance — that is the difference between no column lineage and complete column
lineage.

## Usage

```yaml
transformers:
  - type: "bq_v2_field_paths"
    config: {}
```

`transformers` is a top-level key in the recipe, a sibling of `source`.

Anything it cannot map with certainty is left untouched: a dataset whose schema it cannot
find, or two v2 paths that reduce to the same simplified name. It logs a one-line summary
at end of stream saying how many URNs it rewrote, how many were already valid, and how
many it skipped.

## Config

| key | default | meaning |
|---|---|---|
| `platform` | `bigquery` | only rewrite URNs on this data platform |
| `use_graph_fallback` | `true` | look schemas up in DataHub when not seen in the stream |
| `max_unmapped_logged` | `20` | cap on distinct unmapped columns named in the summary |
