Metadata-Version: 2.4
Name: datahub-ai-classifier
Version: 0.1.1
Summary: Customer-taxonomy LLM classification and tagging for DataHub
License-Expression: MIT
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: acryl-datahub[datahub-rest]>=1.7.0
Requires-Dist: openai>=1.40.0
Provides-Extra: dev
Requires-Dist: pytest; extra == "dev"
Dynamic: license-file

# DataHub Classifier

Classify DataHub fields against a customer-provided tag list using an
OpenAI-compatible LLM, then apply those tags.

Both plugins must be in the same recipe, with the same `taxonomy` map.
Classifier alone leaves glossary terms and does not create tags.
Missing tags are created. Existing tags are not overwritten.

## Install

```bash
pip install datahub-ai-classifier
```

## Configure

Complete recipe with all available options:

```yaml
source:
  type: postgres
  config:
    host_port: "<host>:<port>"
    database: "<database>"
    username: "<user>"
    password: "<password>"
    include_tables: true
    include_views: true
    profiling:
      enabled: false
    classification:
      enabled: true
      sample_size: 10 # How many values DataHub reads from each column.
      max_workers: 1 # Required. DataHub's default uses extra processes and this classifier breaks.

      classifiers:
        - type: datahub_classifier.classifier:AIPIIClassifier
          config:
            llm_model: gpt-4.1-mini # Provider model id.
            llm_base_url: https://api.openai.com/v1 # Any OpenAI-compatible endpoint.
            llm_api_key: "<your-api-key>" # DataHub secret; never hardcode.
            llm_timeout_seconds: 60
            llm_max_retries: 0
            confidence_threshold: 0.6 # Ignore weaker matches.
            taxonomy: &taxonomy
              C1: >
                Restricted identifiers. National IDs, tax numbers, and SSN
                values such as 123-45-6789, often in columns named ssn.
              C2: >
                Confidential personal and payment data. Names, emails, phone
                numbers, home addresses, dates of birth, and money amounts
                such as order_total.
              C3: >
                Internal operational data. Surrogate ids, loyalty tiers,
                currency codes, order status, and timestamps such as
                created_at and shipped_at.

transformers:
  - type: datahub-classifier
    config:
      taxonomy: *taxonomy # Must match the classifier map.

sink:
  type: datahub-rest
  config:
    server: "<datahub-gms-url>"
```

## Development

```bash
pip install -e '.[dev]'
pytest
```
