Metadata-Version: 2.4
Name: datahub-describer
Version: 0.1.2
Summary: Hierarchical LLM-generated descriptions for DataHub
License-Expression: MIT
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: acryl-datahub[datahub-rest]>=1.7.0
Requires-Dist: openai>=1.40.0
Provides-Extra: dev
Requires-Dist: pytest; extra == "dev"
Dynamic: license-file

# DataHub Describer

Generate missing DataHub descriptions with an OpenAI-compatible LLM.

The connector works from fields upward:

```text
fields → datasets → parent containers
```

Field prompts use field metadata. Parent prompts use the names and descriptions
of their immediate children. Existing descriptions are preserved by default.

In the DataHub UI recipe, last step (Advanced), put `datahub-describer` in Extra
Pip Libraries so the UI executor can load this package.

## Install

```bash
pip install datahub-describer
```

## Configure

Complete recipe with all available options:

```yaml
source:
  type: datahub-describer
  config:
    llm_model: "${LLM_MODEL}"
    llm_base_url: "${LLM_BASE_URL}" # Any OpenAI-compatible endpoint.
    llm_api_key: "${LLM_API_KEY}" # Use a DataHub secret.
    llm_timeout_seconds: 60
    llm_max_retries: 0

    field_prompt: |
      You document data catalog fields.
      Return JSON only:
      {
        "fields": [
          {"name": "<exact field name>", "description": "<description>"}
        ]
      }
      Rules:
      - Describe only the requested fields.
      - Use the field name, datatype, and surrounding schema.
      - Do not invent business rules.
      - Write 300-500 characters for each description.

    parent_prompt: |
      You document a data catalog parent from its children.
      Return JSON only:
      {"description": "<description>"}
      Rules:
      - Base the description only on the supplied child names and descriptions.
      - Do not invent business rules.
      - Write 300-500 characters.

    mode: generate
    # delete_generated removes every description that contains [AI GENERATED].

    # Optional scope. The deepest configured level and its children are processed.
    platform: postgres
    container_name: sales
    # dataset_name: customers
    # field_name: email # Requires dataset_name.

    max_datasets: 1000
    max_containers: 5000
    max_fields_per_call: 20
    max_children_per_parent: 100
    document_containers: true
    rewrite_existing_descriptions: false
```

## Development

```bash
pip install -e '.[dev]'
pytest
```
