Metadata-Version: 2.4
Name: datahub-describer
Version: 0.1.1
Summary: Hierarchical LLM-generated descriptions for DataHub
License-Expression: MIT
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: acryl-datahub[datahub-rest]>=1.7.0
Requires-Dist: openai>=1.40.0
Provides-Extra: dev
Requires-Dist: pytest; extra == "dev"
Dynamic: license-file

# DataHub Describer

Generate missing DataHub descriptions with an OpenAI-compatible LLM.

The connector works from fields upward:

```text
fields → datasets → parent containers
```

Field prompts use field metadata. Parent prompts use the names and descriptions
of their immediate children. Existing descriptions are preserved by default.

## Install

```bash
pip install datahub-describer
```

## Configure

Complete recipe with all available options:

```yaml
source:
  type: datahub-describer
  config:
    llm_model: "${LLM_MODEL}"
    llm_base_url: "${LLM_BASE_URL}" # Any OpenAI-compatible endpoint.
    llm_api_key: "${LLM_API_KEY}" # Use a DataHub secret.
    llm_timeout_seconds: 60
    llm_max_retries: 0

    field_prompt: |
      You document data catalog fields.
      Return JSON only:
      {
        "fields": [
          {"name": "<exact field name>", "description": "<description>"}
        ]
      }
      Rules:
      - Describe only the requested fields.
      - Use the field name, datatype, and surrounding schema.
      - Do not invent business rules.
      - Write 300-500 characters for each description.
      - Use clear factual language suitable for semantic search.
      - Do not add an AI-generated label.

    parent_prompt: |
      You document a data catalog parent from its children.
      Return JSON only:
      {"description": "<description>"}
      Rules:
      - Base the description only on the supplied child names and descriptions.
      - Do not invent business rules.
      - Write 300-500 characters.
      - Use clear factual language suitable for semantic search.
      - Do not add an AI-generated label.

    mode: generate # generate or delete_generated

    # Optional scope. The deepest configured level and its children are processed.
    platform: postgres
    container_name: sales
    # dataset_name: customers
    # field_name: email # Requires dataset_name.

    max_datasets: 1000
    max_containers: 5000
    max_fields_per_call: 20
    max_children_per_parent: 100
    document_containers: true
    rewrite_existing_descriptions: false

sink:
  type: datahub-rest
  config:
    server: "${DATAHUB_GMS_URL:-http://localhost:8080}"
```

## Development

```bash
pip install -e '.[dev]'
pytest
```
