Metadata-Version: 2.4
Name: data-connect-hub
Version: 0.2.0
Summary: Python SDK for the Data Connect Hub service
License-Expression: Apache-2.0
Project-URL: Homepage, https://github.com/opendatahub-io/data-connect-hub
Project-URL: Repository, https://github.com/opendatahub-io/data-connect-hub
Project-URL: Documentation, https://opendatahub-io.github.io/data-connect-hub/
Project-URL: Issues, https://github.com/opendatahub-io/data-connect-hub/issues
Keywords: data,data-connect-hub,data-connectivity,flight-sql,opendatahub
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Typing :: Typed
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: httpx<1,>=0.27
Requires-Dist: pydantic<3,>=2.0
Provides-Extra: flight
Requires-Dist: adbc-driver-flightsql>=1.0.0; extra == "flight"
Requires-Dist: pyarrow>=14.0; extra == "flight"
Requires-Dist: pandas>=2.0; extra == "flight"
Requires-Dist: protobuf>=6.33.5; extra == "flight"
Provides-Extra: dev
Requires-Dist: build>=1.2; extra == "dev"
Requires-Dist: numpy<2.5; extra == "dev"
Requires-Dist: pytest>=8.0; extra == "dev"
Requires-Dist: pytest-cov>=5.0; extra == "dev"
Requires-Dist: ruff>=0.5; extra == "dev"
Requires-Dist: twine>=5.0; extra == "dev"
Requires-Dist: mypy>=1.10; extra == "dev"
Dynamic: license-file

# Data Connect Hub Python SDK

Typed Python client for managing [Data Connect Hub](https://github.com/opendatahub-io/data-connect-hub) connections and querying connected data sources.

- Manage connection types and connections through the REST API
- Validate credentials and connection readiness before querying data
- Query tabular data through Apache Arrow Flight SQL
- Download binary data from connections that support binary reads
- Authenticate with static or automatically refreshed bearer tokens

## Requirements

- Python 3.11 or newer
- A running Data Connect Hub deployment with its gateway accessible from your environment
- A tenant namespace and a bearer token or service account authorized to use the Data Connect Hub services

## Installation

```bash
# REST only (default)
pip install data-connect-hub

# REST + Flight SQL
pip install "data-connect-hub[flight]"
```

The default installation includes REST support and installs `httpx` and `pydantic`. Install the `[flight]` extra when you need Flight SQL tabular queries (`read`, `read_pandas`, `read_batches`, `get_tables`). REST-only workflows — connection management, credential testing, readiness checks, and binary downloads — work with the default install. The `flight` extra also installs the Flight SQL driver, PyArrow, and pandas.

## Getting Started

### Create a Client

Provide the Data Connect Hub gateway host, bearer token, and tenant namespace:

```python
from data_connect_hub import DataConnectClient

client = DataConnectClient(
    endpoint="dch.example.com:8443",
    token="<your-token>",
    tenant_id="my-tenant",
)

connections = client.list_connections()
for connection in connections:
    print(connection.id, connection.name, connection.status.state)
```

The SDK derives HTTPS and gRPC+TLS URLs from `endpoint`. A scheme is not required and is ignored when provided. Only TLS endpoints are supported. Use the client as a context manager, or call `client.close()` when finished.

### Client Options

| Option | Type | Default | Description |
|---|---|---|---|
| `endpoint` | `str` | Required | Gateway host or `host:port` |
| `token` | `str` | `""` | Static bearer token without the `Bearer` prefix |
| `token_provider` | `Callable[[], str] \| None` | `None` | Supplies and refreshes a bearer token |
| `tenant_id` | `str` | `""` | Tenant Kubernetes namespace; required for API requests |
| `api_base` | `str` | `"/api/v1alpha1/data"` | REST API path prefix |
| `rest_timeout` | `float` | `30.0` | REST request timeout in seconds |
| `ca_cert` | `str \| None` | `None` | Path to a custom CA certificate |
| `insecure` | `bool` | `False` | Disables TLS certificate verification |
| `max_retries` | `int` | `3` | Retry attempts for transient REST failures; `0` disables retries |
| `backoff_base` | `float` | `0.5` | Initial retry backoff in seconds |
| `backoff_max` | `float` | `30.0` | Maximum retry backoff in seconds |
| `flight_timeout` | `float \| None` | `None` | Timeout for Flight SQL RPC calls |

### Authentication and TLS

`token` and `token_provider` are mutually exclusive. A token provider is called once, cached, and called again when a request receives a `401 Unauthorized` response:

```python
client = DataConnectClient(
    endpoint="dch.example.com:8443",
    token_provider=get_fresh_token,
    tenant_id="my-tenant",
    ca_cert="/path/to/cluster-ca.pem",
)
```

Use `insecure=True` only for development environments where certificate verification is intentionally disabled.

### Configure a Data Source

Before querying data:

1. Define or select a [connection type](#connection-types-rest) for the provider.
2. Create a [connection](#connection-management-rest) with credentials for the data source.

The API reference describes both resources and their credential options. For a complete runnable workflow, see the [quickstart notebook](https://github.com/opendatahub-io/data-connect-hub/blob/main/sdk/python/examples/quickstart.ipynb).

### Query Tabular Data

Install the `flight` extra, then pass the connection ID with the SQL query:

```python
table = client.read("SELECT * FROM prompts", connection_id="conn-uuid")
df = table.to_pandas()
```

## Examples

| Example | Demonstrates |
|---|---|
| [quickstart.ipynb](https://github.com/opendatahub-io/data-connect-hub/blob/main/sdk/python/examples/quickstart.ipynb) | Guided REST and Flight SQL walkthrough |
| [connection_types.py](https://github.com/opendatahub-io/data-connect-hub/blob/main/sdk/python/examples/connection_types.py) | Listing, creating, and deleting connection types |
| [connections.py](https://github.com/opendatahub-io/data-connect-hub/blob/main/sdk/python/examples/connections.py) | Credential testing and connection lifecycle operations |
| [binary_download.py](https://github.com/opendatahub-io/data-connect-hub/blob/main/sdk/python/examples/binary_download.py) | Binary downloads with `download_binary` |
| [flight_query.py](https://github.com/opendatahub-io/data-connect-hub/blob/main/sdk/python/examples/flight_query.py) | Tabular queries with Flight SQL |
| [token_provider.py](https://github.com/opendatahub-io/data-connect-hub/blob/main/sdk/python/examples/token_provider.py) | Refreshing short-lived Kubernetes service account tokens |

## API Reference

The REST API is the source of truth for every model below. See the [REST API reference](https://opendatahub-io.github.io/data-connect-hub/) for the full request/response schemas.

### Connection Types (REST)

Connection types describe a category of data source (e.g. PostgreSQL). They define the provider backend and the credential fields required to connect.

```python
client.list_connection_types() -> list[ConnectionType]
client.get_connection_type(type_id) -> ConnectionType
client.create_connection_type(name=..., provider=..., description=..., credentials_fields=...) -> ConnectionType
client.update_connection_type(type_id, name=..., provider=..., description=..., credentials_fields=...) -> ConnectionType
client.delete_connection_type(type_id) -> None
```

For example, define a PostgreSQL connection type with a required connection URI:

```python
from data_connect_hub import CredentialField

connection_type = client.create_connection_type(
    name="PostgreSQL",
    provider="postgres",
    description="PostgreSQL database",
    credentials_fields=[
        CredentialField(
            name="URI",
            label="Connection URI",
            required=True,
            type="string",
        )
    ],
)
```

Pass `description=None` to remove an existing description. Omitting `description` leaves it unchanged.

#### `ConnectionType`

| Field | Type | Description |
|---|---|---|
| `id` | `str` | Unique identifier |
| `name` | `str` | Display name |
| `provider` | `str` | Backend driver (e.g. `"postgres"`) |
| `description` | `str \| None` | Optional description |
| `tenant_id` | `str` | Owning namespace |
| `created_at` | `datetime \| None` | Creation timestamp |
| `updated_at` | `datetime \| None` | Last update timestamp |
| `credentials_fields` | `list[CredentialField]` | Credential fields required to connect |
| `status` | `ConnectionTypeStatus` | Transports the provider supports |

Pass `id` as the `type_id` argument to `get_connection_type`, `update_connection_type`, and `delete_connection_type` — and as `connection_type_id` to `create_connection`.

`status.capabilities` reports which transports the provider supports (`flight` and `rest`, both `bool`), so you can check before issuing a Flight SQL query:

```python
ct = client.get_connection_type("dct-a1b2c3d4")
if ct.status.capabilities.flight:
    table = client.read("SELECT * FROM prompts", connection_id=conn.id)
```

#### `CredentialField`

Describes a single input field in the connection credential form.

| Field | Type | Description |
|---|---|---|
| `name` | `str` | Field key (used as the secret key) |
| `label` | `str` | Human-readable label |
| `description` | `str \| None` | Optional help text |
| `required` | `bool` | Whether the field must be provided |
| `type` | `str` | Rendering hint for the form (see below) |
| `enum_values` | `list[EnumValue] \| None` | Allowed values when `type` is `"enum"` |
| `default_value` | `str \| None` | Optional default value |

`EnumValue` has two fields: `value` (the stored string) and `label` (the display string).

**`type` values:**

| Value | Meaning |
|---|---|
| `"string"` | Free-text single-line input |
| `"enum"` | One of `enum_values` |

`type` is a client-side rendering hint for credential forms — the server stores it but does not validate or use it when connecting. Backend behavior is determined by the connection type's `provider` field (e.g. `"postgres"`, `"s3"`), not by `CredentialField.type`. The server's credential check is only that every field with `required=True` is present in the submitted secret. Every [built-in connection type](https://github.com/opendatahub-io/data-connect-hub/tree/main/config/connection-types) uses `"string"` for `type`; your own may use any other value (e.g. `"password"` to hint that input should be masked), and clients that do not recognize it should treat it as `"string"`. The authoritative definition is the `Field` schema in the [REST API reference](https://opendatahub-io.github.io/data-connect-hub/).

### Connection Management (REST)

A connection pairs a connection type with the actual credentials (stored in a Kubernetes secret) and tracks the live status of the data source.

```python
client.list_connections() -> list[DataConnection]
client.get_connection(connection_id) -> DataConnection
client.create_connection(name=..., connection_type_id=..., data_format=..., credentials_ref=..., properties=...) -> DataConnection
client.create_connection(name=..., connection_type_id=..., data_format=..., credentials=..., properties=...) -> DataConnection
client.update_connection(connection_id, name=..., connection_type_id=..., data_format=..., credentials_ref=..., properties=...) -> DataConnection
client.delete_connection(connection_id) -> None
client.check_connection_readiness(connection_id) -> None
client.test_credentials(connection_type_id, credentials) -> None
client.export_connection(connection_id, secret_name) -> None
client.download_binary(connection_id, path) -> Generator[bytes, None, None]
```

#### `DataConnection`

| Field | Type | Description |
|---|---|---|
| `id` | `str` | Unique identifier |
| `name` | `str` | Display name |
| `data_connection_type_id` | `str` | `id` of the associated `ConnectionType` |
| `format` | `"tabular" \| "binary"` | Data format of the source (see below) |
| `tenant_id` | `str` | Owning namespace |
| `created_at` | `datetime` | Creation timestamp |
| `updated_at` | `datetime` | Last update timestamp |
| `credentials_ref` | `CredentialsRef` | Credential secret reference |
| `properties` | `dict[str, str]` | Driver-specific properties (values masked in repr) |
| `status` | `DataConnectionStatus` | Live connection health |

Pass `id` as the `connection_id` argument to `get_connection`, `update_connection`, `delete_connection`, and the Flight SQL methods.

**`format` values:**

| Value | Meaning | Providers |
|---|---|---|
| `"tabular"` | Queried with SQL, returns rows | `postgres`, `sqlite`, `elasticsearch`, `milvus`, `neo4j`, `uri`, `s3` |
| `"binary"` | Opaque objects addressed by path | `s3`, `uri` |

Tabular connections are read with the [Flight SQL methods](#tabular-data-queries-flight-sql). `client.download_binary(connection_id, path)` returns a generator that streams byte chunks from a binary connection. HTTP and connection errors are raised while consuming the generator. The response closes automatically when the generator is exhausted or explicitly closed.

You normally set `format` once, at `create_connection`, but it is not immutable: `update_connection(connection_id, data_format=...)` changes it, and the server accepts the new value without checking it against the provider or re-evaluating `status`. So switching a `postgres` connection to `binary` succeeds, leaves `status` reporting `ready`, and fails only when you try to read.

`credentials_ref` is a reference to a Kubernetes secret containing the connection credentials. Use `CredentialsRef(secret="secret-name")` where `secret-name` is the **name** of an existing secret in the tenant namespace (the namespace named by the connection's `tenant_id` that you passed to `DataConnectClient`). This is a bare secret name, not a `namespace/name` pair; cross-namespace references are not supported. If the secret is missing or unreadable, `status.state` becomes `"not_ready"`. The secret's keys must cover every `CredentialField` on the connection type that has `required=True`.

Alternatively, pass `credentials=InlineCredentials(secret="secret-name", properties={...})` when creating a connection. The service creates that Kubernetes secret and stores its reference. Exactly one of `credentials_ref` and `credentials` is required.

Use `test_credentials` to validate credentials without storing them, `check_connection_readiness` to refresh a saved connection's status, and `export_connection` to copy its credentials and metadata into another Kubernetes secret.

The credential keys must match the connection type's `credentials_fields`. For example, a PostgreSQL connection can be tested and created with inline credentials before its status is refreshed:

```python
import os

from data_connect_hub import InlineCredentials

credentials = {"URI": os.environ["POSTGRES_URI"]}
client.test_credentials("dct-a1b2c3d4", credentials)

conn = client.create_connection(
    name="my-db",
    connection_type_id="dct-a1b2c3d4",
    data_format="tabular",
    credentials=InlineCredentials(secret="my-db", properties=credentials),
)
client.check_connection_readiness(conn.id)
conn = client.get_connection(conn.id)
print(conn.status.state)

# Optional: creates or overwrites this secret in the tenant namespace.
client.export_connection(conn.id, "my-db-export")
```

See [`examples/connections.py`](https://github.com/opendatahub-io/data-connect-hub/blob/main/sdk/python/examples/connections.py) for a runnable lifecycle example that loads credentials from a JSON file rather than source code.

Binary downloads from connections with `format="binary"` return a generator of byte chunks. Write each chunk to a temporary file and replace the destination after a successful download to avoid buffering the object or leaving a partial destination:

```python
from pathlib import Path
from tempfile import TemporaryDirectory

destination = Path("model.bin")
with TemporaryDirectory(dir=destination.parent) as temporary_directory:
    temporary_path = Path(temporary_directory) / destination.name
    with temporary_path.open("wb") as output:
        for chunk in client.download_binary("conn-uuid", "models/model.bin"):
            output.write(chunk)
    temporary_path.replace(destination)
```

See [`examples/binary_download.py`](https://github.com/opendatahub-io/data-connect-hub/blob/main/sdk/python/examples/binary_download.py) for a runnable version with environment-based configuration.

**`DataConnectionStatus`:**

| Field | Type | Description |
|---|---|---|
| `state` | `"ready" \| "ingestion_not_ready" \| "not_ready"` | Connection health (see below) |
| `message` | `str \| None` | Status detail message |
| `updated_at` | `datetime \| None` | When the status was last evaluated |

**`state` values:**

| Value | Meaning |
|---|---|
| `"ready"` | Credentials are valid and the source is queryable |
| `"ingestion_not_ready"` | Credentials are valid, but the source cannot be queried |
| `"not_ready"` | The referenced secret is missing or invalid |

### Tabular Data Queries (Flight SQL)

```python
client.read(sql, connection_id) -> pyarrow.Table          # full result as Arrow Table
client.read_pandas(sql, connection_id) -> pd.DataFrame    # full result as pandas DataFrame
client.read_batches(sql, connection_id) -> Generator[RecordBatch]  # stream of Arrow RecordBatches
client.get_tables(connection_id) -> pyarrow.Table         # table metadata
client.server_info() -> dict                              # server metadata
```

`read_batches` returns a generator that streams results instead of buffering the full result set in memory. The underlying cursor and connection are closed automatically when the generator is exhausted or garbage-collected:

```python
for batch in client.read_batches("SELECT * FROM prompts", "conn-uuid"):
    process(batch)
```

A server-side failure surfaced mid-stream raises `DCHQueryError`. Automatic token refresh applies when the stream is opened; an authentication failure that occurs after the stream is open is not retried.

These require the `flight` extra. On a REST-only install the client still imports and all REST calls work; the first Flight call raises `DCHConfigError` telling you to install `data-connect-hub[flight]`.

## Error Handling

Every failure raised by the SDK derives from `DCHError`, so a single `except` covers transport failures, HTTP errors, and malformed responses alike:

```python
from data_connect_hub import DCHError, DCHNotFoundError

try:
    conn = client.get_connection("conn-uuid")
except DCHNotFoundError:
    ...
except DCHError as exc:  # connection, timeout, auth, schema drift, ...
    ...
```

| Exception | Raised when |
|---|---|
| `DCHConfigError` | Invalid client configuration or argument (e.g. a blank id) |
| `DCHConnectionError` | The server was unreachable or the transport failed |
| `DCHTimeoutError` | The request exceeded `rest_timeout` |
| `DCHAuthenticationError` / `DCHForbiddenError` | HTTP 401 / 403 |
| `DCHNotFoundError` | HTTP 404 |
| `DCHValidationError` | HTTP 400 / 422 |
| `DCHServerError` | HTTP 5xx |
| `DCHResponseError` | The response was not JSON, or did not match the expected schema |
| `DCHQueryError` | A Flight SQL query failed |

Transient failures — HTTP 429/502/503/504, timeouts, and network or protocol errors — are retried automatically with exponential backoff on idempotent methods. Binary response-body failures are not retried after streaming starts, because restarting could duplicate data. See `max_retries`, `backoff_base`, and `backoff_max`.

## Contributing

See the [contributing guide](https://github.com/opendatahub-io/data-connect-hub/blob/main/CONTRIBUTING.md) for development setup, commands, and release instructions.
