Metadata-Version: 2.4
Name: etlgen-client
Version: 0.2.0
Summary: Python SDK and CLI for Fibtec's ETL Integrator (etlgen.fibtec.co.uk)
Author: Fibtec Limited
License-Expression: Apache-2.0
Project-URL: Homepage, https://etlgen.fibtec.co.uk
Project-URL: Repository, https://github.com/fibtecltd/fibtec-etlgen
Project-URL: Bug Tracker, https://github.com/fibtecltd/fibtec-etlgen/issues
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: httpx>=0.27.0
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == "dev"
Requires-Dist: pytest-cov; extra == "dev"
Requires-Dist: ruff>=0.15.0; extra == "dev"
Requires-Dist: build; extra == "dev"
Requires-Dist: twine; extra == "dev"
Dynamic: license-file

# etlgen-client

Python SDK and command-line client for [Fibtec's ETL Integrator](https://etlgen.fibtec.co.uk):
describe a source and a destination, and get back the Python that connects
them, as an ETL script (optionally with tests), a REST or GraphQL connector, or
a webhook handler. Nothing you upload is stored by the service, and this
package stores and logs nothing either.

**Status: alpha (0.2.x).** Python 3.11+. One runtime dependency, `httpx`.

```bash
pip install etlgen-client
```

## Before you start: authentication

Read this first, because it decides which address you call and how.

There are two ways in, and they use **two different addresses**:

| | API key | Cognito ID token |
|---|---|---|
| For | CI, scheduled jobs, servers: anything unattended | Trying it by hand, or acting as a signed-in person |
| Lasts | Until you revoke it, or the expiry you chose | One hour |
| `base_url` | The stack's **`KeyApiUrl`** | The stack's **`ApiUrl`** |
| Argument | `api_key=` | `token=` |

Give a `Client` one or the other, never both.

### An API key

An organisation **Owner** creates a key on the portal's **API keys** page. Choose its
access (**Read**: price estimates, the activity summary and the balance; **Generate**: also
creates ETL scripts, connectors and webhook handlers, which **spends credit**) and an expiry.
The key (`elk_...`) is shown once; the service keeps only a hash, so a lost key is revoked and
replaced, not recovered. It acts as your whole organisation, so keep it out of code and
repositories: read it from the environment or a secret store.

```python
import os

from etlgen_client import Client

client = Client(
    api_key=os.environ["ETLGEN_API_KEY"],
    base_url="https://<key-api-id>.execute-api.<region>.amazonaws.com",  # KeyApiUrl, not ApiUrl
)
```

- `base_url` is required. Your administrator gets the key API's address from the `EtlGenApi`
  stack's `KeyApiUrl` output. It must be `https` (plain `http` is only accepted for `localhost`).
- The key API serves what this package calls and nothing else: no checkout, no GitHub
  delivery, no key management (a key cannot create a key).
- **A refused key is a 403, and the API does not say why.** `EtlGenAuthError` is raised for a
  key that is unknown, revoked or expired, that lacks the access the call needs (a read-only key
  cannot generate), **or that has used its allowance for the minute** (each key has a separate
  per-minute allowance for reads and for generation). An over-limit key therefore looks like a
  wrong one: wait a minute before concluding it has been revoked, and don't retry in a tight loop.
  The package does not retry a 403.
- A key never needs refreshing, so `api_key` is a plain string, not a function.

### A Cognito ID token

The portal API authenticates with a **Cognito ID token** that **expires after one hour**. You
give the client either a token, or a function that returns a fresh one:

```python
from etlgen_client import Client

client = Client(token=my_id_token, base_url="https://<your-api-id>.execute-api.<region>.amazonaws.com")
```

- It must be the **ID token**, not the access token: the API reads your
  organisation from the token's `email` claim.
- `base_url` is required. It is the API address your administrator gets from the
  `EtlGenApi` stack's `ApiUrl` output. It must be `https` (plain `http` is only
  accepted for `localhost`).
- **To try it quickly**, copy the ID token from your signed-in portal session
  (browser developer tools, Application, Local Storage, the entry whose name ends in
  `.idToken`). It is valid for an hour.
- **For a script that keeps running on a person's behalf**, pass a function. This example uses
  [`pycognito`](https://pypi.org/project/pycognito/), a third-party library (not a
  dependency of this package) that does Cognito's SRP password login and renews
  tokens. It works for password accounts, not Google or Apple sign-in:

  ```python
  from pycognito import Cognito

  user = Cognito(user_pool_id="<pool id>", client_id="<app client id>", username="you@company.com")
  user.authenticate(password="...")

  def fresh_id_token() -> str:
      user.check_token()        # renews the tokens when they have expired
      return user.id_token

  client = Client(token=fresh_id_token, base_url="https://...")
  ```

  The function is called once per request. For anything unattended, use an API key instead.

## Quick start

```python
from pathlib import Path

from etlgen_client import Client, EtlGenInsufficientCreditError, EtlGenTooLargeError

source = {"format": "xsd", "content": Path("customer.xsd").read_text()}
destination = {"format": "json_schema", "content": Path("customer.schema.json").read_text()}

with Client(token=fresh_id_token, base_url=BASE_URL) as client:
    quote = client.generate.estimate(source, destination)      # free, writes nothing
    print(quote["tier"], quote["price_expected_gbp_pence"], "pence")

    try:
        result = client.generate.etl(source, destination, include_tests=True)
    except EtlGenTooLargeError as exc:
        print(f"{exc.field_count} fields, limit {exc.max_field_count}: split it up")
    except EtlGenInsufficientCreditError as exc:
        print(f"top up first: this job costs {exc.price_gbp_pence} pence")
    else:
        Path("etl_job.py").write_text(result["script"])
        if result["tests"]:
            Path("test_etl_job.py").write_text(result["tests"])
        print(result["unresolved_fields"], result["triangle"]["tes"])
```

The first argument of each call is the dict the API takes: `{"format": ..., "content": ...}`
for a schema pair, an OpenAPI document (or a GraphQL introspection document) for a
connector, a declared webhook surface for a webhook handler. The server validates them.

## Methods

| Call | What it does | Retried automatically? |
|---|---|---|
| `client.generate.etl(source, destination, *, python_version, include_tests, field_mapping_overrides)` | The ETL script, with optional pytest tests | **never** |
| `client.generate.review(source, destination)` | Parse or infer both sides and return them for review, without generating or charging. Put a reviewed schema back as `confirmed_schema` and call `etl` | yes |
| `client.generate.connector(api_surface, *, python_version, platform)` | A REST or GraphQL connector (`files`) | **never** |
| `client.generate.webhook(webhook_surface, *, python_version)` | A webhook handler (`files`) | **never** |
| `client.generate.estimate(source, destination)` | Price quote and developer baseline | yes |
| `client.triangle.rollup()` / `client.triangle.get(job_id)` | The Iron Triangle for your organisation, or for one job | yes |
| `client.billing.get()` | Plan, credit balance, price table, free-plan quota | yes |

**Why the generation calls are never retried.** A paid-plan job deducts credit
before it runs, and the API has no idempotency key, so retrying a call whose
outcome is unknown could charge you twice. Reads and `estimate` are retried on
connection errors, timeouts and 5xx (three retries, with backoff). A timeout on a
generation call raises the underlying `httpx` exception.

The default timeout is 60 seconds, above API Gateway's own 30-second limit.

## Errors

Every API error is an `EtlGenError` subclass carrying `response_status` and
`response_body`. A network failure with no response raises the `httpx` exception.

| Exception | Status | Meaning |
|---|---|---|
| `EtlGenAuthError` | 401, 403 | 401: token missing, expired (an hour) or not an ID token. 403: an API key was refused (unknown, revoked, expired, without the access this call needs, or over its per-minute allowance; the API does not say which) |
| `EtlGenInsufficientCreditError` | 402 | Balance too low for this job (`.price_gbp_pence`). Nothing ran |
| `EtlGenPlanLimitError` | 403 | The free plan cannot take a pair this complex (`.tier`) |
| `EtlGenValidationError` | 422 | Bad request, or a schema that could not be parsed |
| `EtlGenTooLargeError` | 422 | Over the per-job field limit (`.field_count`, `.max_field_count`). Refused before any charge |
| `EtlGenContentSafetyError` | 422 | The result failed the leak check and was not delivered (`.reason`). The request is refunded (`.refunded` is `True`): the charge goes back to your credit, or a free-plan request is not counted. `.refunded` is `None` against an API from before refunds, where the work was done and **was not refunded** |
| `EtlGenQuotaError` | 429 | The free plan's monthly quota is used (`.monthly_request_cap`) |
| `EtlGenRateLimitError` | 429 | API Gateway throttling (`.retry_after`) |
| `EtlGenNotFoundError` | 404 | For example an unknown job id |
| `EtlGenError` | other | Base class. A 502 means the upstream model call failed, and the charge was refunded |

The two 422 subclasses are also `EtlGenValidationError`.

## Command line

`pip install etlgen-client` installs an `etlgen` command.

```bash
export ETLGEN_API_KEY=...                     # an API key, for unattended use
export ETLGEN_BASE_URL=https://<key-api-id>.execute-api.<region>.amazonaws.com   # KeyApiUrl
# or, by hand, with a Cognito ID token and the portal API's address (ApiUrl):
#   export ETLGEN_TOKEN=...                   # prefer this over --token
#   export ETLGEN_BASE_URL=https://<api-id>.execute-api.<region>.amazonaws.com

etlgen billing get
etlgen generate estimate --params params.json
etlgen generate etl --params params.json --out-dir ./generated
etlgen triangle get --params-json '{"job_id": "..."}'
etlgen list-functions --domain generate
```

There is deliberately no `--api-key` flag: a key outlives a one-hour token, and an argument
can show up in process listings and shell history. Set `ETLGEN_API_KEY` in the environment
(from a secret store, or with `read -rs` in your shell). Setting both `ETLGEN_API_KEY` and
`ETLGEN_TOKEN` (or `--token`) is an error: they go to different addresses.

`params.json` is a JSON object of the method's keyword arguments. Any string
that starts with `@file:` is replaced by that file's contents, so a schema never
has to be pasted into JSON:

```json
{
  "source":      {"format": "xsd",         "content": "@file:customer.xsd"},
  "destination": {"format": "json_schema", "content": "@file:customer.schema.json"},
  "include_tests": true
}
```

A method with required arguments, called with neither `--params` nor
`--params-json`, prints its signature and documentation instead.

`--out-dir DIR` (for `etl`, `connector` and `webhook`) writes the generated files
under `DIR` (`etl_job.py` and `test_etl_job.py` for an ETL job, the portal's own
names), overwriting existing files, and prints a summary without the file
bodies. A path in the result that would escape `DIR` is refused.

Exit codes: `0` ok, `1` other error or bad input (including a network failure),
`2` usage, `3` auth, `4` validation (including too large and content safety),
`5` insufficient credit, `6` free-plan limit or quota, `7` rate limited, `8` not
found.

A content-safety failure (exit 4) prints the check's `reason` on its own line: either
that the script embeds a value from your sample data, or that the check itself gave no
usable verdict. Nothing is delivered either way, and the line after it says
`this request was refunded`. Against an API from before refunds it says instead that the
work was done and is not refunded.

A failure with no response at all (connection refused, DNS, a timeout) prints one
`error:` line, not a traceback. If it happened on `generate etl`, `connector` or
`webhook` after the request may have been sent, a second line says the job may
still have run and been charged: those calls are never retried, so run
`etlgen billing get` before trying again.

## Licence

Apache-2.0. Copyright Fibtec Limited.
