Metadata-Version: 2.4
Name: datatoolpack
Version: 0.12.1
Summary: Official Python SDK for the AutoData ML data preparation pipeline API
Home-page: https://autodata.datatoolpack.com
Author: AutoData Team
Author-email: support@datatoolpack.com
Project-URL: Documentation, https://autodata.datatoolpack.com/docs
Project-URL: Bug Tracker, https://github.com/datatoolpack/autodata-client/issues
Keywords: autodata machine-learning data-preparation synthetic-data ml-pipeline
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.8
Description-Content-Type: text/markdown
Requires-Dist: requests>=2.25.0
Dynamic: author
Dynamic: author-email
Dynamic: classifier
Dynamic: description
Dynamic: description-content-type
Dynamic: home-page
Dynamic: keywords
Dynamic: project-url
Dynamic: requires-dist
Dynamic: requires-python
Dynamic: summary

# AutoData Python Client

Official Python SDK for the [AutoData](https://autodata.datatoolpack.com) ML data preparation pipeline API.

## Installation

```bash
pip install datatoolpack
```

Or install from source:

```bash
git clone https://github.com/datatoolpack/datatoolpack
cd datatoolpack
pip install .
```

## Quick Start

```python
from datatoolpack import AutoDataClient

with AutoDataClient(api_key="dtpk_YOUR_API_KEY") as client:
    result = client.process(
        file_path="data.csv",
        target_columns=["price"],
        output_rows=10000,
    )
    print(result["files"])
```

Get your API key from the [AutoData dashboard](https://autodata.datatoolpack.com/dashboard) → API Keys tab.

---

## Authentication

### API Key (recommended)

```python
with AutoDataClient(api_key="dtpk_YOUR_API_KEY") as client:
    ...
```

### Access Code (passcode)

If you have an access code instead of an API key:

```python
with AutoDataClient(passcode="123456789012") as client:
    ...
```

### Environment Variables

Set environment variables to avoid hardcoding credentials:

```bash
export AUTODATA_API_KEY="dtpk_YOUR_API_KEY"
# or
export AUTODATA_PASSCODE="123456789012"
# optional
export AUTODATA_BASE_URL="https://autodata.datatoolpack.com"
```

```python
# No need to pass api_key — reads from AUTODATA_API_KEY automatically
with AutoDataClient() as client:
    result = client.process("data.csv", target_columns="price")
```

---

## Supported File Formats

| Format   | Extensions         |
|----------|--------------------|
| CSV      | `.csv`             |
| Excel    | `.xlsx`, `.xls`    |
| Parquet  | `.parquet`         |
| JSON     | `.json`            |
| Feather  | `.feather`         |
| ORC      | `.orc`             |

---

## Reference

### `AutoDataClient(api_key, passcode, base_url, timeout, max_retries)`

| Parameter     | Type  | Default                              | Env Variable          | Description                                  |
|---------------|-------|--------------------------------------|-----------------------|----------------------------------------------|
| `api_key`     | `str` | `None`                               | `AUTODATA_API_KEY`    | API key starting with `dtpk_`                |
| `passcode`    | `str` | `None`                               | `AUTODATA_PASSCODE`   | Access code (alternative to api_key)         |
| `base_url`    | `str` | `"https://autodata.datatoolpack.com"` | `AUTODATA_BASE_URL`  | Server URL (no trailing slash)               |
| `timeout`     | `int` | `120`                                |                       | Request timeout in seconds                   |
| `max_retries` | `int` | `3`                                  |                       | Auto-retries for 429 / 502 / 503 / 504      |

Either `api_key` or `passcode` is required (via argument or env variable).

Implements context manager — use with `with` to auto-close connections:

```python
with AutoDataClient(api_key="dtpk_...") as client:
    ...
# session is closed automatically
```

Or close manually:

```python
client = AutoDataClient(api_key="dtpk_...")
# ... use client ...
client.close()
```

---

## Core Pipeline

### `client.process(...)` — Upload & run pipeline

```python
result = client.process(
    file_path="data.csv",           # Path to input file (CSV, XLSX, Parquet, ...)
    target_columns=["price"],       # y-column(s) for ML
    output_rows=10000,              # Target row count in the synthetic output (default 10000)
    tools={                         # Toggle pipeline steps (all optional)
        "anomaly": False,           # Anomaly detection (off by default)
        "dtc": True,                # Data Type Conversion
        "mdh": True,                # Missing Data Handler
        "dor": False,               # Dimensionality Reduction (off by default)
        "cds": True,                # Column Scaling
        "dsm": True,                # Data Split Manager
        "dsg": True,                # Synthetic Data Generator
    },
    advanced_params={               # Fine-grained parameters (all optional)
        "excluded_columns": ["id"], # Columns to drop before processing
        "text_mode": 0,             # 0=none, 1=neural, 2=tfidf
        "text_cleaning": True,      # Clean text before encoding
        "zscore_limit": 3.0,        # Z-score outlier threshold
        "dsg_mode": "copula",       # "copula" or "gan"
        "similarity_p": 95,         # Similarity percentile for DSG
    },
    wait=True,                      # Block until complete (default True)
    poll_interval=2,                # Status poll interval in seconds
    download_path="./outputs/",     # Where to save files (default auto)
    auto_download=True,             # Set False to skip download when wait=True
    output_preferences=["dsg.csv"], # Which files to download (default all)
    compressed=True,                # Download as ZIP (default True)
)
```

**Returns** a dict:

```python
{
    "session_id": "abc123...",
    "status": "completed",
    "files": [
        {"name": "dsg.csv", "url": "/download/.../dsg.csv", "size": 2097152, "description": "..."},
        {"name": "dsm_train.csv", ...},
    ],
    "row_count": 10000,
    "duration_seconds": 42.1,
}
```

Set `wait=False` to get back immediately with just `session_id` and `status`:

```python
result = client.process(file_path="data.csv", target_columns="price", wait=False)
session_id = result["session_id"]
```

---

### `client.get_status(session_id)` — Poll progress

```python
status = client.get_status(session_id)
# {
#   "status": "running",           # queued | running | completed | error | cancelled
#   "message": "Running MDH...",
#   "current_step": 3,
#   "total_steps": 6,
#   "progress_percent": 50,
#   "duration_seconds": 15.3,
# }
```

---

### `client.get_result(session_id)` — Fetch completed results

```python
result = client.get_result(session_id)
# {"status": "completed", "files": [...], "row_count": ..., "duration_seconds": ...}
```

---

### `client.wait_for_completion(session_id, poll_interval)` — Block until done

```python
result = client.wait_for_completion(session_id, poll_interval=3)
```

Prints live progress to stdout. Raises `AutoDataError` if processing fails.

---

### `client.cancel(session_id)` — Cancel a running job

```python
cancelled = client.cancel(session_id)  # True if acknowledged
```

---

### `client.retry_session(session_id)` — Retry a failed pipeline

Retry from the last checkpoint — no need to re-upload or re-configure:

```python
result = client.retry_session(session_id="failed-session-id")
# {'success': True, 'session_id': '...', 'message': 'Retrying from last checkpoint'}
```

---

### `client.download_results(session_id, ...)` — Download output files

```python
path = client.download_results(
    session_id,
    download_path="./my_outputs/",      # Directory to save into
    output_preferences=["dsg.csv"],     # Specific files only (None = all)
    compressed=True,                    # ZIP download (default) or individual files
)
print(f"Saved to {path}")
```

---

### `client.download_file(url, output_path)` — Download a single file

```python
client.download_file("/api/v1/download/abc123.../dsg.csv", "dsg.csv")
```

---

## Feature Selection (F4)

Rank a completed session's columns by predictive importance — **without re-running the pipeline**. Handy for trimming wide datasets before training.

### `client.recommend_features(session_id, target_columns, methods, top_k)`

```python
ranking = client.recommend_features(
    session_id="abc123...",            # a completed session you own
    target_columns=["price"],          # rank features against these target(s)
    methods=["mutual_information"],    # one or more methods (see below); default ["mutual_information"]
    top_k=20,                          # number of top features to return (default 20)
)
# {
#   "ranking":    [{"feature": "sqft", "score": 0.81}, ...],   # combined across methods
#   "per_method": {"mutual_information": [...], "shap_importance": [...]},
#   "warnings":   [...],
# }
```

**Available methods:** `variance_threshold`, `correlation_pruning`, `mutual_information`, `rfe`, `shap_importance`, `pca`. Pass several to blend their rankings, e.g. `methods=["mutual_information", "shap_importance"]`. (SHAP-based ranking uses a fast LightGBM model on the server; it degrades gracefully to `mutual_information` if those libraries are unavailable.)

---

## Inference & Retraining (v0.11.0+)

Apply a **completed** session's fitted transforms to new data — the missing
half of the ML lifecycle: train once, then score fresh production rows with
*exactly* the same encoding, imputation and scaling.

### `client.infer(...)` — Score new data

Lightweight pipeline (Anomaly Detection → DTC → MDH → CDS). Outputs
`features.csv` (+ `y_columns.csv` if targets are present in the new file).

```python
result = client.infer(
    original_session_id="5d3bc0f0-...",   # a completed training session you own
    file_path="new_rows.csv",             # columns must match the training data
    download_path="./scored",             # optional: save outputs locally
    return_dataframes=True,               # optional: get pandas DataFrames back
)
features_df = result["dataframes"]["features"]
```

### `client.retrain(...)` — Refresh a training set with new rows

Full pipeline including Split → CDS → DSM → DSG; produces
`retraining_output.csv` (features + targets combined).

```python
result = client.retrain(
    original_session_id="5d3bc0f0-...",
    file_path="new_rows.csv",
    run_dsg=True, output_rows=20000,      # synthetic augmentation target
    download_path="./retrained",
)
```

Both calls are **synchronous** (the response arrives when processing ends;
default per-request timeout is ≥ 900 s — raise `timeout=` for very large
files), require the original session to be **completed and owned by you**,
and are free of credit charges in v1.

---

## API Keys & Usage

### `client.list_keys()` — List API keys

```python
keys = client.list_keys()
# [{"id": "...", "name": "My Key", "prefix": "dtpk_abc123", "created_at": "..."}]
```

### `client.get_usage()` — Usage statistics

```python
usage = client.get_usage()
# {
#   "daily_credits_used": 500,
#   "daily_credit_limit": 10000,
#   "daily_remaining": 9500,
#   "lifetime_credits_used": 12340,
#   "lifetime_credit_limit": 1000000,
#   "lifetime_remaining": 987660,
#   "daily_request_count": 3,
#   "last_used_at": "2026-04-12T10:30:00Z",
# }
```

---

## Connectors

Read data directly from databases and cloud storage instead of uploading files.

### Supported connector types

| Type          | Description                        | Required secrets                                    |
|---------------|------------------------------------|-----------------------------------------------------|
| `sql`         | PostgreSQL, MySQL, SQL Server      | `connection_string`                                 |
| `snowflake`   | Snowflake Data Cloud               | `account`, `user`, `password`, `database`, `schema`, `warehouse` |
| `bigquery`    | Google BigQuery                    | `credentials_json`, `project_id`, `dataset`         |
| `mongodb`     | MongoDB / Atlas                    | `connection_string`, `database`                     |
| `s3`          | Amazon S3                          | `bucket`, `access_key_id`, `secret_access_key`, `region` |
| `gcs`         | Google Cloud Storage               | `bucket`, `credentials_json`                        |
| `databricks`  | Databricks Lakehouse               | `host`, `token`, `catalog`, `schema`                |
| `delta`       | Delta Lake                         | `path` (+ cloud credentials if remote)              |
| `fabric`      | Microsoft Fabric                   | `workspace_id`, `lakehouse_id`, `tenant_id`, `client_id`, `client_secret` |
| `kafka`       | Apache Kafka                       | `bootstrap_servers`, `topic`, `group_id`            |
| `kinesis`     | Amazon Kinesis                     | `stream_name`, `region`, `access_key_id`, `secret_access_key` |

> **More connectors:** AutoData supports 30+ source types. Beyond the common ones above you can also pass `oracle`, `redshift`, `cassandra`, `dynamodb`, `elasticsearch`, `clickhouse`, `timescaledb`, `synapse`, `azure_blob`, `adls`, `sap_hana`, `mqtt`, `opcua`, `influxdb`, `pi_web`, `salesforce`, `hubspot`, `stripe`, `netsuite`, `google_sheets`, `ga4`, `sharepoint`, and `http`/`rest` as `connector_type`. See **Dashboard → Documentation → Connector Data Sources** for each type's required `secrets` fields.

### Quick connector example

```python
with AutoDataClient(api_key="dtpk_...") as client:
    # Test the connection
    client.test_connector("sql", secrets={
        "connection_string": "postgresql://user:pass@host/db"
    })

    # List tables
    tables = client.discover("sql", secrets={
        "connection_string": "postgresql://user:pass@host/db"
    })
    print(tables)  # ["users", "orders", ...]

    # Preview columns
    info = client.preview("sql", table="orders", secrets={
        "connection_string": "postgresql://user:pass@host/db"
    })
    print(info["columns"])  # ["id", "price", "date", ...]

    # Run the full pipeline from the connector
    result = client.process_from_connector(
        connector_type="sql",
        table="orders",
        target_columns=["price"],
        secrets={"connection_string": "postgresql://user:pass@host/db"},
        output_rows=10000,
    )
    print(result["files"])
```

### Connector examples by type

**Snowflake:**
```python
secrets = {
    "account": "abc123.us-east-1",
    "user": "analyst",
    "password": "...",
    "database": "PROD",
    "schema": "PUBLIC",
    "warehouse": "COMPUTE_WH",
}
tables = client.discover("snowflake", secrets=secrets)
result = client.process_from_connector("snowflake", table="ORDERS",
    target_columns=["AMOUNT"], secrets=secrets)
```

**BigQuery:**
```python
secrets = {
    "credentials_json": '{"type":"service_account",...}',
    "project_id": "my-project",
    "dataset": "analytics",
}
tables = client.discover("bigquery", secrets=secrets)
```

**Amazon S3:**
```python
secrets = {
    "bucket": "my-data-lake",
    "access_key_id": "AKIA...",
    "secret_access_key": "...",
    "region": "us-east-1",
}
result = client.process_from_connector("s3", table="data/sales.csv",
    target_columns=["revenue"], secrets=secrets, output_rows=50000)
```

**MongoDB:**
```python
secrets = {
    "connection_string": "mongodb+srv://user:pass@cluster.mongodb.net",
    "database": "analytics",
}
tables = client.discover("mongodb", secrets=secrets)
```

---

### `client.test_connector(connector_type, secrets, credential_id)` — Test connection

```python
result = client.test_connector("s3", secrets={...})
# {"success": True, "message": "Connection successful"}
```

### `client.discover(connector_type, ...)` — List tables / files

```python
tables = client.discover("sql", secrets={...})
# ["users", "orders", "transactions"]
```

### `client.preview(connector_type, table, ...)` — Preview columns & row count

```python
info = client.preview("sql", table="orders", secrets={...})
# {"success": True, "columns": ["id", "price", ...], "row_count": 50000}
```

Supports optional `custom_query` parameter for SQL-based connectors:
```python
info = client.preview("sql", table="orders", custom_query="SELECT * FROM orders WHERE date > '2025-01-01'", secrets={...})
```

### `client.process_from_connector(...)` — Run pipeline from connector

```python
result = client.process_from_connector(
    connector_type="sql",
    table="orders",
    target_columns=["price"],
    secrets={"connection_string": "postgresql://..."},
    output_rows=10000,
    incremental=True,          # Only fetch new rows since last sync
    type_casts={"age": "int"}, # Override column types before processing
    wait=True,
    download_path="./outputs/",
)
```

Supports all the same options as `client.process()`: `tools`, `advanced_params`, `wait`, `poll_interval`, `auto_download`, `output_preferences`, `compressed`.

### `client.write_output(session_id, ...)` — Write results to a target

```python
client.write_output(
    session_id="abc123...",
    connector_type="sql",
    table_name="ml_prepared_data",
    secrets={"connection_string": "postgresql://..."},
    output_stage="dsg",        # which pipeline output to write
    if_exists="replace",       # "replace", "append", "fail", or "merge"
    merge_keys=["id"],         # column names for merge matching (when if_exists='merge')
)
# {"success": True, "rows_written": 10000, "table": "ml_prepared_data"}
```

---

## Credential Management

Save credentials on the server so you don't need to pass secrets every time:

```python
# Save a new credential
cred = client.save_credential(
    name="Production DB",
    connector_type="sql",
    secrets={"connection_string": "postgresql://..."},
)
cred_id = cred["id"]

# Use credential_id instead of secrets
tables = client.discover("sql", credential_id=cred_id)
result = client.process_from_connector("sql", table="orders",
    target_columns=["price"], credential_id=cred_id)

# List saved credentials (secrets are never exposed)
creds = client.list_credentials()

# Update a credential
client.update_credential(cred_id, name="Prod DB v2", secrets={...})

# Delete a credential
client.delete_credential(cred_id)
```

---

## Schema Mapping

Auto-suggest and apply column mappings between source and target schemas:

### `client.suggest_mapping(source_columns, target_columns)` — Auto-suggest mapping

```python
mapping = client.suggest_mapping(
    source_columns=["customer_name", "order_amt", "order_dt"],
    target_columns=["name", "amount", "date"],
    threshold=0.6,  # minimum similarity score (0-1)
)
# {
#   "mapping": {"customer_name": "name", "order_amt": "amount", "order_dt": "date"},
#   "scores": {"customer_name": 0.85, "order_amt": 0.72, "order_dt": 0.68}
# }
```

### `client.apply_mapping(session_id, mapping)` — Rename columns

```python
result = client.apply_mapping(
    session_id="abc123...",
    mapping={"old_col_name": "new_col_name", ...},
    output_stage="dsg",  # which pipeline stage to apply to
)
# {"columns": ["new_col_name", ...], "preview": [...]}
```

---

## Quality Alerts

Monitor pipeline metrics and get notified when thresholds are breached:

```python
# Create a quality alert rule
rule = client.create_quality_alert(
    name="High row loss",
    metric="row_loss_pct",    # row_loss_pct, null_pct, column_drop_count, duration_seconds
    operator=">",             # >, <, >=, <=, ==
    threshold=10.0,
    severity="critical",      # warning or critical
    stage="mdh",              # optional: anomaly, dtc, mdh, cds, dsm, dsg
)

# List rules
rules = client.list_quality_alerts()

# Update a rule
client.update_quality_alert(rule_id="...", threshold=15.0, severity="warning")

# Get fired alert events
events = client.get_alert_events(session_id="optional-filter")

# Delete a rule
client.delete_quality_alert(rule_id="...")
```

**Available metrics:**

| Metric              | Description                          |
|---------------------|--------------------------------------|
| `row_loss_pct`      | Percentage of rows lost in a step    |
| `null_pct`          | Percentage of null values remaining  |
| `column_drop_count` | Number of columns dropped            |
| `duration_seconds`  | Time taken by a pipeline step        |

---

## Sync Watermarks

Track incremental sync progress for connector-based pipelines:

```python
# List all watermarks
watermarks = client.list_watermarks()
# [{"id": "wm-123", "connector_type": "sql", "table_name": "orders",
#   "watermark_column": "updated_at", "last_value": "2026-04-12T00:00:00Z"}]

# Reset a watermark (re-sync from beginning)
client.reset_watermark(watermark_id="wm-123")
```

---

## Scheduled Runs

Automate recurring pipeline executions:

```python
# Create an interval-based schedule (every 24 hours)
schedule = client.create_scheduled_run(
    name="Daily ETL",
    schedule_type="interval",
    interval_minutes=1440,
    connector_type="snowflake",
    table="orders",
    credential_id="cred-id",
    y_columns=["target"],
    output_rows=50000,
)

# Create with cron expression (weekdays at 8 AM)
schedule = client.create_scheduled_run(
    name="Weekday ETL",
    schedule_type="cron",
    cron_expression="0 8 * * 1-5",
    connector_type="snowflake",
    table="orders",
    credential_id="cred-id",
    y_columns=["target"],
    tools={"anomaly": True, "dsg": False},  # customize pipeline steps
)

# List all schedules
schedules = client.list_scheduled_runs()

# Enable/disable a schedule
client.toggle_scheduled_run(run_id="run-id")

# Trigger immediately (ignore schedule)
client.run_scheduled_now(run_id="run-id")

# Delete a schedule
client.delete_scheduled_run(run_id="run-id")
```

---

## Triggers (F7)

Auto-start a pipeline when an external event fires — an inbound webhook, a connector watermark advancing, another pipeline completing, or a quality alert. Triggers are feature-flagged on the server (`FEATURE_FLAG_TRIGGERS`); if the flag is off, these endpoints return `404`.

### `client.create_trigger(name, trigger_type, config, target_pipeline_config, enabled)`

```python
trigger = client.create_trigger(
    name="Reprocess on new S3 batch",
    trigger_type="inbound_webhook",   # inbound_webhook | watermark_advance | pipeline_completion | quality_alert
    config={},                         # type-specific (see table below)
    target_pipeline_config={           # the pipeline to run when the trigger fires
        "source_type": "s3",
        "credential_id": "cred-uuid",
        "table": "incoming/sales.csv",
        "target_columns": ["revenue"],
        "output_rows": 10000,
    },
    enabled=True,
)
# For inbound_webhook the secret is returned ONLY once — save it now:
print(trigger["webhook_secret"], trigger["inbound_url"])
```

**Trigger types & their `config`:**

| `trigger_type`        | `config` keys |
|-----------------------|---------------|
| `inbound_webhook`     | optional `{"hmac_key": "..."}` — require HMAC-SHA256-signed requests |
| `watermark_advance`   | `{"credential_id", "table", "watermark_column", "min_advance"}` |
| `pipeline_completion` | `{}` — chain off another pipeline finishing |
| `quality_alert`       | `{}` — fire when a quality-alert rule trips |

```python
# List / fetch / update / delete
triggers = client.list_triggers()
t        = client.get_trigger(trigger_id)
client.update_trigger(trigger_id, enabled=False)   # name | enabled | config | target_pipeline_config
client.delete_trigger(trigger_id)

# Dry-run: validate config + target WITHOUT firing
report = client.test_trigger(trigger_id)
# {"would_fire": True, "checks": {...}}
```

**Firing an inbound webhook** from your own system:

```bash
curl -X POST "https://autodata.datatoolpack.com/trigger/inbound/<webhook_secret>" \
     -H "Content-Type: application/json" \
     -d '{"any": "payload"}'
# If you set an hmac_key, also sign the raw body:
#   X-Signature: sha256=<hex HMAC-SHA256 of the request body>
```

Inbound firing is rate-limited and HMAC-verified server-side; repeated failures auto-disable the trigger.

---

## Folder Listeners

Automatically trigger pipelines when new files appear in cloud storage:

```python
# Create an S3 folder listener
listener = client.create_listener(
    name="S3 Ingest",
    source_type="s3",            # s3, gcs, azure_blob, local, sftp
    watch_path="s3://bucket/incoming/",
    credential_id="cred-id",
    y_columns=["target"],
    pipeline_config={"enable_dsg": False},
    output_rows=10000,
)

# List all listeners
listeners = client.list_listeners()

# Update a listener
client.update_listener(listener_id="...", watch_path="s3://bucket/new-path/", enabled=False)

# Delete a listener
client.delete_listener(listener_id="...")
```

**Supported source types:** `s3`, `gcs`, `azure_blob`, `local`, `sftp`

---

## SFTP Upload

Upload files to the AutoData server via SFTP for processing:

```python
# Get SFTP server connection info
info = client.sftp_info()
# {"host": "...", "port": 22, "username": "..."}

# Manage SFTP credentials
cred = client.create_sftp_credential(name="My Upload Key")
# {"id": "...", "username": "...", "password": "..."} (password shown only once)

creds = client.list_sftp_credentials()
client.delete_sftp_credential(credential_id="...")
```

---

## Spark (Large Data)

For datasets too large for pandas, AutoData can use PySpark on the server:

### `client.spark_status()` — Check Spark availability

```python
status = client.spark_status()
# {"available": True, "spark_version": "3.5.0", ...}
```

### `client.spark_read(file_path, sample_rows)` — Read large files

```python
info = client.spark_read(
    file_path="/data/huge_dataset.parquet",
    sample_rows=20,
)
# {"row_count": 50000000, "columns": [...], "dtypes": {...}, "sample": [...]}
```

### `client.spark_transform(...)` — Apply Spark transformations

```python
result = client.spark_transform(
    file_path="/data/huge_dataset.parquet",
    output_path="/data/transformed.csv",
    operations=[
        {"op": "cast", "type_casts": {"age": "int"}},
        {"op": "drop_nulls", "subset": ["col1"]},
        {"op": "fill_nulls", "strategy": "mean"},
        {"op": "normalize", "method": "minmax"},
        {"op": "sample", "n": 50000},
    ],
    output_format="csv",  # csv or parquet
)
# {"row_count": 50000, "columns": [...]}
```

---

## Worker Status

Check the processing worker fleet:

```python
status = client.worker_status()
# {"backend": "local", "active_jobs": 2, "queue_size": 0, ...}
```

---

## Error Handling

All API errors raise `AutoDataError`:

```python
from datatoolpack import AutoDataClient, AutoDataError

with AutoDataClient() as client:
    try:
        result = client.process("data.csv", target_columns="price")
    except AutoDataError as e:
        print(f"API error {e.status_code}: {e}")
    except FileNotFoundError as e:
        print(f"File not found: {e}")
    except ValueError as e:
        print(f"Invalid input: {e}")  # e.g. unsupported file format
```

`AutoDataError` attributes:
- `str(e)` — human-readable error message from the server
- `e.status_code` — HTTP status code (e.g. `401`, `429`, `500`), or `None` for non-HTTP errors

Transient errors (429, 502, 503, 504) are automatically retried up to `max_retries` times with exponential back-off.

---

## Advanced Example: Non-blocking with manual polling

```python
import time
from datatoolpack import AutoDataClient, AutoDataError

with AutoDataClient() as client:
    # Start job without blocking
    job = client.process(
        "large_dataset.csv",
        target_columns=["churn"],
        wait=False,
    )
    session_id = job["session_id"]
    print(f"Job started: {session_id}")

    # Poll manually
    while True:
        status = client.get_status(session_id)
        print(f"  {status['progress_percent']}% — {status['message']}")
        if status["status"] == "completed":
            break
        elif status["status"] in ("error", "cancelled"):
            raise AutoDataError(f"Job {status['status']}: {status['message']}")
        time.sleep(5)

    # Download results
    path = client.download_results(session_id, download_path="./outputs/")
    print(f"Results saved to {path}")
```

---

## Sessions, Preferences, Webhooks (v0.12.0+)

The whole dashboard surface is now reachable from the SDK — sessions,
parameter profiles, scheduled runs, webhooks, templates and sharing all live
under `/api/v1`.

### Sessions

```python
for s in client.list_sessions():
    print(s["session_id"], s["status"])

client.rename_session(session_id, "BBVA credit — August")
detail = client.get_session(session_id)      # incl. the parameters it ran with
```

### Dry-run validation

Check how the server reads a file *before* spending a run on it:

```python
info = client.validate("new_data.csv")
print(info["columns"], info["row_count"])
```

### Preference profiles

Four groups — `outputs`, `advanced`, `anomaly`, `completion`:

```python
prefs = client.get_preferences("outputs")
prefs["dsg_output"]["disk"] = False
client.set_preferences("outputs", prefs)
client.reset_preferences("anomaly")          # back to server defaults
```

### Webhooks — get told when a job finishes, instead of polling

```python
wh = client.create_webhook(
    url="https://your-app.example.com/hooks/autodata",
    events=["session.completed", "session.failed"],
    secret="a-long-random-string",           # sign & verify: see note below
)
client.test_webhook(wh["webhook"]["id"])     # confirm your endpoint accepts it
client.webhook_deliveries(wh["webhook"]["id"])
```

Always set `secret` and verify the signature on your side — without it,
anyone who learns your URL can forge a "your job finished" callback.

### Templates and sharing

```python
client.create_template("credit-monthly", config={...})
client.share_session(session_id, shared_with_id="user-b", permission="read",
                     expires_at="2026-12-31T00:00:00Z")
```

Give public share links an `expires_at` — a link without one never stops
working.

### Account & credits

```python
client.account_profile()   # credit balance, role, limits (whole account)
client.account_usage()     # usage across every key, broken down by tool
client.get_usage()         # just the key you're authenticating with
```

---

## Credits & cost estimation (v0.12.1+)

Runs are billed by data volume and by which stages you enable, scaled by how
many rows you ask the synthesiser to produce. Cost isn't guessable from the
outside, so you can price a run before committing to it:

```python
quote = client.estimate(
    input_size_mb=250,
    input_rows=1_000_000,
    output_rows=2_000_000,      # 2x synthesis -> 2x the price
    tools={"anomaly": True, "dtc": True, "mdh": True,
           "cds": True, "dsm": True, "dsg": True},
)
print(quote["total_cost"], quote["tool_costs"])
```

`estimate()` runs nothing and charges nothing. Pass
`session_type="inference"` to price a scoring call, which is charged at a
reduced rate compared to a full pipeline run.

After a run, `get_result()` tells you what it actually cost:

```python
result = client.get_result(session_id)
result["credits_charged"]      # what this run cost
result["credits_remaining"]    # balance afterwards
result.get("credit_warning")   # present once the balance goes negative
```

A negative balance does **not** stop your jobs — they keep running and the
warning is there so you can top up before it becomes a conversation. Check the
account-wide picture any time with `client.account_profile()`.

---

## Data residency — keeping data away from external LLMs (v0.12.1+)

Three pipeline stages can send data to a language model: parameter
optimization (your project description and row/column counts), anomaly
detection (column names plus up to ten sample values per column) and data
completion/validation (identifiable record content). Everything else —
encoding, imputation, scaling, splitting, synthesis — is local computation.

If your data must not leave your infrastructure, turn the calls off:

```python
result = client.process(
    file_path="customers.csv",
    target_columns="churn",
    advanced_params={"llm_enabled": False},   # no stage contacts a model
)
```

Anomaly detection still runs using local heuristics; completion/validation is
skipped, since it has no local equivalent.

**Fail instead of degrading.** By default a stage that can't reach a model
falls back to heuristics — convenient, but silent. If you would rather know:

```python
advanced_params={"strict_llm": True}          # raise instead of degrading
```

**Proof, per run.** Every result carries an audit block, so "did our data
leave?" is answerable rather than assumed:

```python
result["llm"]
# {
#   "llm_enabled": False,
#   "llm_used": False,                  # the auditable claim
#   "endpoint_host": "llm.internal.example",
#   "endpoint_is_public": False,        # flags api.x.ai, api.openai.com, ...
#   "stages": {
#     "anomaly_detection": {"llm_used": False, "reason": "disabled by policy"},
#     "dcv":               {"llm_used": False, "reason": "disabled by policy"},
#     "grok_optimizer":    {"llm_used": False, "reason": "not requested"}
#   }
# }
```

Every stage that didn't call out records *why*, so a fallback is never silent.
Runs from before this feature have no `llm` block at all — that reads as
"unknown", not "clean".

Operators can also point the whole deployment at their own OpenAI-compatible
endpoint (`GROQ_BASE_URL`) or disable LLM calls server-wide
(`LLM_ENABLED=0`), which overrides whatever a caller asks for.

---

## Complete Method Reference

### Sessions & Preferences (v0.12.0+)
| Method | Description |
|--------|-------------|
| `list_sessions()` | List the account's sessions |
| `get_session(session_id)` | Full detail incl. run parameters |
| `rename_session(session_id, name)` | Set a display name |
| `set_session_retraining(session_id, enabled)` | Mark as a retraining base |
| `list_session_outputs(session_id)` | In-memory outputs for a session |
| `validate(file_path)` | Dry-run a CSV without starting a pipeline |
| `get_preferences(kind)` / `set_preferences(kind, values)` / `reset_preferences(kind)` | `outputs`, `advanced`, `anomaly`, `completion` |
| `account_profile()` / `account_usage()` | Credit balance and account-wide usage |
| `estimate(input_size_mb=..., ...)` | Price a run before starting it (charges nothing) |

### Webhooks, Templates & Sharing (v0.12.0+)
| Method | Description |
|--------|-------------|
| `list_webhooks()` / `create_webhook(url, ...)` / `get_webhook(id)` | Webhook CRUD |
| `update_webhook(id, **fields)` / `delete_webhook(id)` | |
| `test_webhook(id)` / `webhook_deliveries(id)` | Send a test delivery; inspect attempts |
| `list_templates()` / `create_template(name, config, ...)` | Reusable pipeline configs |
| `update_template(id, **fields)` / `delete_template(id)` | |
| `share_session(session_id, ...)` / `unshare_session(share_id)` | Share a session or mint a link |
| `list_shared_sessions()` / `list_received_sessions()` | Shares you made / received |

### Core Pipeline
| Method | Description |
|--------|-------------|
| `process(file_path, target_columns, ...)` | Upload a file and run the pipeline |
| `get_status(session_id)` | Poll processing progress |
| `get_result(session_id)` | Get completed session results |
| `wait_for_completion(session_id)` | Block until session finishes |
| `cancel(session_id)` | Cancel a running job |
| `retry_session(session_id)` | Retry a failed session from checkpoint |
| `download_results(session_id, ...)` | Download result files |
| `download_file(url, output_path)` | Download a single file |

### API Keys & Usage
| Method | Description |
|--------|-------------|
| `list_keys()` | List all API keys for the account |
| `get_usage()` | Get credit usage statistics |

### Feature Selection
| Method | Description |
|--------|-------------|
| `recommend_features(session_id, target_columns, ...)` | Rank a completed session's features (F4) |

### Connectors
| Method | Description |
|--------|-------------|
| `test_connector(connector_type, ...)` | Test connectivity to a data source |
| `discover(connector_type, ...)` | List tables/files in a data source |
| `preview(connector_type, table, ...)` | Preview columns and row count |
| `process_from_connector(connector_type, table, ...)` | Run pipeline from a connector source |
| `write_output(session_id, connector_type, ...)` | Write results to a database/storage target |

### Credentials
| Method | Description |
|--------|-------------|
| `list_credentials()` | List saved credentials |
| `save_credential(name, connector_type, secrets)` | Save a new credential |
| `update_credential(credential_id, ...)` | Update a saved credential |
| `delete_credential(credential_id)` | Delete a saved credential |

### Schema Mapping
| Method | Description |
|--------|-------------|
| `suggest_mapping(source_columns, target_columns)` | Auto-suggest column mapping |
| `apply_mapping(session_id, mapping)` | Rename columns using a mapping |

### Quality Alerts
| Method | Description |
|--------|-------------|
| `list_quality_alerts()` | List quality alert rules |
| `create_quality_alert(name, metric, operator, threshold, ...)` | Create an alert rule |
| `update_quality_alert(rule_id, ...)` | Update an alert rule |
| `delete_quality_alert(rule_id)` | Delete an alert rule |
| `get_alert_events(session_id=None)` | List fired alert events |

### Sync Watermarks
| Method | Description |
|--------|-------------|
| `list_watermarks()` | List all sync watermarks |
| `reset_watermark(watermark_id)` | Reset a watermark to re-sync |

### Scheduled Runs
| Method | Description |
|--------|-------------|
| `list_scheduled_runs()` | List all scheduled runs |
| `create_scheduled_run(name, schedule_type, ...)` | Create a scheduled run |
| `delete_scheduled_run(run_id)` | Delete a scheduled run |
| `toggle_scheduled_run(run_id)` | Enable/disable a scheduled run |
| `run_scheduled_now(run_id)` | Trigger a scheduled run immediately |

### Triggers
| Method | Description |
|--------|-------------|
| `list_triggers()` | List your triggers |
| `create_trigger(name, trigger_type, ...)` | Create an event trigger (F7) |
| `get_trigger(trigger_id)` | Fetch one trigger |
| `update_trigger(trigger_id, ...)` | Update a trigger |
| `delete_trigger(trigger_id)` | Delete a trigger |
| `test_trigger(trigger_id)` | Dry-run a trigger without firing |

### Folder Listeners
| Method | Description |
|--------|-------------|
| `list_listeners()` | List folder listeners |
| `create_listener(name, source_type, watch_path, ...)` | Create a folder listener |
| `update_listener(listener_id, ...)` | Update a folder listener |
| `delete_listener(listener_id)` | Delete a folder listener |

### SFTP
| Method | Description |
|--------|-------------|
| `sftp_info()` | Get SFTP server connection info |
| `list_sftp_credentials()` | List SFTP credentials |
| `create_sftp_credential(name)` | Create SFTP credential |
| `delete_sftp_credential(credential_id)` | Delete SFTP credential |

### Spark (Large Data)
| Method | Description |
|--------|-------------|
| `spark_status()` | Check if Spark is available |
| `spark_read(file_path, sample_rows)` | Read a large file via Spark |
| `spark_transform(file_path, output_path, operations, ...)` | Apply Spark transformations |

### Worker Status
| Method | Description |
|--------|-------------|
| `worker_status()` | Get worker fleet status |

---

## Requirements

- Python >= 3.8
- `requests` >= 2.25.0

## License

MIT
