Metadata-Version: 2.5
Name: segmentstream-pipeline
Version: 0.1.0a28
Summary: Dagster and Ibis runtime integration for SegmentStream pipelines
Project-URL: Homepage, https://segmentstream.com
Author: SegmentStream
License-Expression: Apache-2.0
License-File: LICENSE
License-File: NOTICE
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Typing :: Typed
Requires-Python: <3.14,>=3.12
Requires-Dist: dagster==1.13.16
Requires-Dist: httpx<1,>=0.28
Requires-Dist: ibis-framework<13,>=12
Requires-Dist: pydantic<3,>=2.10
Provides-Extra: bigquery
Requires-Dist: ibis-framework[bigquery]<13,>=12; extra == 'bigquery'
Description-Content-Type: text/markdown

# SegmentStream pipeline SDK

Declare source tables with `s.ingestion` and warehouse SQL transformations with
`s.model`. The SDK resolves inputs, validates explicit partitioning and capabilities,
and compiles native Dagster definitions using the existing warehouse loader.

```python
from collections.abc import Iterator

import ibis.expr.types as ir
import segmentstream as s


@s.ingestion(
    tier=s.Tier.Bronze,
    partitioning=s.Partitioning.Unpartitioned,
    schema={"amount": "float64"},
)
def orders() -> Iterator[s.IngestionRow]:
    yield {"amount": 25.0}


@s.model(
    tier=s.Tier.Gold,
    partitioning=s.Partitioning.Unpartitioned,
    schema={"total_revenue": "float64"},
)
def revenue(orders: ir.Table) -> ir.Table:
    return orders.aggregate(total_revenue=orders.amount.sum())


defs = s.definitions(
    assets=[orders, revenue],
    jobs=[s.job("revenue", assets=[revenue], include_upstream=True)],
)
```

The [workspace documentation](../../docs/README.md) is organized by concept:
[connections](../../docs/connections.md), [ingestion](../../docs/ingestion.md),
[models](../../docs/models.md), [partitioning](../../docs/partitioning.md),
[jobs](../../docs/jobs.md), [execution](../../docs/execution.md), [checks](../../docs/checks.md),
[runtime values](../../docs/runtime.md), and [warehouse storage](../../docs/warehouse.md).
The CLI bundles these same pages into every starter workspace.

`s.validate(defs)` checks the graph and named job configuration without executing
assets. `s.describe(defs)` explains resolved inputs, per-asset policies, storage,
capabilities, and job selections without resolving credentials. See the
[modular commerce starter](../../templates/workspace-app/pipeline/README.md)
for daily row ingestion, batched models, and a partitioned check.

## Compatibility

The SDK contains authoring, graph inspection, Ibis resources, connection HTTP
clients, and managed secret access. Managed orchestration, workers, PostgreSQL setup, and the metadata
webserver live in the internal [pipeline runtime](../pipeline-runtime/README.md).

In `0.1.0a28`, use `s.get_connection(key).get_http_client()` for provider requests.
The client forwards requests through the backend, where provider credentials and
refresh remain. The access-token and full-credential accessors have been removed.
See [connections](../../docs/connections.md) for declaring API destinations.

Workspace pipelines declare `segmentstream-pipeline[bigquery]`; our image supplies
the runtime. The old SDK `cloud-run` and `webserver` extras and
`segmentstream.runtime` module are removed in a22.

`segmentstream.dagster` retains its native exports. Native assets and high-level
declarations can coexist in `s.definitions`. Root `s.resource` and
`s.IngestionResource` remain available for legacy ingestion. Import their helpers
from `segmentstream`; the old public `segmentstream.ingestion` module is retired
so `s.ingestion` can be the decorator. [Migration examples](../../docs/native-dagster.md)
cover the mapping and IO defaults.

## Managed Google clients

The Sandbox runtime integration supplies refreshable credentials to the built-in
BigQuery warehouse and GCS IO manager. Ibis remains the user-facing warehouse API;
its internal official Google client renews credentials automatically throughout
the authorized run. Ten minutes is the lifetime of each access token, not a limit
on pipeline duration. Managed execution does not provide general ADC support.
See [runtime lifecycle and credential brokering](../../docs/sandbox-runtime.md).
Managed Dagster execution uses the backend compute provider API for every step,
including SQL assets and checks. The coordinator owns scheduling and retries; the
backend owns execution identity, scoped credentials and cleanup. See the
[compute provider contract](../../docs/provider-compute.md).

## Releases

This change targets `0.1.0a28`. Publish the SDK before releasing a CLI whose starter
pins it. PyPI versions are immutable; do not republish an existing version.

The protected `pipeline-sdk-release.yml` workflow builds a wheel and source
distribution from a `pipeline-sdk-v<version>` tag, then uses PyPI Trusted Publishing
from the `pypi` GitHub environment. No long-lived publishing token is stored in GitHub.
