Metadata-Version: 2.5
Name: segmentstream-pipeline
Version: 0.1.0a42
Summary: Dagster and Ibis runtime integration for SegmentStream pipelines
Project-URL: Homepage, https://segmentstream.com
Author: SegmentStream
License-Expression: Apache-2.0
License-File: LICENSE
License-File: NOTICE
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Typing :: Typed
Requires-Python: <3.14,>=3.12
Requires-Dist: anyio<5,>=4
Requires-Dist: dagster==1.13.16
Requires-Dist: httpx<1,>=0.28
Requires-Dist: ibis-framework<13,>=12
Requires-Dist: pydantic<3,>=2.10
Provides-Extra: bigquery
Requires-Dist: ibis-framework[bigquery]<13,>=12; extra == 'bigquery'
Description-Content-Type: text/markdown

# SegmentStream pipeline SDK

Declare source tables with `s.ingestion` and warehouse SQL transformations with
`s.model`. The SDK resolves inputs, validates explicit partitioning and capabilities,
and compiles native Dagster definitions using the existing warehouse loader.

```python
from collections.abc import Iterator

import ibis.expr.types as ir
import segmentstream as s


@s.ingestion(
    tier=s.Tier.Bronze,
    partitioning=s.Partitioning.Unpartitioned,
    schema={"amount": "float64"},
)
def orders() -> Iterator[s.IngestionRow]:
    yield {"amount": 25.0}


@s.model(
    tier=s.Tier.Gold,
    partitioning=s.Partitioning.Unpartitioned,
    schema={"total_revenue": "float64"},
)
def revenue(orders: ir.Table) -> ir.Table:
    return orders.aggregate(total_revenue=orders.amount.sum())


defs = s.definitions(
    assets=[orders, revenue],
    jobs=[s.job("revenue", assets=[revenue], include_upstream=True)],
)
```

The [workspace documentation](../../docs/README.md) is organized by concept:
[connections](../../docs/connections.md), [ingestion](../../docs/ingestion.md),
[models](../../docs/models.md), [partitioning](../../docs/partitioning.md),
[jobs](../../docs/jobs.md), [execution](../../docs/execution.md), [checks](../../docs/checks.md),
[runtime values](../../docs/runtime.md), and [warehouse storage](../../docs/warehouse.md).
The CLI bundles these same pages into every starter workspace.

`s.validate(defs)` checks the graph and named job configuration without executing
assets. `s.describe(defs)` explains resolved inputs, per-asset policies, storage,
capabilities, same-day input mappings, and job selections without resolving
credentials. See the
[modular commerce starter](../../templates/workspace-app/pipeline/README.md)
for daily row ingestion, batched models, and a partitioned check.

Daily inputs may begin earlier or end later than their consumers when they use
the same timezone and day boundaries. The upstream must cover every downstream
date. The SDK supplies strict same-day mappings for inputs and ordering-only
dependencies; no native Dagster decorator or mapping option is needed. Named
jobs still select one daily calendar, with other calendars left as inputs.

## Identity-resolution support (0.1.0a39)

Models can declare `s.AllPartitions(asset)` inputs for retained-history rollups.
`WarehouseResource.materialize_scratch` and `iterate` execute bounded fixed-point
algorithms in warehouse SQL and retain their results until output publication.
The SDK owns scratch cleanup; each algorithm owns its convergence and key invariants.

## Compatibility

The SDK contains authoring, graph inspection, Ibis resources, connection HTTP
clients, and managed secret access. Managed orchestration, workers, PostgreSQL setup, and the metadata
webserver live in the internal [pipeline runtime](../pipeline-runtime/README.md).

In `0.1.0a28`, use `s.get_connection(key).get_http_client()` for provider requests.
The client forwards requests through the backend, where provider credentials and
refresh remain. The access-token and full-credential accessors have been removed.
See [connections](../../docs/connections.md) for declaring API destinations.

Workspace pipelines declare `segmentstream-pipeline[bigquery]`; our image supplies
the runtime. The old SDK `cloud-run` and `webserver` extras and
`segmentstream.runtime` module are removed in a22.

`segmentstream.dagster` retains its native exports. Native assets and high-level
declarations can coexist in `s.definitions`. Root `s.resource` and
`s.IngestionResource` remain available for legacy ingestion. Import their helpers
from `segmentstream`; the old public `segmentstream.ingestion` module is retired
so `s.ingestion` can be the decorator. [Migration examples](../../docs/native-dagster.md)
cover the mapping and IO defaults.

## Managed Google clients

The Sandbox runtime integration supplies refreshable credentials to the built-in
BigQuery warehouse and GCS IO manager. Ibis remains the user-facing warehouse API;
its internal official Google client renews credentials automatically throughout
the authorized run. Ten minutes is the lifetime of each access token, not a limit
on pipeline duration. Managed execution does not provide general ADC support.
See [runtime lifecycle and credential brokering](../../docs/sandbox-runtime.md).
Managed Dagster execution uses the backend compute provider API for every step,
including SQL assets and checks. The coordinator owns scheduling and retries; the
backend owns execution identity, scoped credentials and cleanup. See the
[compute provider contract](../../docs/provider-compute.md).

## Releases

SDK `0.1.0a39` adds explicit all-partition model inputs and managed warehouse
iteration with temporary SQL tables, supporting full-history identity resolution.

SDK `0.1.0a38` adds compatible daily calendars with strict same-day dependency
mappings. Consumers can start later than their upstream assets; each job still
selects one daily calendar.

The GA4 query client targets `0.1.0a37`. A model constructs it with
`s.ga4_bigquery_client(key, context=context)` and submits SQL through the backend;
BigQuery writes the result directly to the model's warehouse partition. See
[GA4 BigQuery connections](../../docs/bigquery-connections.md).
Publish the SDK before releasing a CLI whose starter
pins it. PyPI versions are immutable; do not republish an existing version.

The protected `pipeline-sdk-release.yml` workflow builds a wheel and source
distribution from a `pipeline-sdk-v<version>` tag, then uses PyPI Trusted Publishing
from the `pypi` GitHub environment. No long-lived publishing token is stored in GitHub.

## Unreleased: full-refresh storage layout

Models with `s.Partitioning.Unpartitioned` may set
`storage=s.TableStorage(partition_by_date="session_date")`. They run as global
steps and atomically replace the whole table while retaining physical BigQuery
DATE partitions. Daily models keep selected-date replacement. `s.describe`
reports the storage layout independently of execution partitioning. See
[the model contract](../../docs/models.md#full-refresh-tables-with-physical-date-partitions).
Available in SDK `0.1.0a40`; update the workspace dependency and lockfile before using it.

## Semantic models (0.1.0a41)

`segmentstream.semantics` defines portable dimensions, measures, and expressions
over declared warehouse assets. Register models with `s.definitions(semantic_models=...)`.
Inspection publishes manifest version 8 without adding jobs or materializations.
See [semantic models](../../docs/semantic-models.md). Upgrade backend readers and
the internal runtime before deploying definitions from this SDK.

## HubSpot connection (0.1.0a42)

`HubSpotConnection(key="hubspot")` declares read-only access to one HubSpot
account's CRM through SegmentStream's HubSpot app. Deploy, authorize, and use
`s.get_connection("hubspot").get_http_client()`; the default `objects` API reads
contacts, companies and deals. See [connections](../../docs/connections.md#hubspot).
Upgrade the backend before deploying declarations from this SDK.
