Metadata-Version: 2.4
Name: ds-common-placeholder-py-lib
Version: 0.1.0a7
Summary: Parse, validate, and resolve standard-dialect @{...} placeholders
Author-email: Kristoffer Varslott <hello@grasplabs.com>
Maintainer-email: Kristoffer Varslott <hello@grasplabs.com>
License-Expression: Apache-2.0
Project-URL: Documentation, https://grasp-labs.github.io/ds-common-placeholder-py-lib/
Project-URL: Repository, https://github.com/grasp-labs/ds-common-placeholder-py-lib/
Project-URL: Issues, https://github.com/grasp-labs/ds-common-placeholder-py-lib/issues/
Project-URL: Changelog, https://github.com/grasp-labs/ds-common-placeholder-py-lib/releases
Keywords: ds,placeholder,python
Classifier: Development Status :: 3 - Alpha
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Operating System :: OS Independent
Classifier: Intended Audience :: Developers
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.12
Description-Content-Type: text/markdown
License-File: LICENSE-APACHE
Requires-Dist: ds-vault-py-sdk[postgres]<1.1,>=1.0.0b1
Requires-Dist: tzdata>=2024.1
Provides-Extra: dev
Requires-Dist: bandit>=1.9.3; extra == "dev"
Requires-Dist: ruff>=0.1.8; extra == "dev"
Requires-Dist: mypy>=1.7.0; extra == "dev"
Requires-Dist: pytest>=7.4.0; extra == "dev"
Requires-Dist: pytest-cov>=4.1.0; extra == "dev"
Requires-Dist: pytest-xdist>=3.3.0; extra == "dev"
Requires-Dist: duckdb<2,>=1.2; extra == "dev"
Requires-Dist: pandas>=2.0; extra == "dev"
Requires-Dist: pre-commit>=3.6.0; extra == "dev"
Requires-Dist: build>=1.0.0; extra == "dev"
Requires-Dist: twine>=4.0.0; extra == "dev"
Requires-Dist: sphinx>=7.1.0; extra == "dev"
Requires-Dist: sphinx-autoapi>=3.0.0; extra == "dev"
Requires-Dist: sphinx-material>=0.0.35; extra == "dev"
Requires-Dist: myst-parser>=3.0.0; extra == "dev"
Provides-Extra: test
Requires-Dist: pytest>=7.4.0; extra == "test"
Requires-Dist: pytest-cov>=4.1.0; extra == "test"
Requires-Dist: pytest-xdist>=3.3.0; extra == "test"
Requires-Dist: duckdb<2,>=1.2; extra == "test"
Requires-Dist: pandas>=2.0; extra == "test"
Provides-Extra: docs
Requires-Dist: sphinx>=7.1.0; extra == "docs"
Requires-Dist: sphinx-autoapi>=3.0.0; extra == "docs"
Requires-Dist: sphinx-material>=0.0.35; extra == "docs"
Requires-Dist: myst-parser>=3.0.0; extra == "docs"
Dynamic: license-file

# ds-common-placeholder-py-lib

[![Docs](https://img.shields.io/badge/docs-github.io-blue)](https://grasp-labs.github.io/ds-common-placeholder-py-lib/)

The Python implementation of the `standard` placeholder dialect: parse,
validate, resolve, and scrub `@{…}` placeholders in computing-block
definitions.

The behaviour is defined by the
[placeholder contract](https://github.com/grasp-labs/ds-pipeline-infra/blob/main/docs/placeholder-contract.md)
(§ below). The calling service supplies the namespace values as plain JSON;
the package reads secrets from ds-vault itself. Nothing resolved outlives
the run: each run of a computing block builds its own context, and the vault
client keeps no secret material between reads.

**Documentation:** <https://grasp-labs.github.io/ds-common-placeholder-py-lib/>,
published on each release (source in `docs/source/`, build locally with
`make docs`).

## Install

```bash
uv add ds-common-placeholder-py-lib
```

Python 3.12+. Dependencies: `ds-vault-py-sdk` (secrets) and `tzdata`, so
IANA time zones work on
images without system zone data.

## Use

The package knows the `standard` dialect, not the computing blocks. A service
passes the namespace values it has as plain JSON; a namespace it does not pass
does not exist.

```python
from ds_common_placeholder_py_lib import Context, PlaceholderError, validate

# 1. Static check of the whole definition: syntax and dialect.
errors = validate(definition)

# 2. One context per run; ctx.set(name, value) adds to it as the run goes.
ctx = Context.from_values({
    "tenant_id": tenant_id,  # required
    "run": run, "owner": owner,
})
ctx.set("variables", ctx.resolve(declared_variables))  # resolved once, at the start
ctx.set("execution", execution)

# 3. Resolve each part right before using it.
try:
    settings = ctx.resolve(dataset_settings)
    sql = ctx.resolve_sql(sql_query)
except PlaceholderError as err:
    fail_execution(code=err.code, message=str(err))  # the message holds no values

# 4. Scrub anything that leaves the process.
log.error(ctx.scrub(str(some_exception)))
```

`get_secret` reads ds-vault through `ds-vault-py-sdk`, configured like the
dispatchers: `VAULT_DATABASE_URL` and AWS credentials for KMS. In tests,
`ds_common_placeholder_py_lib.testing.fake_vault` stands in for it.

A worker that cannot reach the vault gets its secrets from its dispatcher,
through `ctx.resolve_secrets(job)`
([details](https://grasp-labs.github.io/ds-common-placeholder-py-lib/secrets.html#workers-without-vault-access)).

## Examples

`examples/` in the repository has one script per computing block. Each opens with a real
definition and shows, as service code, which namespaces that block supplies:

| Script | Block | Shows |
| --- | --- | --- |
| `01_pipeline.py` | Owner-input pipeline | The stage-request mapping (§9.1), bronze secrets, silver SQL in DuckDB |
| `02_workflow.py` | Workflow | A legacy definition, a dry-run migration report, and a `standard` run where tasks read earlier outputs by task id |
| `03_clone.py` | Clone | Opting in, `get_secret` resolved by the clone service, tenant scope |

## Rules

| Topic | Rule |
| --- | --- |
| Whole vs embedded | `"@{x}"` keeps the JSON type; `"a-@{x}"` renders text. Objects and arrays can't be embedded. |
| Missing | `null`, `""`, blank, or no such path → `missing_value`. A default only with `@{coalesce, …}`. |
| `@{` as text | `@{'@{'}` → `@{`; `escape(text)` writes any string so. Resolved values are never scanned again. |
| Secrets | The key's tenant segment must be the context's tenant. Secrets never go into SQL. |
| Protected | No placeholders in `id`, `name`, `type`, `version` (the default list) outside `settings`. |
| SQL | Values become SQL literals; placeholders inside quotes or comments are rejected. |

The full rules, every function, and the error codes are in the
[documentation](https://grasp-labs.github.io/ds-common-placeholder-py-lib/). Moving from the `legacy` dialect:
[migration notes](https://grasp-labs.github.io/ds-common-placeholder-py-lib/legacy.html).

## Conformance vectors

`src/ds_common_placeholder_py_lib/vectors/*.json` ship with the package and
are the conformance suite for every implementation (format in
`vectors/README.md`).
Any bug fix adds a vector first.

## Development

```bash
uv sync --all-extras --dev
make lint type-check test-cov
```
