Metadata-Version: 2.4
Name: composable-data-stack
Version: 0.15.1
Summary: CLI for validating and composing composable data stack profiles
Author: Ronald Hensbergen
License-Expression: Apache-2.0
Project-URL: Homepage, https://github.com/RonaldHensbergen/composable-data-stack
Project-URL: Documentation, https://github.com/RonaldHensbergen/composable-data-stack/tree/main/docs
Project-URL: Issues, https://github.com/RonaldHensbergen/composable-data-stack/issues
Project-URL: Repository, https://github.com/RonaldHensbergen/composable-data-stack
Keywords: data-platform,data-stack,docker-compose,infrastructure-as-code
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Software Development :: Build Tools
Classifier: Topic :: System :: Systems Administration
Requires-Python: >=3.14
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: PyYAML>=6.0
Requires-Dist: jsonschema>=4.22.0
Requires-Dist: packaging>=23.0
Requires-Dist: rich>=15.0
Provides-Extra: completion
Requires-Dist: argcomplete<4,>=3.0.0; extra == "completion"
Provides-Extra: test
Requires-Dist: jinja2>=3.0; extra == "test"
Requires-Dist: sqlalchemy>=2.0; extra == "test"
Provides-Extra: dev
Requires-Dist: coverage>=7.6; extra == "dev"
Requires-Dist: bandit>=1.7; extra == "dev"
Requires-Dist: pip-audit>=2.7; extra == "dev"
Requires-Dist: ruff==0.17.3; extra == "dev"
Dynamic: license-file

# 🚀 Composable Data Stack (CDS)

> **The modern data stack has a composition problem. CDS is an open framework
> for building data platforms that can evolve without being rebuilt.**

![Composable Data Stack logo](https://raw.githubusercontent.com/RonaldHensbergen/composable-data-stack/main/assets/branding/logo.svg)

Project:

[![CI](https://github.com/RonaldHensbergen/composable-data-stack/actions/workflows/ci.yml/badge.svg)](https://github.com/RonaldHensbergen/composable-data-stack/actions/workflows/ci.yml)
[![PyPI](https://img.shields.io/pypi/v/composable-data-stack.svg)](https://pypi.org/project/composable-data-stack/)
[![License](https://img.shields.io/badge/license-Apache--2.0-blue.svg)](LICENSE)

SonarCloud — Quality Gate & Ratings:

[![Quality Gate Status](https://sonarcloud.io/api/project_badges/measure?project=RonaldHensbergen_composable-data-stack&metric=alert_status)](https://sonarcloud.io/summary/new_code?id=RonaldHensbergen_composable-data-stack)
[![Reliability Rating](https://sonarcloud.io/api/project_badges/measure?project=RonaldHensbergen_composable-data-stack&metric=reliability_rating)](https://sonarcloud.io/summary/new_code?id=RonaldHensbergen_composable-data-stack)
[![Security Rating](https://sonarcloud.io/api/project_badges/measure?project=RonaldHensbergen_composable-data-stack&metric=security_rating)](https://sonarcloud.io/summary/new_code?id=RonaldHensbergen_composable-data-stack)
[![Maintainability Rating](https://sonarcloud.io/api/project_badges/measure?project=RonaldHensbergen_composable-data-stack&metric=sqale_rating)](https://sonarcloud.io/summary/new_code?id=RonaldHensbergen_composable-data-stack)

SonarCloud — Issues:

[![Bugs](https://sonarcloud.io/api/project_badges/measure?project=RonaldHensbergen_composable-data-stack&metric=bugs)](https://sonarcloud.io/summary/new_code?id=RonaldHensbergen_composable-data-stack)
[![Vulnerabilities](https://sonarcloud.io/api/project_badges/measure?project=RonaldHensbergen_composable-data-stack&metric=vulnerabilities)](https://sonarcloud.io/summary/new_code?id=RonaldHensbergen_composable-data-stack)
[![Code Smells](https://sonarcloud.io/api/project_badges/measure?project=RonaldHensbergen_composable-data-stack&metric=code_smells)](https://sonarcloud.io/summary/new_code?id=RonaldHensbergen_composable-data-stack)

SonarCloud — Metrics:

[![Coverage](https://sonarcloud.io/api/project_badges/measure?project=RonaldHensbergen_composable-data-stack&metric=coverage)](https://sonarcloud.io/summary/new_code?id=RonaldHensbergen_composable-data-stack)
[![Duplicated Lines (%)](https://sonarcloud.io/api/project_badges/measure?project=RonaldHensbergen_composable-data-stack&metric=duplicated_lines_density)](https://sonarcloud.io/summary/new_code?id=RonaldHensbergen_composable-data-stack)
[![Technical Debt](https://sonarcloud.io/api/project_badges/measure?project=RonaldHensbergen_composable-data-stack&metric=sqale_index)](https://sonarcloud.io/summary/new_code?id=RonaldHensbergen_composable-data-stack)

---

You start with a straightforward goal: orchestrate some pipelines, store the
data, and make it useful.

Then the tool decisions begin.

Airflow, Dagster, or Prefect? Iceberg, Delta, or Hudi? Superset or Metabase?
Every choice opens another comparison, another integration, and another
architecture diagram. Every diagram looks different. Every tool has its own
configuration model, assumptions, and lifecycle. Before long, the platform is
less a designed system than a collection of decisions held together by custom
code and institutional memory.

The hardest question is no longer **"Which tool is best?"** It is **"How do
these tools fit together, and can we change our minds later?"**

## Why Data Platforms Have a Composition Problem

The modern data ecosystem is excellent at producing capable tools and terrible
at making them work as interchangeable parts.

Teams are expected to assemble orchestration, storage, transformation,
cataloging, observability, security, and BI into one dependable platform. But
the boundaries between those tools are usually implicit. Connection details
leak into environment variables. Service names become dependencies. Deployment
logic assumes a specific vendor. Compatibility lives in a staff engineer's
head—or is discovered during an incident.

That fragmentation creates a familiar cycle:

- evaluate an overwhelming number of tools
- debate architecture instead of delivering data products
- hand-build integrations that only one team understands
- duplicate stacks across development, CI, and production
- postpone upgrades because nobody knows what they will break
- accept lock-in because replacing one component feels like replacing the
  platform

This is not primarily a tooling problem. **It is a composition problem.** We
have many good components, but no shared way to declare what they provide,
what they need, and whether a particular assembly is valid.

## The Cost of Choosing Wrong

A technology choice that looks local rarely stays local.

Choose an orchestrator and its concepts spread into deployment, secrets,
logging, and team workflows. Choose a table format and it constrains engines,
catalogs, and migration paths. Choose a BI layer and dashboards become coupled
to authentication, database behavior, and operational assumptions.

The bill arrives later:

- migrations become multi-quarter programs
- upgrades become risky, manual compatibility exercises
- platform teams maintain glue code instead of platform capabilities
- product teams wait while engineers renegotiate architecture
- vendors gain leverage because the cost of leaving keeps rising

There is no universally correct stack, and pretending otherwise is expensive.
The right choice today may be the wrong choice at a different scale, under a
new regulatory constraint, or after the team learns more. A healthy platform
must make change routine—not heroic.

## What If Data Platforms Were Composable?

What if a data platform described capabilities and interfaces before naming
products?

What if an orchestrator could request a database contract without hardcoding a
specific database? What if a profile could declare a supported combination,
validate it before deployment, and produce the runtime configuration? What if
replacing a component meant satisfying the same contract instead of rebuilding
every integration around it?

That is the goal of **Composable Data Stack (CDS)**.

CDS is an open-source framework for defining, validating, securing, and
assembling data platforms from reusable components. Think of it as
**Infrastructure as Code for the shape of a data platform**: the components,
the interfaces between them, and the rules that make the whole stack coherent.

CDS introduces three building blocks:

- 🔧 **Modules** package reusable capabilities such as orchestration, storage,
  BI, and secret management.
- 🔗 **Contracts** define explicit, testable interfaces between those
  capabilities.
- 🧩 **Profiles** compose modules into supported, reproducible stacks.

Instead of burying platform architecture in scripts and tribal knowledge, CDS
makes it declarative. It answers:

- **How do these tools fit together?** Through explicit contracts and declared
  bindings.
- **What happens if I replace a component later?** CDS validates the new
  composition against the same requirements before runtime.
- **How do I avoid vendor and technology lock-in?** Consumers depend on
  capabilities, not hardcoded product identities.

The result is composability without chaos, flexibility with guarantees, and a
platform that can evolve without starting over.

## ⚡ Why CDS

Most teams are forced into one of two bad options:

|Approach|What You Gain|What It Costs|
|---|---|---|
|Monolithic stack|A consistent starting point|Rigid boundaries and an expensive exit|
|Custom integrations|Freedom to choose every component|Fragile glue, inconsistent stacks, and hidden coupling|
|CDS|Explicit composition and replaceable components|A contract and profile model the community maintains together|

CDS does not claim that every tool is interchangeable or that integration
complexity disappears. It makes compatibility explicit, validates known
constraints early, and gives the ecosystem a shared place to improve how tools
compose.

---

## 🎯 When To Use CDS

Use CDS if you:

- expect your architecture to evolve as requirements change
- want to swap tools (Airflow ↔ Dagster, Superset ↔ Metabase)
- need reproducible environments across dev, CI, and prod
- are building a platform for multiple teams
- want contract-driven integration instead of implicit coupling
- want to contribute to an open interoperability model for data tooling

CDS may be overkill if:

- you only run a single-tool stack
- you do not need interchangeable components

---

## 🏗️ See Composition in Practice

The `local-dagster-postgres-superset` profile defines:

- Dagster -> orchestration
- Postgres -> storage
- Superset -> BI

CDS then:

1. Validates module definitions
2. Resolves contract bindings
3. Checks compatibility and security constraints
4. Produces a fully wired stack definition

`cds plan` resolves the full dependency graph before runtime configuration is generated, ensuring all module interactions are valid and predictable.

The composition model creates a path to replace components while preserving
the contracts the rest of the platform depends on:

```text
Dagster -> Airflow
Superset -> Metabase
Postgres -> MariaDB
```

---

## 🗺️ Architecture Overview

CDS wires modules through **contracts**, not direct dependencies. This section has two levels: a high-level picture of what gets wired together ([Overview](#overview)), and a detailed look at what happens when you run a CDS command ([Internal Flow](#internal-flow)).

### Overview

Below, `local-dagster-postgres-superset` wires Dagster to Postgres to Superset through contracts:

![Dagster, Postgres, and Superset connected through contracts](https://raw.githubusercontent.com/RonaldHensbergen/composable-data-stack/main/docs/diagrams/architecture/architecture_overview.svg)

### Internal Flow

CDS splits into two phases: **compile-time**, where `cds` itself validates, resolves, and renders a plain `docker-compose.yaml`; and **runtime**, where the real `docker compose` binary builds and starts containers from that file. CDS never runs containers itself.

`cds test` runs the full compile-time pipeline in order — **validate → security → plan → render**. A `validate` (or `plan`) failure skips all downstream stages, but a **security** failure does not skip `plan`/`render`: those stages still run so `cds test` reports every stage's status in one pass, while the overall command still exits non-zero if any stage failed. `cds up` runs the same pipeline **minus security** (`validate → plan → render`), then hands off to `docker compose build`/`docker compose up`. See the [CLI table](#️-cli) below for exactly what each command runs.

- **Validate** checks profile shape, module configs, dependencies, secret refs, contract bindings, and outputs.
- **Security** (`cds test` only) runs rule-based checks against modules and resolved secrets; a failing check is reported but does not prevent `plan`/`render` from also running.
- **Plan** resolves contract bindings and substitutes secrets and defaults.
- **Render** generates the final `docker-compose.yaml`, with secret values as `${CDS_VAR}` placeholders; never the raw value.
- **Runtime** (`cds up` only): `docker compose build` (skippable with `--no-build`), then `docker compose up --detach`, then a live `cds state` view that polls until the stack settles (skippable with `--detach`). Build/up/log output is persisted to a log file (`.cds/logs/` by default). Docker Compose, not CDS, resolves `${CDS_VAR}` placeholders from a `.env` file (see `cds init`) and starts the containers.

Passing `validate → security → plan → render` proves the profile *compiles*
correctly — it is not proof that the resulting stack *runs* correctly. For the
broader success criteria used to certify a profile (booting the rendered
stack, service health, an end-to-end pipeline run, and downstream data
consumption in Superset), see
[docs/profile-testing/test-plan.md](docs/profile-testing/test-plan.md) and
[docs/profile-testing/failure-path-and-ci.md](docs/profile-testing/failure-path-and-ci.md).

![CDS compile-time and runtime flow](https://raw.githubusercontent.com/RonaldHensbergen/composable-data-stack/main/docs/diagrams/processes/internal_flow.svg)

This mirrors the [`cds` command table](#️-cli) below: `validate`, `plan`, and `render` are each callable on their own; `security` only runs as part of `cds test`, not `cds up`. Module and contract definitions follow the [Contract-First](#contract-first) design principle, so most of what "Validate" and "Plan" check comes directly from `module.yaml` and `profile.yaml`.

**See also:** [Security](#-security) for what the security stage checks, [Troubleshooting](#️-troubleshooting) for what each error code means and how to fix it, and [docs/architecture.md § Secrets and contract resolution](docs/architecture.md#secrets-and-contract-resolution) for a deeper look at how `secrets.*` and `contractRef` values actually resolve.

## 🔐 Security

CDS includes built-in security validation to prevent unsafe configurations before a stack is deployed.

The `cds security` checks analyze profiles and modules for common risks such as:

- weak or default passwords
- missing secret configurations
- insecure service exposure
- unsafe defaults in module configuration
- incomplete contract bindings that may leak data

Security checks run as part of validation and can be extended with custom rules.

### Example

```bash
cds security local-dagster-postgres-superset
```

---

## 📦 What You Get

When a profile compiles, CDS produces:

- validated module graph
- resolved contract bindings
- dependency-aware execution plan
- generated Docker Compose configuration
- generated Helm chart for Kubernetes
- reproducible stack definition

This turns a declarative platform design into a reproducible, runnable local
data stack—and turns architectural assumptions into things the project can
inspect, validate, and improve.

---

## ✅ Prerequisites

CDS has separate compile-time and runtime requirements.

### Validate, Plan, And Render

The commands `cds validate`, `cds security`, `cds plan`, and `cds render`
do not require Docker. Install:

- Python 3.14 or newer, including `pip` and `venv`
- Git when cloning and installing CDS from source
- Internet access for the initial source and Python dependency downloads

### Build And Run A Stack

The `cds up` command and generated local profiles additionally require:

- Docker Engine (Linux) or Docker Desktop (macOS and Windows)
- Docker Compose v2, invoked as `docker compose`
- A running Docker daemon accessible to the current user
- Internet access for the initial container image pulls and builds, unless all
  required artifacts are already cached
- Free host ports required by the selected profile
- Write access to the checkout for `.env`, `docker-compose.yml`, and workdir
  data

For Kubernetes, install Helm and kubectl and provide an explicit kube context.
The local k3s workflow uses k3d and is documented in
[docs/kubernetes.md](docs/kubernetes.md).

Docker Desktop on Windows must use the WSL 2 backend. See the
[support policy](docs/support-policy.md) for supported operating systems and
runtime versions. For platform-specific setup from an empty machine, follow
the [complete installation guide](docs/installation.md).

For the complete example stack, allocate at least 8 GB of memory to Docker and
keep at least 10 GB of disk space free for images, build layers, volumes, and
logs. Larger workloads require additional resources.

### Preflight Check

Linux/macOS:

```bash
python3 --version
git --version
docker --version
docker compose version
docker info >/dev/null
```

Windows PowerShell:

```powershell
py --version
git --version
docker --version
docker compose version
docker info | Out-Null
```

Python must report version 3.14 or newer, Docker Compose must report v2, and
`docker info` must complete without a daemon or permission error. Git is
optional when CDS is installed from a package rather than source.

Tools such as `make`, Node.js, and `pre-commit` are contributor tooling and are
not required to run CDS profiles.

## 🚀 Quickstart

### CLI package preview

The Python distribution contains the `cds` command and its built-in security
rules. After production PyPI publishing is enabled, install the isolated CLI
with:

```bash
pipx install composable-data-stack
```

The package does not yet download profiles, modules, image build contexts, or
runtime workdirs. Continue with a repository checkout for the reference stack,
or configure external project assets with `CDS_PROFILE_PATH` and
`CDS_MODULE_PATH`. See [Packaging](docs/packaging.md) for the current TestPyPI
flow.

### 1. Clone

```bash
git clone https://github.com/RonaldHensbergen/composable-data-stack.git
cd composable-data-stack
```

### 2. Setup Environment

Linux/macOS:

```bash
python3 -m venv .venv
source .venv/bin/activate
pip install -e .
```

Windows PowerShell:

```powershell
py -m venv .venv
.\.venv\Scripts\Activate.ps1
python -m pip install -e .
```

Windows CMD:

```bat
py -m venv .venv
.venv\Scripts\activate.bat
python -m pip install -e .
```

If PowerShell blocks the activation script, run
`Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass` in the same
terminal session and activate the environment again.

### 3. Configure Environment

```bash
cds init local-dagster-postgres-superset
```

Set:

```text
CDS_ANALYTICS_DB_NAME
CDS_ANALYTICS_DB_USER
CDS_DAGSTER_DB_NAME
CDS_DAGSTER_DB_USER
CDS_SUPERSET_DB_NAME
CDS_SUPERSET_DB_USER
CDS_POSTGRES_SUPERUSER_PASSWORD
CDS_ANALYTICS_DB_PASSWORD
CDS_DAGSTER_DB_PASSWORD
CDS_SUPERSET_DB_PASSWORD
CDS_SUPERSET_SECRET_KEY
CDS_SUPERSET_ADMIN_PASSWORD
```

### 4. Validate A Stack

```bash
cds validate local-dagster-postgres-superset
```

Expected output:

```text
Profile is valid.
```

### 5. Check Runtime Prerequisites

```bash
cds preflight local-dagster-postgres-superset
```

This checks the runtime CLI and daemon, Compose support, required environment
values, and declared host ports without starting services. `cds init` only
creates configuration, while `cds validate` checks profile structure.

### 6. Run Security Checks

```bash
cds security local-dagster-postgres-superset
```

### 7. Generate A Plan

```bash
cds plan local-dagster-postgres-superset
```

This resolves:

- module dependencies
- contract bindings
- execution order

### 8. Render The Stack

```bash
cds render local-dagster-postgres-superset
```

By default, this writes `docker-compose.yml` to the project root.

Render the same plan as a Helm chart with:

```bash
cds render local-dagster-postgres-superset --target helm
```

Use a custom location when needed:

```bash
cds render local-dagster-postgres-superset --output build/docker-compose.yml
```

This generates:

- docker-compose.yml
- service definitions
- fully wired module configuration

### 9. Run The Stack

```bash
cds up local-dagster-postgres-superset
```

This runs `validate` → `plan` → `render` → `docker compose build` → `docker compose up` in one step.

`docker compose build` output is streamed to the terminal (grouped by image) and also persisted to a log file (default: `.cds/logs/up-<profile>-<timestamp>.log`; override with `--log-file`). `docker compose up` always runs detached: in the default mode its output goes only to the log file while the terminal shows the live state view; with `--detach` it is streamed to the terminal like the build output. Once the stack starts, `cds up` polls `docker compose ps` and redraws the same grouped/colored view as `cds state` until every service is healthy/running/exited or a timeout is hit (default 180s; override with `--timeout`; disable colored labels with `--no-color`). Container logs from that point on go only to the log file, not the terminal, since the terminal is showing the live state view. `cds up` exits `0` once the stack settles, or `1` on timeout or an unhealthy service.

The rendered `docker-compose.yml` is a standard Compose file, so the stack also runs without `cds` — `docker compose -f docker-compose.yml up -d` works on its own from the project root. See [docs/underlying-commands.md](docs/underlying-commands.md) for the full mapping of every `cds` lifecycle command (`render` / `up` / `state` / `down`, Compose and Helm) to its underlying `docker compose` / `helm` / `kubectl` commands.

Add `--detach` (or `-d`) to skip the live state view and return as soon as the stack starts, e.g. for scripting or CI:

```bash
cds up local-dagster-postgres-superset --detach
```

Use `--no-build` to skip the build step when images are already available:

```bash
cds up local-dagster-postgres-superset --no-build
```

### 9. Persistent Incoming Data Folder For Dagster

The Dagster module mounts a host directory into the containers so incoming files survive reboots.

- Host path: `workdirs/shared-data/incoming`
- Container path: `/app/data/cds/incoming`

Dagster includes a sensor that detects new files in `/app/data/cds/incoming` and runs a pickup job.
Picked files are moved to:

- Host path: `workdirs/shared-data/processed`
- Container path: `/app/data/cds/processed`

Create the directories once if they do not exist:

```bash
mkdir -p workdirs/shared-data/incoming workdirs/shared-data/processed
```

---

## 🧩 Core Concepts

### Modules

Reusable building blocks:

- orchestration (Dagster, Airflow)
- warehouse (Postgres, MariaDB)
- BI (Superset, Metabase)
- secrets (env, vault)

Structure:

```text
modules/<category>/<name>/
├── module.yaml
├── defaults.yaml
├── compose.yaml
├── scripts/
└── tests/
```

### Contracts

Contracts define how modules interact.

Examples:

|Contract|Purpose|
|---|---|
|sql-database|database interface|
|http-service|service exposure|
|secrets-provider|secret resolution|

Example binding:

```text
dagster.database -> postgres.sql-database
superset.database -> postgres.sql-database
```

No implicit dependencies. Everything is explicit.

### Profiles

Profiles define supported stacks:

```text
local-dagster-postgres-superset
local-airflow-postgres-superset
integration-airflow-postgres-dbt
```

Structure:

```text
profiles/[profile]/
├── profile.yaml
├── values.yaml
└── README.md
```

### Environment Overlays

A profile can be promoted from local development through staging to
production without duplicating its configuration. Add an `environments/`
directory next to `profile.yaml` containing one YAML file per environment;
each file is merged over the base profile when `--environment <name>` is
passed to a profile-consuming command:

```text
profiles/[profile]/
├── profile.yaml
└── environments/
    ├── dev.yaml
    └── prod.yaml
```

An overlay only needs to declare the values that differ from the base
profile. Module entries are merged by their stable `id` (not array
position), so an overlay can override just one field of one module:

```yaml
# profiles/[profile]/environments/prod.yaml
metadata:
  environment: production   # promotes the profile's security classification

spec:
  modules:
    - id: postgres
      config:
        storage:
          size: 20Gi
```

```bash
cds validate my-profile --environment prod
cds plan my-profile --environment prod
cds test my-profile --environment prod
cds up my-profile --environment prod
```

`local-dagster-postgres-superset` ships a working `dev`/`prod` example under
`profiles/local-dagster-postgres-superset/environments/` you can inspect or
copy.

Setting `metadata.environment` in an overlay also changes which security
policy `cds security`/`cds test` applies (see [Security](#-security)) —
promoting to `production` enables stricter checks, so a profile that passes
locally may report new findings once resolved with `--environment prod`.
That's expected: it surfaces settings that are fine for local development but
unsafe to carry into production.

Compare what an overlay actually changes with `cds diff`, without ever
printing a secret value (profiles only ever hold secret *references*, never
resolved values):

```bash
cds diff my-profile --from dev --to prod
```

```text
Differences from 'dev' to 'prod':

  ~ metadata.environment: "development" -> "production"
  ~ spec.modules[postgres].config.storage.size: "2Gi" -> "20Gi"
```

Persist an environment default for a project with `cds config set environment
prod`; commands without `--environment` then use that overlay. An explicit
`--environment` always takes precedence.

---

### Profile Composition (`extends`)

A profile can also factor out shared configuration into one or more parent
profiles instead of duplicating it, using a top-level `extends` field:

```yaml
# profiles/analytics-prod/profile.yaml
apiVersion: cds/v1alpha1
kind: Profile
metadata:
  name: analytics-prod
  environment: production
extends:
  - analytics-base   # a bare name resolves to profiles/analytics-base/profile.yaml
spec:
  modules:
    - id: postgres
      config:
        storage:
          size: 20Gi
```

`extends` accepts a non-empty list of parent references, each either a bare
profile name (resolved under the profiles root) or a path relative to the
child profile's directory (e.g. `../shared/profile.yaml`). Parents are
resolved and merged left-to-right — later parents win over earlier ones —
and then the child profile's own document is merged on top of all parents.
Composition uses the exact same deep-merge/module-merge-by-id engine as
environment overlays: mappings merge recursively, `spec.modules` entries
merge by stable `id`, and any other array is replaced wholesale rather than
concatenated. Parent profiles may themselves use `extends` (chains are
resolved transitively).

`extends` is not a CLI flag — it's read directly from `profile.yaml`, so
every profile-consuming command resolves it automatically, with no new
syntax to learn:

```bash
cds validate analytics-prod
cds plan analytics-prod
cds up analytics-prod
```

A profile can extend more than one parent, which is merged in the order
listed (later entries win over earlier ones):

```yaml
extends:
  - networking-base
  - observability-base
```

`extends` and `--environment` compose together: parents are merged first,
then the child, then the selected environment overlay is applied on top of
that fully-composed result — so a shared base profile and environment
promotion can both be used without duplicating configuration in either
dimension.

A malformed `extends` chain fails validation/planning before anything else
runs, with a dedicated diagnostic code:

|Code|Meaning|
|---|---|
|E110|`extends` is missing, not a list, empty, or contains a non-string/empty entry|
|E111|A parent reference resolves outside the profiles root|
|E112|A referenced parent profile does not exist|
|E113|A cycle was detected in the `extends` chain|

---

## ⚙️ CLI

See [docs/cli-reference.md](docs/cli-reference.md) for the full command
table, flag reference, and `[profile]` resolution rules. The commands used in
the [Quickstart](#-quickstart) above are `cds init`, `cds validate`,
`cds preflight`, `cds security`, `cds plan`, `cds render`, and `cds up`; run
`cds --help` or `cds <command> --help` for inline usage at any time.

---

## 🪟 Windows Task Runner

Windows contributors without `make` can use `Makefile.ps1`, a PowerShell equivalent covering the core developer tasks:

```powershell
# Install in editable mode
.\Makefile.ps1 install

# Validate the default profile
.\Makefile.ps1 validate

# Validate a specific profile
.\Makefile.ps1 validate-profile -P profiles/local-dagster-postgres-superset/profile.yaml

# Build distribution packages
.\Makefile.ps1 package

# List available targets
.\Makefile.ps1 help
```

This does not replace the Linux/macOS `Makefile`, both exist side by side. Windows users can still install `make` via WSL or Chocolatey if they prefer the original workflow. `lint` and `docker-build` are not ported here. Run `yamllint .` and `npx markdownlint-cli` directly, or use `pre-commit` if it is set up in this repo. Docker Desktop's `docker build` works the same on Windows as it does elsewhere.

---

## 🛠️ Troubleshooting

Common errors from `cds validate`, `cds plan`, and `cds render`, and how to fix them.

| Error | Cause | Fix |
| --- | --- | --- |
| `[E020] ... YAML file not found: <path>` | The profile identifier or file path passed to `cds validate`, `cds plan`, or `cds render <profile>` doesn't resolve to an existing YAML file. | Run `cds list profiles` to see valid identifiers. Set `CDS_PROFILE_PATH` to a profile name, a `profile.yaml` file path, or a profiles root directory. |
| `[E081] ... Required secret "CDS_X_PASSWORD" not found in environment` | A secret marked `required: true` in the profile's `spec.secrets.values` is missing from the shell environment or the `.env` file in the current working directory. | Run `cds init <profile>` to generate `.env` in the project root, set the missing `CDS_*` variable, or export it directly before running the command. |
| `[E041] ... Contract ref "x.y" points to unknown module "x"` | A `consumes` binding's `contractRef` refers to a module ID that isn't defined in the profile. | Check `spec.modules` for the correct module `id`, and confirm the contract ref follows `<module-id>.<contract-name>`. |
| `[E041] ... but it does not provide "<contract-name>"` | The referenced module exists, but its `spec.provides` list doesn't expose that contract name. | Check the producing module's `module.yaml` for the contracts it actually provides, and fix the consumer's `contractRef` to match. |
| `[E129] ... Certificate ref ... is not defined in spec.certificates.values` (or `malformed`) | A module config value references `certificates.<alias>.cert`/`.key`, but the alias isn't declared under `spec.certificates.values` -- or the reference is bare/has an unknown trailing part instead of `.cert` or `.key`. | Declare the alias under `spec.certificates.values` with its `certEnv`/`keyEnv` names, or fix the reference to name one half of a declared pair. CDS only maps names; the certificate/key material itself is supplied at runtime by the external secret/cert provider. |
| `[E042] ... Contract kind mismatch` | The consumer expects one contract kind (e.g. `sql-database`) but the producer exposes a different kind. | Point the binding at a module that provides the expected contract kind, or update the consumer's expected kind if the mismatch is intentional. |
| `[E043] ... recorded as unsupported in the compatibility registry` | The binding's `kind` matches, but this exact provider/consumer pairing is explicitly recorded as `unsupported` in `cli/resources/compatibility-registry.json` -- it's known NOT to work together despite matching shape. | Use a different provider/consumer pairing for this contract, or if the pairing has since been fixed/verified, update its `status` to `tested` in the registry (see [docs/architecture.md § Secrets and contract resolution](docs/architecture.md#secrets-and-contract-resolution)). |
| `[E128] ... requires a TLS-protected endpoint (tls: required), but ... does not advertise one` | A module's `consumes` entry declares `tls: required` (issue #577), but the bound provider's contract spec does not set `tls: true` (absent, `false`, or a `${config.<path>}` reference that does not resolve to `true` against the producer's profile config). | Point the binding at a provider that advertises TLS, or have the provider set `tls: true` in its provide entry's contract spec. Drop `tls: required` only if the binding is genuinely allowed to run unencrypted. |
| `[E154] ... expects database engine ... but contract ref ... advertises engine ...` / `... advertises no "engine" in its spec` | A module's `consumes` entry declares `engineFrom: spec.config.<path>` (issue #682), but the bound provider's `sql-database` contract advertises a different `engine`, or none at all -- the consumer would speak the wrong wire protocol to the bound database (e.g. Keycloak's `KC_DB` vs the bound provider). The expected engine is read from the profile config at that path, falling back to the `configSchema` default when unset. | Set the consumer's config field (e.g. keycloak `metadataDatabase.driver`) to the provider's advertised engine, bind a provider advertising the expected engine, or have the provider set `engine` in its provide entry's contract spec. Drop `engineFrom` only if the consumer genuinely accepts any engine. |

| `[E130] ... Duplicate backup policy for workload class "<class>"` | `spec.backup.policies` declares more than one policy for the same workload class. | Keep one policy per workload class. |
| `[E131] ... Backup encryption key ref ... is not defined in spec.secrets.values` | `spec.backup.encryption.keyFrom` references a `secrets.<alias>` that the profile does not declare. | Declare the alias under `spec.secrets.values` with its `env` name, or fix the reference. |
| `[E132] ... Backup target contractRef ...` | The pinned `spec.backup.target.contractRef` is malformed, points to an unknown module or contract, is not a `backup-target` provider, or (in production) the provider does not advertise `encrypted: true` / `supportsRestoreVerification: true`. | Point at a module providing a `backup-target` contract that advertises the required capabilities, or drop the pin to leave the storage provider open. |
| `[E133] ... Production profiles must declare spec.backup ...` / `... missing workload classes: ...` | A `metadata.environment: production` profile has no `spec.backup` block, or lacks a policy for a workload class it runs (warehouse, orchestration/BI/identity/secrets metadata, or configuration). | Declare encrypted, restore-verified backup policies for every workload class the profile runs; see `docs/backup-restore.md`. |
| `[E134] ... Production backups must be encrypted ...` / `... must be declared required: true` | A production profile has no `spec.backup.encryption.keyFrom`, or the referenced secret alias is declared `required: false` so the key can be missing at runtime. | Declare the encryption key alias with `required: true` so a missing key fails closed at startup. |
| `[E135] ... Production backup policy "<class>" must declare restoreVerification` | A production backup policy has no `restoreVerification` block. | Add `restoreVerification.frequencyDays` to every production backup policy.|
| `[E136] ... Reverse-proxy route ...` (malformed or unknown contractRef / no resolvable http-service endpoint / backend address not a rendered compose service / entrypoint not declared by the proxy / undeclared certificate alias / route with neither host nor pathPrefix / duplicate route rule) | A `config.routes` entry on a reverse-proxy provider has a `contractRef` that is not `<module-id>.<contract-name>` or names a module that is missing, or that module does not provide the named `http-service` contract with host/port (or a port outside 1-65535), advertises a backend address that matches no rendered compose service (multi-service modules must advertise `serviceName`), names an entrypoint the proxy's `config.entryPoints` does not declare, references an undeclared certificate alias, names neither `host` nor `pathPrefix`, or duplicates an earlier route's rule on the same proxy (#581, #883). | Fix the route's `contractRef`/`certificate`/`host`/`pathPrefix`/`entryPoints` (entrypoints must match the proxy's declared `config.entryPoints` keys), or advertise the serving compose service via `serviceName` in the target's http-service provide entry. |
| `[E150] ... Provided contract ... advertises network host ...` | A module's provided `http-service`, `sql-database`, `cache-service`, `object-storage`, `lineage-sink`, `log-sink`, `metrics-sink`, or `trace-sink` contract advertises a network host (the `host` field, the `endpoint`'s or `connectionUri`'s hostname part, or a `serviceName` mapped through the renderer's service-naming rule; a scheme-less `host:port` value contributes its host part, #1036) that matches no service the render produced for that module -- no compose service on the compose target, no rendered Kubernetes Service on the helm target -- so consumers would resolve a hostname nothing answers (#1010, #1029, #1035). | Compose: instantiate the module with an id matching its compose service name, advertise the serving service via a `serviceName` field in the provide entry, or build the endpoint from `${service.dns}` or `${service.services.<name>}`. Helm: render the serving workload with a `service` block (or as a `StatefulSet`, which gets a governing Service), advertise the serving workload via a `serviceName` field, or build the endpoint from `${service.dns}`. |
| `[E151] ... Raw manifest ... must be an object with apiVersion, kind, and metadata.name` / `... must declare non-empty string apiVersion, kind, and metadata.name` | A raw manifest under a module's `implementation.kubernetes.manifests` is malformed -- a plan loaded from a file bypasses the module schema, so the Helm renderer fails closed instead of shipping a broken manifest (issue #767). | Fix the manifest shape in the module (or the saved plan): every entry needs string `apiVersion`/`kind` and a `metadata.name`. |
| `[E103] ... config.image.tag is required ... when config.image.source is "registry"` | A module instance sets `config.image.source: registry` without also setting `config.image.tag`. | Set `config.image.tag` to a version, or switch back to `config.image.source: build`. |
| `[E139] ... without an explicit host authentication advertisement` / `... host authentication method "trust"` | A `metadata.environment: production` profile runs a `sql-database` provider whose contract does not advertise a host authentication method, or advertises `trust` (issue #206). | Configure the provider's authentication explicitly (e.g. postgres `config.authentication.hostAuth: scram-sha-256`); trust authentication is prohibited in production profiles. |
| `[E140] ... configures webServer.mode "development"` | A production profile sets an HTTP-serving module's `config.webServer.mode` to `development`, selecting its built-in development web server (issue #216). | Set `config.webServer.mode` to `production` (or remove the override) so the service runs behind its production process manager; see `docs/serving-modes.md`. |
| `[E137] ... accepts unauthenticated clients` | A production profile runs a `cache-service` provider that does not advertise `authentication: required` (no password credential configured). | Supply a cache password through a `secrets.<alias>` reference (e.g. keydb `config.passwordFrom`), declared `required: true` in `spec.secrets.values`. |
| `[E138] ... Credential field ... must reference a secret alias` / `... is empty` / `... uses a ${CDS_*} profile input` / `... contains a plaintext value` | A credential-named config field of a production database/cache provider is not a `secrets.<alias>` reference, or its alias is not declared `required: true` (issue #206). | Supply the credential through a secret reference declared `required: true` so a missing value fails closed at startup. |
| `[W105] ... legacy md5 host authentication` | A production `sql-database` provider advertises `md5`, which validates but is a deprecated password method. | Prefer `scram-sha-256` for production profiles. |
| `[E148] ... RBAC rule ... must be a mapping with non-empty "apiGroups", "resources", and "verbs" lists` / `... The "serviceAccount" block ... must be a mapping ...` | A module's `implementation.kubernetes.serviceAccount` block (or one of its `rbac` rules) is malformed -- a plan loaded from a file bypasses the module schema, so the Helm renderer fails closed instead of shipping a broken Role (#75). | Fix the `serviceAccount`/`rbac` shape in the module (or the saved plan). |
| `[E149] ... The "networkPolicy"/... block ... must be a mapping with optional "ingress"/"egress" lists ...` | A module's custom NetworkPolicy allow rules are malformed (plan-file bypass of the module schema) (#75). | Fix the `networkPolicy` shape; each rule must be a mapping. |
| `[E152] ... waitFor gate of workload "<module>/<workload>" names host "<host>", which the module's default-deny NetworkPolicy does not allow egress to ...` | On the Helm target, a `waitFor` gate names a host that is neither a Service of the same module emitted by this render (a workload disabled via `enabledFrom` gets no Service), the endpoint of a provider the module consumes, nor the target of one of the module's reverse-proxy routes, and no custom `networkPolicy.egress` ipBlock rule or declared `implementation.kubernetes.egress` entry covers the host:port, so the default-deny NetworkPolicy blocks the init container's connection and it would only hang until its timeout (#1031, #1033, #1023). | Bind the host's module through a `consumes` contract, route to it via `config.routes`, gate on a same-module `${k8s.service.*}` Service, declare the destination under `implementation.kubernetes.egress` (an IP/CIDR with the gate's port), or add a custom `networkPolicy.egress` ipBlock rule covering the host and the gate's port (a DNS-name host cannot be expressed in a NetworkPolicy and stays denied). |
| `[E153] ... External egress entry of module \"<module>\" at ... is invalid ...` | A declared external egress entry under `implementation.kubernetes.egress` (or `workloads.<name>.egress`) is malformed -- plan-file bypass of the module schema (#1023): exactly one of `host` (IP literal; DNS names are rejected) / `cidr` (network form, no host bits), `except` CIDRs inside the `cidr`, `port`/`ports` within 1-65535, `protocol` TCP/UDP/SCTP, and all `${config.*}` expressions must resolve. | Fix the entry shape in the module (or the saved plan); see `docs/kubernetes.md` ("Declaring external egress"). |
| `[W128] ... carries no numeric port and provider module ... advertises no service ports ...` | A contract binding carries no port evidence, so its generated NetworkPolicy egress rule allows all ports to the provider (#75). | Declare the port in the contract so the rule stays port-scoped. |
| `[W129] ... Reverse-proxy route ... cannot be derived as a NetworkPolicy edge ...` | On the Helm target, a `config.routes` entry of a reverse-proxy module has a `contractRef` naming an unknown module, a module without the named `http-service` provide, or a target whose http-service contract carries no usable port evidence (absent, or outside 1-65535, and the target advertises no service ports), so no port-scoped NetworkPolicy edge can be derived and the proxy's default-deny policy keeps the target unreachable (#1033, #883). | Fix the route's `contractRef` (`<module-id>.<contract-name>` naming an existing http-service provide) and declare a valid `port` in that contract, or add a custom `networkPolicy.egress` rule for the target. |
| `[W130] ... renders resources for operator "<name>" (CRD <crd>), which CDS does not install itself ...` | On the Helm target, a module declares an out-of-band operator under `implementation.kubernetes.operators` (e.g. the CloudNativePG operator of `modules-experimental/warehouse/postgres-operator`): CDS renders the CRs but never installs the operator, so the chart only reconciles once the operator is on the cluster (#768). The warning names the install command; the same note is in the chart's `NOTES.txt`. | Run the install command from the warning (e.g. `helm repo add cnpg https://cloudnative-pg.github.io/charts && helm upgrade --install cnpg --namespace cnpg-system --create-namespace cnpg/cloudnative-pg`), verify with `kubectl get crd clusters.postgresql.cnpg.io`, then re-run; see `docs/kubernetes.md` ("Operator-managed modules"). |
| `[W131] ... provides open-data-provider contract ... with classification ... which is outside the recommended DCAT-AP-NL data-theme vocabulary ...` | A module provides an `open-data-provider` contract whose `classification` is free text outside the recommended standard vocabulary (the DCAT-AP-NL data themes: AGRI, ECON, EDUC, ENER, ENVI, GEOJ, GOVE, HEAH, INTR, JUST, REGI, SOCI, TRAN, or the full EU data-theme authority URI), so the dataset is harder to compare and find across domains (issue #845). The value still renders; this is a warning only. | Set `classification` to a theme from the DCAT-AP-NL data-theme vocabulary (e.g. in the postgres module's `config.dataProvider.classification`); see `shared/contracts/open-data-provider.yaml` and `docs/fds-alignment.md`. |
| `[E155] ... needs operator "<name>", but its CRD "<crd>" is not present on the cluster ...` | `cds up --target helm` checked the cluster (`kubectl get crd <crd>`) and the operator a module depends on is confirmed absent, so `helm upgrade --install` would fail with "no matches for kind" (#768). | Install the operator with the command named in the error (e.g. `helm upgrade --install cnpg --namespace cnpg-system --create-namespace cnpg/cloudnative-pg` after `helm repo add cnpg https://cloudnative-pg.github.io/charts`), then re-run `cds up --target helm`. |
| `[E156] ... forEach marker of manifest ... resolved ... to nothing / to a <type>, but fan-out needs a list / produced the duplicate metadata.name ...` | A raw manifest under `implementation.kubernetes.manifests` declares `forEach: <expr>`, but the expression does not resolve to a list in the plan, or two fanned-out copies resolve to the same `metadata.name` (they would collide in the release namespace and in the chart's template file names) (issue #1170). | Point `forEach` at a list-valued config path (e.g. `config.services`), and reference `${item...}` in the manifest's `metadata.name` so every fanned-out copy is uniquely named. |
| `[E157] ... Duplicate datasetId ...` | `spec.dataSharing.datasets` declares the same `datasetId` more than once; each disclosed dataset must be declared once (issue #844). | Remove the duplicate entry, or give each entry a distinct `datasetId`. |
| `[E158] ... Dataset ... names module ... which is not an enabled module instance ...` | A `spec.dataSharing.datasets` entry's `module` names a module id that is not an enabled module instance in the profile (issue #844). | Point `module` at an existing, enabled module instance id from `spec.modules`, or drop the field. |
| `[E159] ... enrolls service "<name>" in its service-mesh config (config.services[<i>].name), but no module instance with that id exists in the profile ...` | At plan time, a module providing the `service-mesh` contract has a `config.services[].name` that matches no module instance id of the profile (a typo such as `warehose` for `warehouse`, or a disabled module) (issue #1179). The rendered per-service `AuthorizationPolicy` would select zero pods via the `cds.dev/module` label -- a silent no-op allow policy. | Fix the enrollment to name an existing module instance id (the error lists the planned ids), add the module with that id, or remove the entry from `config.services`; enable a disabled module you want to enroll. |
| `[E161] ... Unknown module id '<id>' ...` / `... addresses/traverses a non-scalar value ...` / `... matches a resolved secret value ...` | A persisted `cds config set module.<id>.config.<path> <value>` override in `.cds/config.json` or the shared `cds.shared.json` (issue #1220) names a module id that is not in the resolved profile, points at a dict/list instead of a scalar leaf, or tries to store a resolved secret value. The same code is emitted at load time when a hand-edited `config.json` breaks, so `validate`/`plan`/`render` fail loudly instead of silently ignoring the override. | Fix the id or path (only `module.<id>.config.<scalar leaf>` keys are accepted), remove the override with `cds config unset [--shared] module.<id>.config.<path>`, or store a `secrets.<alias>` reference instead of a secret value; see [docs/cli-reference.md](docs/cli-reference.md#local-module-config-overrides-moduleidconfigpath). |
| `[W132] ... enrolls service "<name>" in its service-mesh config, but the module renders no Kubernetes workload pods ...` | On the Helm target, a known enrollment names a module with no rendered workloads, such as a manifest-only module or one whose workloads are all disabled by `enabledFrom` (issue #1198). If enabled, the per-service `AuthorizationPolicy` would select zero pods via the `cds.dev/module` label. | Enroll a module that renders at least one Kubernetes workload, or enable one of its gated workloads; remove the enrollment if no pod workload is intended. |
| `[W097] ... config.image.tag is "latest" with config.image.source "registry"` | `config.image.tag: latest` under `source: registry` still validates, but drifts silently between deploys instead of pinning a reproducible version. | Pin an explicit tag from `publish-images.yml`'s output (or `tests/fixtures/signed-images.json`) instead of `latest`. |
| `[E127] ... Unknown confinement key ...` / `... must be 'default', 'unconfined' or 'file:<path>' ...` / `... Invalid SELinux label(s) ...` | A `security.runtime.confinement` block (or a module instance's confinement) is malformed: an unsupported key, an invalid `seccomp`/`apparmor`/`selinux` value, or a `file:` seccomp profile path that is missing or resolves outside the profile directory. | Use `seccomp: default`/`unconfined`/`file:<relative path inside the profile directory>`, `apparmor: runtime/default`/`unconfined`/`<host profile name>`, and `selinux: disable` or a `label: value` list; see `docs/runtime-confinement.md`. |
| `[W071] ... has kind "docker-compose" but no "compose" definition` | A module implementation declares the `docker-compose` kind without an `implementation.compose` block, so the renderer skips the module. | Add the `implementation.compose` block (services, volumes) to the module's `module.yaml`, or set a supported implementation `kind`. |
| `[W072] ... Container "<name>" in Kubernetes workload "<module>/<workload>" has no resource requests or limits` | Rendering the Helm target: a container or init container listed in a Kubernetes workload has no matching entry under the workload's `resources`. | Add a `resources.<container>` entry with `requests` and `limits` to the workload in the module's Kubernetes implementation (see `docs/kubernetes.md`). |
| `[W090] ... Invalid .env line format: "<line>" (expected KEY=VALUE)` | A line in the `.env` file is neither blank, a `#` comment, nor in `KEY=VALUE` form, so it is ignored. | Fix the line to `KEY=VALUE` form, comment it out with `#`, or remove it. |
| `[W098] ... Applied plaintext endpoint exposure waiver for production profile (modules: ...)` | A production profile's `spec.security.waivers.plaintextEndpointExposure` waived the CDS-SEC-074 plaintext-endpoint findings; CDS reports the waiver so it stays visible in security output. | Confirm the waiver is still justified and its `reason` is accurate; see `docs/security-rules.md` (Warnings). |
| `[W099] ... plaintextEndpointExposure is set but has no effect for profile class ...` | `spec.security.waivers.plaintextEndpointExposure` is declared on a non-prod profile, where CDS-SEC-074 and its waiver only apply to prod-class profiles. | Remove the waiver from the non-prod profile, or move it to the prod profile (overlay) it is meant for. |
| `[W103] ... Applied runtime policy waiver for production profile (checks: ...)` | A production profile's `spec.security.waivers.runtimePolicy` suppressed CDS-SEC-090 violations; CDS reports the waiver so it stays visible in security output. | Confirm the waiver is still justified and its `reason` is accurate; see `docs/security-rules.md` (Warnings). |
| `[W104] ... runtimePolicy is set but has no effect for profile class ...` | `spec.security.waivers.runtimePolicy` is declared on a non-prod profile, where CDS-SEC-090 and its waiver only apply to prod-class profiles. | Remove the waiver from the non-prod profile, or move it to the prod profile (overlay) it is meant for. |

All diagnostics print with their error code and YAML path (e.g. `spec.modules[1].config`), so search the profile file for that path to find the exact line to fix.

---

## 🔄 Workflow

```text
1. cds validate -> check module definitions
2. cds security -> detect unsafe configurations
3. cds plan -> resolve dependencies and bindings
4. cds render -> generate Docker Compose stack
5. cds up -> start services
6. cds test -> one-shot validate + security + plan + render smoke check
```

---

## 📂 Repository Structure

```text
.
├── cli/
├── modules/
│   ├── bi/
│   ├── orchestration/
│   ├── secrets/
│   └── warehouse/
├── profiles/
├── docs/
├── pyproject.toml
└── Makefile
```

---

## 🧱 Design Principles

### Contract-First

Modules declare:

- what they provide
- what they require
- configuration inputs
- health checks
- lifecycle hooks

### Profile-Driven

Profiles define supported stacks.
The profile is the unit of support, not individual modules.

### Zero Hidden Coupling

- no implicit environment variables
- no cross-module assumptions
- no shared mutable state

All interactions happen through explicit contracts.

### Security By Default

CDS validates configurations before runtime, ensuring that:

- weak credentials are detected early
- secrets are properly configured
- services are not unintentionally exposed

Security is part of platform composition, not an afterthought.

### One Model, Multiple Environments

The same composition model applies across:

- local development
- CI environments
- production

Only runtime packaging differs.

---

## 📊 Comparison

|Capability|Monolith|Custom pipelines|CDS|
|---|---|---|---|
|Swap components|❌|⚠️|✅|
|Reuse modules|❌|❌|✅|
|Explicit contracts|❌|❌|✅|
|Reproducibility|⚠️|⚠️|✅|
|Security validation|❌|❌|✅|
|Vendor lock-in|✅|⚠️|❌|

---

## 📌 Status

MVP ready:

- module validation
- contract resolution
- security checks
- profile composition
- Docker Compose rendering
- Kubernetes Helm rendering and local k3s lifecycle
- runtime startup, health state, and shutdown

Next:

- advanced secret providers
- stack bootstrap and health checks

See [docs/roadmap.md](docs/roadmap.md) for milestones and detailed status.
See [docs/support-policy.md](docs/support-policy.md) for OS support policy and platform-specific limitations.

---

## 🤝 Contributing

CDS will only become a useful interoperability layer if the people building and
operating data platforms shape it together.

- **Run a profile** and report what works, what breaks, and what is missing.
- **Document compatibility** between providers and consumers with evidence.
- **Add or improve modules** without coupling them to a specific stack.
- **Strengthen the model** through contracts, validation, security rules, and
  representative profiles.
- **Star and follow** the
  [repository](https://github.com/RonaldHensbergen/composable-data-stack) if
  you want to track the project.

Real-world feedback is especially valuable. CDS should reflect how platforms
actually fail and evolve, not how architecture diagrams pretend they do.

Please read these first:

- [CONTRIBUTING.md](CONTRIBUTING.md)
- [docs/maintainer-merge-policy.md](docs/maintainer-merge-policy.md)
- [SECURITY.md](SECURITY.md) — how to report a vulnerability
- [ACCESSIBILITY.md](ACCESSIBILITY.md) — our accessibility commitment and how
  to report a barrier
- [LEGAL.md](LEGAL.md) — license, code of conduct, security, support, and
  regulatory-compliance documentation index
- [CHANGELOG.md](CHANGELOG.md)
- [RELEASE.md](RELEASE.md)

Development helper tools are located in the gitignored `tools/` directory. See
`tools/pr-cli/README.md` for PR creation scripts.

---

## 📖 Documentation

- [Quickstart](README.md#-quickstart) — get running in 5 minutes
- [CLI Reference](docs/cli-reference.md) — full command/flag table and `[profile]` resolution rules
- [Underlying Commands](docs/underlying-commands.md) — the exact `docker compose` / `helm` / `kubectl` commands behind `cds render` / `up` / `state` / `down`, for running the stack without the `cds` CLI
- [How CDS Works](docs/how-it-works.md) — C4 + sequence diagrams of the validate → plan → render engine
- [Kubernetes Target](docs/kubernetes.md): Helm rendering, Docker Compose migration guide, and isolated local k3s workflow — with ready-made example profiles (`profiles/local-kubernetes-example`, `profiles/production-kubernetes-example` with its `prod` overlay)
- [From Docker Compose to CDS Profile](docs/from-docker-to-cds-profile.md) — complete transformation guide
- [Architecture](docs/architecture.md) — design and core concepts
- [Modules](docs/modules.md) — how to structure reusable components
- [Repository Layout](docs/repo-layout.md) — best practice for keeping platform assets (modules, profiles) separate from project code (Dagster pipelines, dbt/dlt, database scripts)
- [Workload Runtime API](docs/runtime-api.md) — `cds-runtime`: named database connections for Python workloads
- [Production Hardening Checklist](docs/production-hardening-checklist.md) — control-to-owner mapping and the operator verification procedure for a production profile
- [Roadmap](docs/roadmap.md) — planned features and milestones

---

## 📜 License

See `LICENSE`. For security, support, and regulatory-compliance
documentation, see [LEGAL.md](LEGAL.md).
