Metadata-Version: 2.4
Name: trustops-security-data-lake
Version: 0.2.7
Summary: Open-source, self-hosted trust operations — SOC 2, NIST AI RMF, FedRAMP, ISO, HIPAA, PCI DSS. Read-only evidence collection, deterministic control tests, and audit-ready proof with an API-first, MCP-native design.
Author: Mohamed Saad
License: Apache-2.0
Project-URL: Homepage, https://github.com/msaad00/trustops-security-data-lake
Project-URL: Documentation, https://github.com/msaad00/trustops-security-data-lake/tree/main/docs
Project-URL: Repository, https://github.com/msaad00/trustops-security-data-lake
Project-URL: Changelog, https://github.com/msaad00/trustops-security-data-lake/blob/main/CHANGELOG.md
Project-URL: Bug Tracker, https://github.com/msaad00/trustops-security-data-lake/issues
Keywords: compliance,soc2,nist,fedramp,hipaa,pci-dss,iso27001,grc,devsecops,audit,security,evidence,controls,mcp,agents,data-lake
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: System Administrators
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Security
Classifier: Topic :: System :: Systems Administration
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: server
Requires-Dist: fastapi>=0.141.1; extra == "server"
Requires-Dist: uvicorn>=0.30; extra == "server"
Requires-Dist: sqlalchemy>=2.0; extra == "server"
Requires-Dist: alembic>=1.13; extra == "server"
Requires-Dist: psycopg[binary]>=3.1; extra == "server"
Requires-Dist: authlib>=1.3; extra == "server"
Requires-Dist: itsdangerous>=2.2; extra == "server"
Requires-Dist: python3-saml>=1.16; extra == "server"
Requires-Dist: reportlab>=5.0.1; extra == "server"
Requires-Dist: redis>=8.1.0; extra == "server"
Provides-Extra: dev
Requires-Dist: jsonschema>=4.23; extra == "dev"
Requires-Dist: pytest>=8.0; extra == "dev"
Requires-Dist: ruff>=0.8.0; extra == "dev"
Requires-Dist: pip-audit>=2.7; extra == "dev"
Requires-Dist: pre-commit>=3.7; extra == "dev"
Requires-Dist: commitizen>=3.27; extra == "dev"
Requires-Dist: fastapi>=0.141.1; extra == "dev"
Requires-Dist: uvicorn>=0.30; extra == "dev"
Requires-Dist: httpx>=0.28.1; extra == "dev"
Requires-Dist: sqlalchemy>=2.0; extra == "dev"
Requires-Dist: alembic>=1.13; extra == "dev"
Requires-Dist: authlib>=1.3; extra == "dev"
Requires-Dist: itsdangerous>=2.2; extra == "dev"
Requires-Dist: python3-saml>=1.16; extra == "dev"
Requires-Dist: reportlab>=5.0.1; extra == "dev"
Requires-Dist: redis>=8.1.0; extra == "dev"
Requires-Dist: pandas>=2.0; extra == "dev"
Provides-Extra: analytics
Requires-Dist: duckdb>=1.0; extra == "analytics"
Provides-Extra: cloud
Requires-Dist: boto3>=1.34; extra == "cloud"
Requires-Dist: azure-identity>=1.17; extra == "cloud"
Requires-Dist: azure-mgmt-authorization>=4.0; extra == "cloud"
Requires-Dist: azure-mgmt-resource>=26.0.0; extra == "cloud"
Requires-Dist: google-cloud-asset>=3.20; extra == "cloud"
Requires-Dist: google-cloud-resource-manager>=1.12; extra == "cloud"
Requires-Dist: google-cloud-org-policy>=1.11; extra == "cloud"
Requires-Dist: snowflake-connector-python[pandas]>=4.7.1; extra == "cloud"
Requires-Dist: clickhouse-connect>=0.7; extra == "cloud"
Provides-Extra: mcp
Requires-Dist: mcp>=1.2.0; extra == "mcp"
Provides-Extra: sdk
Requires-Dist: httpx>=0.28.1; extra == "sdk"
Provides-Extra: agents
Requires-Dist: langgraph>=0.2.0; extra == "agents"
Dynamic: license-file

<p align="center">
  <img src="docs/images/trustops-social-preview.svg" alt="TrustOps turns customer-owned evidence into deterministic controls and audit-ready proof" width="960">
</p>

<p align="center">
  <sub>Open source · self-hosted · customer-owned evidence</sub>
</p>

<p align="center">
  <a href="https://pypi.org/project/trustops-security-data-lake/"><img src="https://img.shields.io/pypi/v/trustops-security-data-lake?style=flat-square&amp;color=4f7cff&amp;label=PyPI" alt="PyPI version"></a>
  <a href="https://pypi.org/project/trustops-security-data-lake/"><img src="https://img.shields.io/badge/Python-3.11%2B-30a8c9?style=flat-square" alt="Python 3.11+"></a>
  <a href="https://github.com/msaad00/trustops-security-data-lake/actions/workflows/ci.yml"><img src="https://img.shields.io/github/actions/workflow/status/msaad00/trustops-security-data-lake/ci.yml?branch=main&amp;label=Build&amp;style=flat-square" alt="Build status"></a>
  <a href="LICENSE"><img src="https://img.shields.io/badge/License-Apache%202.0-64748b?style=flat-square" alt="License: Apache 2.0"></a>
</p>

<h1 align="center">TrustOps</h1>

<p align="center">
  <strong>Read-only evidence. Deterministic controls. Audit-ready proof.</strong><br/>
  One self-hosted platform for operators, agents, and auditors — backed by the same verifiable JSON.
</p>

<p align="center">
  <a href="#quick-start"><strong>Quick start</strong></a> ·
  <a href="docs/PRODUCT_WALKTHROUGH.md"><strong>Product tour</strong></a> ·
  <a href="docs/CONNECTORS.md"><strong>Connectors</strong></a> ·
  <a href="docs/ARCHITECTURE.md"><strong>Architecture</strong></a> ·
  <a href="docs/api/AGENT_API.md"><strong>API &amp; MCP</strong></a> ·
  <a href="deploy/README.md"><strong>Deploy</strong></a>
</p>

## One operating loop

<p align="center">
  <img src="docs/images/trustops-readme-banner.svg" alt="Collect evidence, evaluate controls, operate findings, and prove posture" width="100%">
</p>

**Collect** evidence read-only, **evaluate** controls with deterministic rules,
**operate** the findings, and **prove** posture with immutable snapshots.

Regulatory requirements are consolidated into a
[Common Control Framework](docs/COMMON_CONTROL_FRAMEWORK.md): you operate one
safeguard, and it satisfies every requirement mapped to it across frameworks.
The catalog holds 942 requirements across 13 frameworks; run
`security-lakehouse frameworks safeguards` for current coverage.

Evidence stays in your environment. Models may summarize and prioritize; they do
not silently change evidence or decide pass/fail.

**The open-source, self-hosted alternative to managed GRC SaaS — interoperable, secure, and built to scale in your environment.**

|                    | TrustOps                                | Managed GRC SaaS         |
| ------------------ | --------------------------------------- | ------------------------ |
| Evidence location  | Your VPC or laptop — data never leaves  | Vendor servers           |
| Control evaluation | Deterministic rules, no model drift     | Varies by vendor         |
| API surface        | Native `/api/v1` + MCP stdio for agents | Web-first, limited API   |
| Interoperability   | CCF maps one safeguard to 13 frameworks | Proprietary control sets |
| Cost model         | Open source (Apache 2.0)                | Per-seat subscription    |
| Self-hosted        | Full stack on your infra or CI          | Cloud-only options       |

## Where to start

| I am…                             | Start here                                                                                                  |
| --------------------------------- | ----------------------------------------------------------------------------------------------------------- |
| **DevSecOps / security engineer** | [Quick start](#quick-start) → [Connectors](docs/CONNECTORS.md) → [CI gate](docs/CI_GATE.md)                 |
| **Compliance / GRC lead**         | [Product tour](docs/PRODUCT_WALKTHROUGH.md) → [Framework coverage](docs/FRAMEWORK_COVERAGE.md)              |
| **AI / agent builder**            | [Agent API](docs/api/AGENT_API.md) → [MCP server](docs/MCP.md) → [AIBOM](docs/AIBOM.md)                     |
| **Auditor / assessor**            | [Audit room](docs/PRODUCT_WALKTHROUGH.md#audit-room) → [Snapshot integrity](docs/ARCHITECTURE.md#snapshots) |
| **Operator deploying**            | [deploy/README.md](deploy/README.md) → [Server auth](docs/SERVER_AUTH.md)                                   |

## Quick start

Requires Python 3.11+ and Node 22+ (the console is built from source; it is not
committed to the repository). For the fastest path, use `make demo-local` which
runs the whole sequence in one step.

```bash
python -m venv .venv
source .venv/bin/activate
pip install -e ".[dev,server]"

make web-install web-build   # builds the console into src/security_lakehouse/web/dist

security-lakehouse fixtures load --company golden --out build/lakehouse
security-lakehouse db upgrade --lake build/lakehouse
security-lakehouse serve \
  --lake build/lakehouse \
  --server \
  --allow-insecure-no-auth \
  --port 8787
```

Open [http://127.0.0.1:8787/console/dashboard/](http://127.0.0.1:8787/console/dashboard/).

> **Note:** Skipping `make web-install web-build` leaves `/console/` as a 404.
> The server mounts the console only when a built dist is present.
> `--allow-insecure-no-auth` is for local development only.

**Other install paths:**

```bash
# Evaluator: container (no build required)
docker run -p 8787:8787 ghcr.io/msaad00/trustops:latest

# Kubernetes
helm install trustops deploy/helm/trustops

# CLI / local lake only (zero dependencies)
pip install trustops-security-data-lake
security-lakehouse fixtures load --company golden --out ./lake
```

See [deploy/README.md](deploy/README.md) for production configuration and authentication setup.

<details>
<summary><b>Surfaces</b> — console, API, CLI, MCP, CI</summary>

| Surface          | Purpose                                                            |
| ---------------- | ------------------------------------------------------------------ |
| **Console**      | Posture, controls, evidence, findings, workflows, and audit room   |
| **API**          | Versioned `/api/v1` contract                                       |
| **CLI**          | Local pipelines, validation, snapshots, and server operations      |
| **MCP & agents** | Read posture and propose governed actions with approval boundaries |
| **CI**           | Block releases when posture or control-test thresholds regress     |

</details>

<details>
<summary><b>Connectors</b> — least-privilege, read-only</summary>

AWS · Azure · GCP · GitHub · GitLab · Okta · Snowflake · ClickHouse, plus the
scanner, ticketing, and AI-platform entries in
[`connectors/catalog.json`](connectors/catalog.json).

</details>

## Product preview

|                                         Trust Home                                         |                                         Audit room                                          |
| :----------------------------------------------------------------------------------------: | :-----------------------------------------------------------------------------------------: |
| <img src="docs/images/trustops-demo-dashboard.png" alt="TrustOps Trust Home" width="100%"> | <img src="docs/images/trustops-demo-audit-room.png" alt="TrustOps audit room" width="100%"> |

|                                           Evidence                                           |                                             Connectors                                              |
| :------------------------------------------------------------------------------------------: | :-------------------------------------------------------------------------------------------------: |
| <img src="docs/images/trustops-demo-evidence.png" alt="TrustOps evidence room" width="100%"> | <img src="docs/images/trustops-demo-connectors.png" alt="TrustOps connector registry" width="100%"> |

More views: [frameworks](docs/images/trustops-demo-frameworks.png) · [insights](docs/images/trustops-demo-insights.png) · [workflows](docs/images/trustops-demo-workflows.png) · [trust center](docs/images/trustops-demo-trust-center.png)

## Connect a live source

The default path is agentless and read-only; no pre-existing data lake is required.

- **Console:** open **Connectors**, choose a source, then run **Discover → Test → Enable → Sync**.
- **Headless:** follow the [connector setup playbook](docs/playbooks/HEADLESS_CONNECTOR_SETUP.md) for API, CLI, and MCP flows.
- **Existing lake:** connect Snowflake or ClickHouse when evidence already lives there.

Cloud connectors use short-lived provider credentials or workload identity. No connector requires pasted long-lived cloud keys. TrustOps stores non-secret identifiers, redacted fingerprints, sync history, and evidence hashes.

Connector security contracts:

- **AWS** uses STS AssumeRole, one External ID per deployed role, short-lived session credentials, and read-only IAM posture APIs. Temporary credentials expire after each session; TrustOps stores no long-lived access keys. Scale rollout with CloudFormation StackSets or Terraform workspaces; Bulk account import is the next operator surface.
- **Azure** uses a customer-owned Entra application, managed identity, or federated workload identity with Reader scope. Tokens are short-lived, and no Azure password or raw client secret is stored.
- **Snowflake** supports browser SSO for human proof or a read-only service identity with a key-pair or OAuth token reference held by the runtime secret manager. TrustOps stores account, role, and view identifiers — not passwords or private-key contents. Snowflake is the existing security-data-lake path.

<p align="center">
  <img src="docs/images/trustops-aws-sts-lifecycle.svg" alt="TrustOps AWS STS AssumeRole lifecycle" width="96%">
</p>

## Architecture

```text
read-only source → raw observation → normalized fact → deterministic evaluation
                 → finding/current posture → immutable snapshot → governed action
```

<p align="center">
  <img src="docs/images/trustops-assessment-architecture.svg" alt="TrustOps collection, evaluation, and proof architecture" width="96%">
</p>

The console, CLI, MCP server, agents, and CI gate share the same API and assessment engine. This keeps browser output and headless automation consistent.

## What ships

| Area              | Included                                                                             |
| ----------------- | ------------------------------------------------------------------------------------ |
| **Compliance**    | SOC 2, NIST AI RMF, FedRAMP, ISO, CIS AWS, HIPAA, PCI DSS, GDPR, and EU AI Act packs |
| **Evidence**      | Freshness SLAs, provenance, SHA-256 verification, tags, and saved views              |
| **GRC workflows** | Remediation, policies, attestations, vendor risk, access reviews, and approvals      |
| **Identity**      | OIDC, SAML, API keys, RBAC, tenant boundaries, and SCIM scaffolding                  |
| **Deployment**    | Local, Docker, Helm, EKS reference IaC, Snowflake, and ClickHouse                    |
| **Exports**       | Snapshots, executive PDF, trust shares, OpenAPI, MCP, and GitHub posture gate        |

See the [product shape](docs/PRODUCT_SHAPE.md) for shipped, partial, and planned capability status.

## Verify

```bash
make smoke       # backend, contracts, docs, brand, pipeline, API
make web-ci      # install, typecheck, production build
make security    # dependency audits and pre-commit checks
```

Regenerate documentation screenshots with `make demo-screenshots-full`.

## Repository map

```text
src/security_lakehouse/   assessment engine, API, auth, connectors, MCP
app/web/                  Next.js console
controls/ frameworks/     control catalogs, packs, and mappings
deploy/                   Docker, Helm, cloud, warehouse, and IaC examples
docs/                     product, architecture, operations, and API guides
```

## Documentation

- [Product walkthrough](docs/PRODUCT_WALKTHROUGH.md)
- [Architecture](docs/ARCHITECTURE.md)
- [Connector catalog](docs/CONNECTORS.md)
- [Continuous ingestion](docs/CONTINUOUS_INGESTION.md)
- [Audit readiness](docs/AUDIT_READINESS.md)
- [Agent API](docs/api/AGENT_API.md)
- [Deployment](docs/DEPLOYMENT.md)
- [Roadmap](ROADMAP.md)

Apache-2.0 licensed. Third-party visual assets and usage terms are documented in [THIRD_PARTY_ASSETS.md](docs/THIRD_PARTY_ASSETS.md).
