Metadata-Version: 2.4
Name: data360-autodoc
Version: 0.4.0
Summary: Auto-generate human-readable documentation for Salesforce Data 360 orgs.
Author: data360-autodoc contributors
License-Expression: MIT
Project-URL: Homepage, https://github.com/valentinatihova/data360-autodoc
Project-URL: Repository, https://github.com/valentinatihova/data360-autodoc
Project-URL: Issues, https://github.com/valentinatihova/data360-autodoc/issues
Keywords: salesforce,data-cloud,data-360,documentation,cli
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Documentation
Classifier: Topic :: Software Development :: Documentation
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: click>=8.1
Requires-Dist: PyJWT>=2.8
Requires-Dist: cryptography>=42.0
Requires-Dist: requests>=2.31
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == "dev"
Requires-Dist: responses>=0.25; extra == "dev"
Requires-Dist: black<27,>=26.1; extra == "dev"
Dynamic: license-file

# data360-autodoc

[![PyPI version](https://img.shields.io/pypi/v/data360-autodoc)](https://pypi.org/project/data360-autodoc/)
[![Python](https://img.shields.io/pypi/pyversions/data360-autodoc)](https://pypi.org/project/data360-autodoc/)
[![Tests](https://github.com/valentinatihova/data360-autodoc/actions/workflows/tests.yml/badge.svg)](https://github.com/valentinatihova/data360-autodoc/actions/workflows/tests.yml)
[![License: MIT](https://img.shields.io/badge/license-MIT-green)](LICENSE)

**Auto-generate human-readable documentation for Salesforce Data 360 (Data Cloud) orgs — in minutes, not days.**

Point it at an org and it produces a full data dictionary (DMOs, DLOs, fields, keys), the data streams and field-level mappings behind them, the DMO relationship graph, an ERD, and a deterministic JSON snapshot.

## See the result first

[![Sample output: DLO → DMO mapping graph and a DMO field table](https://raw.githubusercontent.com/valentinatihova/data360-autodoc/main/docs/images/sample-report.png)](https://github.com/valentinatihova/data360-autodoc/blob/main/examples/sample-report/acme-data-cloud-developer-edition.md)

This is real output from a Salesforce **Developer Edition** org (the standard data model and its built-in CRM data streams, no customer data; host name redacted): 22 DMOs, 17 DLOs, 26 mappings. Browse the full sample:

- 📓 [Data dictionary (Markdown)](https://github.com/valentinatihova/data360-autodoc/blob/main/examples/sample-report/acme-data-cloud-developer-edition.md)
- 🔗 [DLO → DMO graph (Mermaid)](https://github.com/valentinatihova/data360-autodoc/blob/main/examples/sample-report/acme-data-cloud-developer-edition.mmd)
- 🧊 [JSON snapshot (input for `diff`)](https://github.com/valentinatihova/data360-autodoc/blob/main/examples/sample-report/acme-data-cloud-developer-edition.json)

What it covers:

- 📓 **Data dictionary** — every DMO and DLO as clean Markdown tables, with field names, types, and keys.
- 🌊 **Data streams + field mappings** — per-stream source/refresh metadata, the Stream → DLO field map, and the DLO → DMO field map (with real labels, not just API names).
- 🔗 **Relationships + ERD** — DMO-to-DMO relationships with cardinality and status, plus a Mermaid graph of DLO → DMO mappings and relationship edges.
- 🧊 **JSON snapshot** — a deterministic, diff-friendly export of your whole org schema.
- 🚨 **Drift detection** — `data360-autodoc diff` compares two snapshots and reports exactly what changed (DMOs, DLOs, fields, mappings, relationships) as Markdown or JSON, with a `--fail-on-change` exit code for CI — see [Drift detection](#drift-detection).

## For who

Built for **Salesforce SI consultants and Data Cloud practitioners** who lose days hand-writing org documentation for every engagement. It works against any Data 360 org you can log into with the Salesforce CLI (or a connected app), and is **tested against Developer Edition / Data Cloud dev orgs** — try it on a sandbox before pointing it at a client. If your org has a shape it doesn't handle yet, [open an issue](https://github.com/valentinatihova/data360-autodoc/issues) and it gets fixed.

## Quick start

Needs Python 3.11+. `pipx` (or `uvx`) installs it in its own environment, so an older system Python doesn't get in the way:

```bash
pipx install data360-autodoc        # or: uvx data360-autodoc ..., or: pip install data360-autodoc

sf org login web --alias my-org     # once: reuse your Salesforce CLI login, no connected app needed
data360-autodoc generate --target-org my-org --output ./docs
```

```
Using API version v68.0
  Listing DMOs… 1046 scanned
Found 22 DMO(s); fetching metadata…
Fetching data streams…
Wrote mydomain-my-salesforce-com.md
Wrote mydomain-my-salesforce-com.mmd
Wrote mydomain-my-salesforce-com.json
Generated docs for 22 DMOs, 17 DLOs, 0 Identity Rulesets
```

A new org lists about a thousand standard DMOs, so the first step takes a few minutes; the counter shows progress.

**Ways to sign in** (use exactly one):

| Mode | Flags | Environment variables |
|------|-------|-----------------------|
| Salesforce CLI session (easiest) | `--target-org ALIAS` | `DATA360_TARGET_ORG` |
| Access token | `--instance-url URL --access-token TOKEN` | `DATA360_INSTANCE_URL`, `DATA360_ACCESS_TOKEN` |
| JWT bearer (CI, connected app + private key) | `--instance-url URL --client-id KEY --private-key server.pem --username USER` | `DATA360_INSTANCE_URL`, `DATA360_CLIENT_ID`, `DATA360_PRIVATE_KEY`, `DATA360_USERNAME` |

Environment variables are read only when no sign-in flag is given, and keep tokens out of the process list. If the session has expired, you get a one-line hint (for `--target-org`: run `sf org login web --alias ALIAS` again) instead of empty docs.

**Other options:**

- `--org-name "Acme Prod"` — document title and output file name (default: the org's host name).
- `--sandbox` — JWT auth against `test.salesforce.com` (sandbox / scratch orgs).
- `--all-dmos` — document every DMO. By default only the DMOs your org uses are documented: **enabled** standard DMOs plus all **custom** DMOs. A new org lists about a thousand disabled standard DMOs, which would only add empty tables. The choice is recorded in the JSON snapshot, and `diff` warns if two snapshots used different settings.
- `--api-version` — the Salesforce REST API version used for the `/ssot/*` metadata calls (e.g. `v62.0`). **By default the tool auto-detects your org's highest supported version** (from `GET /services/data/`), so you normally don't set this. Force it only if auto-detection picks a version where a Data Cloud endpoint misbehaves, or to pin output to a specific version. It must be a valid Salesforce REST API version your org supports.
- `--verbose` — errors are short by default (no host name or API response body, safe on a shared screen); this shows the full detail.

(The `Identity Rulesets` count is currently always `0` — see "Not supported yet" below.)

## What you get

`--format` controls the output:

| Format | Files | What it is |
|--------|-------|------------|
| `markdown` | `.md` + `.mmd` | Data dictionary + Mermaid ERD |
| `json` | `.json` | Deterministic org-schema snapshot |
| `pdf` | — | _Coming soon_ |
| `all` | all of the above | Everything |

### Example output

The Markdown data dictionary. DMO field types come from the org's relationships metadata, and fall back to the mapped DLO field type when the DMO endpoint returns a generic type (shown as `(via DLO)`); DLO keys come from the data streams:

```markdown
## Data Model Objects (DMOs)

### Individual (`Individual__dmo`)

| Name | Type | Key |
| --- | --- | --- |
| Email__c | EmailAddress |  |
| Id__c | Text |  |

## Data Lake Objects (DLOs)

### Order (Home) (`Order_Home__dll`)

| Name | Type | Key |
| --- | --- | --- |
| Amount | Number |  |
| OrderId | Text | PrimaryKey |
```

Beyond the dictionary, the document includes (in this order):

- **Data Streams** — one row per stream: data source, category, primary key, schedule, refresh mode.
- **Field Mapping (Streams → DLO)** — every Data Lake field with its source field, DLO label, type, and a `KQ_`-prefix foreign-key flag.
- **DLO → DMO Field Mappings** — field-level source → target mappings, grouped by DLO → DMO pairing, with real labels joined from DLO metadata.
- **Relationships** — DMO-to-DMO links with cardinality and status (inactive standard relationships stay visible, never dropped):

```markdown
## Relationships

| Object | Field | Cardinality | Related Object | Related Field | Status |
| --- | --- | --- | --- | --- | --- |
| Account | ssot__PrimarySalesContactPointId__c | N:1 | ssot__ContactPointEmail__dlm | ssot__Id__c | INACTIVE |
```

The ERD (renders natively in GitHub). Solid arrows are DLO → DMO mappings; dashed, cardinality-labeled arrows are active DMO → DMO relationships:

```mermaid
graph LR
  Order_Home__dll["Order (Home)"]
  Individual__dmo["Individual"]
  Order_Home__dll --> Individual__dmo
  ContactPointEmail__dlm["Contact Point Email"] -.->|N:1| Individual__dmo
```

Output is **deterministic** — the same org always produces byte-identical docs (collections are sorted alphabetically). That makes the output safe to commit and easy to diff.

### What it reads — and what it doesn't yet

Under the hood it calls the **Data 360 Connect REST API** (`/services/data/v…/ssot/*`): `data-model-objects` (DMOs), `data-model-object-mappings` (DLO→DMO mappings + field names), `…/{dmo}/relationships` (DMO field types **and** the DMO-to-DMO relationship graph), and `data-streams` (DLOs + their fields + the per-stream and field-level mappings, including primary keys). Full request/response shapes are in [`agent_docs/api_reference.md`](agent_docs/api_reference.md).

**Not supported yet.** Calculated Insights and Identity Resolution rulesets are **not fetched** — those sections render as empty placeholders (e.g. `_No Calculated Insights found._`) and the `Identity Rulesets` count stays `0`. Documenting them is on the roadmap. (Profile and Engagement DMOs *are* covered — those are DMO categories, not separate entities.)

**Resilient by default.** If one DMO's metadata can't be read, that DMO is skipped with a warning and the rest of the document is still produced. If more than 500 DMOs match, the list is capped (with a warning). A DMO with no relationships or no DLO mappings is documented with empty sections, not skipped. A failure fetching the DMO list or the data streams stops the run with a clean one-line error — never a stack trace.

## Drift detection

An org documented today is stale next week: a client admin adds a DLO, retypes a field, re-points a mapping, deactivates a relationship. Save a JSON snapshot, take another one later (or from another org — sandbox vs production), and `diff` tells you exactly what changed:

```bash
data360-autodoc diff snapshots/acme-before.json snapshots/acme-after.json
```

No org credentials needed — `diff` only reads the two snapshot files. Try it on the synthetic snapshots in [`tests/fixtures/`](tests/fixtures/):

```markdown
# Data 360 Schema Drift — Acme Data Cloud

_Old: 2026-10-01 06:00:00 UTC · https://acme.my.salesforce.com_
_New: 2026-10-08 06:00:00 UTC · https://acme.my.salesforce.com_

**18 changes detected.**

| Area | Added | Removed | Changed |
| --- | --- | --- | --- |
| DMOs | 1 | 0 | 2 |
| DMO fields | 2 | 0 | 1 |
| DLOs | 1 | 1 | 2 |
| DLO fields | 2 | 0 | 1 |
| DLO → DMO object mappings | 1 | 1 | — |
| DLO → DMO field mappings | 4 | 1 | — |
| Relationships | 1 | 0 | 1 |

## Data Model Objects (DMOs)

### Added

- Account (`ssot__Account__dlm`) — 2 fields

### Changed

#### Sales Order (`ssot__SalesOrder__dlm`)

| Change | Field | Details |
| --- | --- | --- |
| added | ssot__AccountId__c | Text |
| changed | ssot__TotalAmount__c | type: Text → Number |

…

## Relationships

| Change | Object | Field | Related Object | Related Field | Details |
| --- | --- | --- | --- | --- | --- |
| added | ssot__SalesOrder__dlm | ssot__AccountId__c | ssot__Account__dlm | ssot__Id__c | N:1, ACTIVE |
| changed | ssot__ContactPointEmail__dlm | ssot__PartyId__c | ssot__Individual__dlm | ssot__Id__c | status: ACTIVE → INACTIVE |
```

**What is compared:**

| Area | Identified by | Reported |
| --- | --- | --- |
| DMOs, DLOs | API name | added / removed; label changes |
| Fields | object + field name | added / removed; `type`, `is_key`, `key_qualifier` changes |
| Data streams | stream name | added / removed; target DLO, source, category, primary keys, formula fields, schedule, refresh / extraction mode changes |
| Stream → DLO field mappings | stream + DLO field | added / removed; source field, data type, primary / foreign key and nullability changes |
| DLO → DMO object mappings | source DLO + target DMO | added / removed |
| DLO → DMO field mappings | source DLO + field → target DMO + field | added / removed (a re-pointed field shows as one of each) |
| DMO relationships | object + field → related object + field | added / removed; `cardinality`, `status`, label changes |

The snapshot timestamp, display labels on mapping and stream rows, and the `(via DLO)` type-provenance flag are ignored — they are not schema changes. A missing value and an empty string count as the same value.

**Options:**

- `--format markdown|json` — human-readable report (default) or a machine-readable JSON report (`has_changes`, `change_count`, per-area `summary`, and full added/removed/changed detail) for scripts.
- `--output FILE` — write the report to a file instead of stdout. A one-line summary (`Schema drift: 18 changes` / `No schema drift.`) always goes to stderr, so piped stdout stays a clean report.
- `--fail-on-change` — exit with status `1` when anything drifted. Without it, `diff` exits `0` whether or not drift was found. Any error (an unreadable, malformed or incompatible snapshot, or a report that can't be written) exits `2`, so it is never mistaken for drift.

### Drift checks in CI

[`examples/github-actions/data360-drift.yml`](examples/github-actions/data360-drift.yml) is a ready-to-copy GitHub Actions workflow: on a weekly schedule it fetches a fresh snapshot with JWT auth from repository secrets, diffs it against a baseline committed to the repo, writes the drift report to the job summary, uploads the new snapshot as an artifact, and fails the run when the schema drifted. The core of it:

```bash
data360-autodoc generate ... --format json --output current
data360-autodoc diff snapshots/acme-data-cloud.json current/acme-data-cloud.json \
  --output drift/drift.md --fail-on-change
```

To accept a change, replace the baseline with the new snapshot and commit it.

## Future: hosted drift monitoring (paid tier)

The open-source CLI documents your org and detects drift between any two snapshots — free, forever. What it doesn't do is keep watching: storing the snapshot history, alerting the right people, and tracking many client orgs at once.

A hosted tier (planned) adds exactly that layer on top of `diff`: scheduled runs, snapshot history, client-ready changelogs, alerts, and a multi-org dashboard — without ever handing over your org credentials (drift runs in your own environment; the hosted service only stores snapshots and sends alerts).

## Hosted version

A hosted web UI is in the works at **[data360doc.com](https://data360doc.com)** _(placeholder)_ — same docs, plus scheduled drift alerts and a multi-org dashboard for agencies.

## License

MIT
