Metadata-Version: 2.5
Name: mule-discovery
Version: 3.2.0
Summary: Scan Mule applications for migration complexity assessment
Project-URL: Homepage, https://github.com/KongHQ-CX/mule-discovery
Author: Stephen Brown
License-Expression: MIT
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Requires-Python: >=3.10
Requires-Dist: pyyaml>=6.0
Provides-Extra: anypoint
Requires-Dist: anypoint-sdk>=0.2.0; extra == 'anypoint'
Provides-Extra: dev
Requires-Dist: pytest-cov>=4.0; extra == 'dev'
Requires-Dist: pytest>=7.0; extra == 'dev'
Description-Content-Type: text/markdown

# mule-discovery

Scan Mule applications for migration complexity assessment.

Parses Mule 4 (and 3) XML source files, POM dependencies, DataWeave scripts, and API specifications to produce a structured migration readiness report with complexity scoring.

## Estate Analysis

The output produced by `mule-discover` (JSON or YAML) can be fed into the [estate-analyzer](https://github.com/KongHQ-CX/kong-ps-agent-skills/tree/main/mule-analysis/estate-analyzer) agent skill to generate pre-sales migration reports. The estate-analyzer processes discovery output across your entire Mule application estate to produce complexity summaries, connector frequency analysis, PoC candidate recommendations, and migration sizing reports.

## Quick Start (uv)

No install required - just run from the project directory:

```bash
cd mule-discovery

# Discover all Mule apps under a directory
uv run mule-discover /path/to/apps --output-dir ./inventory

# JSON output instead of YAML
uv run mule-discover /path/to/apps --json --output-dir ./inventory
```

`uv run` reads `pyproject.toml`, resolves dependencies into an ephemeral environment, and runs the command. Nothing is installed globally.

## Installation

### From PyPI with `uv` (recommended)

Install as a global CLI tool - `uv` puts the entry points on your `PATH`:

```bash
uv tool install mule-discovery

# Then run directly from anywhere
mule-discover /path/to/apps --output-dir ./inventory
```

Upgrade or remove later with:

```bash
uv tool upgrade mule-discovery
uv tool uninstall mule-discovery
```

For Anypoint Platform integration (policy scanning), install with the extra:

```bash
uv tool install "mule-discovery[anypoint]"
```

### From PyPI with `pip`

```bash
pip install mule-discovery

# Then run directly
mule-discover /path/to/apps --output-dir ./inventory
```

For Anypoint Platform integration (policy scanning):

```bash
pip install mule-discovery[anypoint]
```

### From source (development)

```bash
uv sync --extra dev
```

Requires Python 3.10+.

## CLI Tools

### `mule-discover`

Recursively find all Mule applications under a directory and produce migration complexity reports for each.

```bash
# Discover all apps, write YAML inventories (default) to ./inventory
uv run mule-discover /path/to/apps --output-dir ./inventory

# JSON output
uv run mule-discover /path/to/apps --json --output-dir ./inventory

# Suppress progress output
uv run mule-discover /path/to/apps -o ./inventory -q

# Custom scoring weights (bands, divisors, catalogue)
uv run mule-discover /path/to/apps --weights ./my-weights.yaml -o ./inventory

# Resolve parent POMs and external flow-refs against a customer's shared frameworks
uv run mule-discover /path/to/apps --common-libs /path/to/common_libs_home -o ./inventory

# Score an Exchange-hosted API spec the operator downloaded
uv run mule-discover /path/to/apps --specs /path/to/exchange_specs -o ./inventory

# Or pull Exchange-hosted specs with the Anypoint SDK first, in one step
uv run mule-discover /path/to/apps --specs-from-anypoint -o ./inventory

# Share-safe output: pseudonymise hostnames, usernames, client ids and queue/topic/destination names
uv run mule-discover /path/to/apps --redact -o ./inventory
```

Beside the per-app `<app>-inventory.yaml` files, every run writes one `estate-summary.yaml` (`.json` with `--json`) to the output directory: the number of apps in each band (`bands`, with `M+` counted under `M` and again under `bands_widened`), the estate's total points, each app's band, points and unknowns largest first, and any app that failed. The console also prints a one-line `Bands: S 3, M 10 (2+), L 6, XL 3` summary, even with `-q`.

`--redact` is the recommended flag for output a customer sends to us before the first call; see [docs/customer-runbook.md](docs/customer-runbook.md) for the one-page walkthrough.

#### `--common-libs`

Customers often ship their Mule apps alongside a separate "common libs" / shared frameworks directory containing parent POMs (which the apps inherit dependencies from) and shared Mule projects defining flows that the apps reference via `flow-ref`. Without this directory, the discovery tool only sees what's local to each app and reports an empty connector list and unresolved flow-refs.

When you pass `--common-libs PATH`, the tool will:

- **Resolve parent POM chains.** Each app's `<parent>` coordinates are looked up in the directory; if found, the parent's `<dependencies>` are merged into the app's connector inventory (with `source_file` indicating which parent contributed each connector). Multi-level chains are walked.
- **Resolve external flow-refs.** Every `flow-ref` whose target is not defined inside the app is checked against an index of `<flow>` and `<sub-flow>` definitions found anywhere under the common-libs directory. Resolved refs include the library name, source file, and line number; unresolved ones are reported separately so they can be flagged for follow-up.
- **Pull in only what's referenced.** The common-libs directory may be large; the tool does not blanket-include everything from it. Only parent POMs the apps actually inherit from and flows the apps actually reference are surfaced.

Expected directory layout (one Mule project per top-level subdirectory):

```
common_libs_home/
├── hvcp-mule4-common-framework-handler/
│   ├── pom.xml
│   └── src/main/mule/*.xml
├── hvcp-mule4-common-messaging-framework/
│   └── ...
└── shared-parent-poms/
    └── pom.xml
```

The discovery output gains two new fields under the per-app inventory:

- `resolved_parent_poms`: list of parent POMs in the inheritance chain, each with `groupId`, `artifactId`, `version`, `resolved` (boolean), `source_file`, and the `dependencies` declared there.
- `external_flow_refs`: `{"resolved": [...], "unresolved": [...]}`. Each resolved ref shows the library and source file/line of the matching definition.

#### `--specs`

Customers often keep the RAML or OAS contract for an API in Anypoint Exchange rather than committing it to the app's source tree. When an `apikit:config`'s `raml`/`api` attribute does not resolve on disk, the tool checks the app's `pom.xml` (and its resolved parent chain, when `--common-libs` is also given) for a dependency shaped like an Exchange asset - an explicit `raml`, `oas`, `wsdl` or `raml-fragment` classifier, or a `zip`-typed dependency whose groupId looks like an Exchange organisation id (a UUID) or ends in `.exchange`. Without `--specs`, that becomes a specific, recoverable `api-spec-in-exchange` unknown naming the coordinates; a recoverable unknown is still listed and counted but does not widen the band.

Pass `--specs PATH` to score the spec instead. For each Exchange dependency found, the directory is searched, in order, for:

```
specs_home/
├── <artifact_id>/                              # unzipped asset directory
│   └── api.raml                                # or the single *.raml / *.yaml / *.json
├── <artifact_id>-<version>/                     # same, version-qualified
├── <artifact_id>-<version>-<classifier>.zip      # opened in place, not unzipped to disk
└── <artifact_id>.raml                            # (or .yaml / .json) bare file
```

The entry-point file is `api.raml` when present, else the single `.raml`, else the single `.yaml` or `.json`; its `!include` chain resolves inside the same directory or zip. A resolved spec is scored exactly like a RAML/OAS committed to the source tree. When the directory does not have the asset, the `api-spec-in-exchange` unknown returns, and its detail names the directory that was searched.

##### Specs from Exchange

When the specs directory doesn't exist yet, the Anypoint SDK the tool already uses for policy work can build it directly, in one of two ways.

**Two-step**: download once, reuse the directory across scans and other apps:

```bash
export ANYPOINT_CLIENT_ID=...
export ANYPOINT_CLIENT_SECRET=...

uv run mule-download-specs /path/to/apps --output-dir ./exchange_specs
uv run mule-discover /path/to/apps --specs ./exchange_specs -o ./inventory
```

`mule-download-specs` finds every app under `search_path` (and, with `--common-libs`, their resolved parent-chain poms too), collects the Exchange-hosted specs their poms declare, downloads each distinct one into `--output-dir` in the same layout `--specs` reads, and prints a JSON summary (coordinates, `status` - `downloaded`, `cached` or `failed` - and `path` per asset) to stdout. A directory that already has an asset is left alone and reported `cached`, so re-running the command after adding apps only downloads what's new. `ANYPOINT_ORG_ID` is not required here - Exchange downloads are addressed by group id, not organisation.

**One-step**: `--specs-from-anypoint` downloads into `<output-dir>/specs` and scores in the same command, as if `--specs` had named that directory:

```bash
export ANYPOINT_CLIENT_ID=...
export ANYPOINT_CLIENT_SECRET=...

uv run mule-discover /path/to/apps --specs-from-anypoint -o ./inventory
```

`--specs-from-anypoint` and `--specs` are mutually exclusive. Both require the `anypoint` extra; without it, the CLI exits with the same install hint `mule-download-policies` gives. Each per-app inventory's `summary.specs_source` records which of the three states applied: `"anypoint"`, `"directory"` or `"none"`.

**Naming an asset directly**: `--asset GROUP:ARTIFACT:VERSION[:CLASSIFIER]` (repeatable) downloads one Exchange asset by its coordinates, without it being a pom dependency - useful for an asset a demo organisation or a customer published but never wired into a pom, or to exercise the resolved path on an app whose pom lacks it. `CLASSIFIER` defaults to `raml`; accepted values are `raml`, `oas`, `wsdl`, `raml-fragment`. With `mule-download-specs`, `search_path` becomes optional once at least one `--asset` is given:

```bash
uv run mule-download-specs --asset 68ef9520-24e9-4cf2-b2f5-620025690913:combined-api:1.0.2
uv run mule-download-specs /path/to/apps --asset 68ef9520-24e9-4cf2-b2f5-620025690913:combined-api:1.0.2:oas --output-dir ./exchange_specs
```

`--specs-from-anypoint` takes the same `--asset` flag, so the one-step form can name an asset too:

```bash
uv run mule-discover /path/to/apps --specs-from-anypoint --asset 68ef9520-24e9-4cf2-b2f5-620025690913:combined-api:1.0.2 -o ./inventory
```

An `--asset` spec is merged with the pom-derived ones and deduplicated on the same coordinates; the JSON summary row for each asset carries `"source": "asset-flag"` or `"source": "pom"` so it's clear which named it.

#### Connector discovery without a parent POM

When an app's `pom.xml` declares no dependencies (typical when everything is inherited from a parent POM that isn't on disk), the tool falls back to deriving connectors from the `xmlns:` declarations in the Mule XML files themselves. These entries are tagged with `notes: "derived from XML namespaces - parent POM not resolved"` and `source_file: "(xml-namespaces)"` so the source of each connector remains traceable. Pass `--common-libs` to replace this fallback with proper parent-POM resolution where possible.

Each per-app report includes:
- Flow inventory, each flow typed API / EVENT / SCHEDULED / BATCH / SUBFLOW and, for an APIkit route flow, labelled `source.type: apikit-route` with its `config_ref`, `path` and `method`
- DataWeave transformation analysis and classification
- HTTP listener and scheduled job detection
- Connector inventory with migration weights
- API specification detection (OpenAPI, WSDL)
- External dependency and out-of-scope item tracking
- AWS service usage (SQS, S3, DynamoDB)
- SOAP/WSDL service detection
- HTTP request-config inventory and connector authentication metadata (`request_configs`, `connector_auth`)
- Effort points, band (S / M / L / XL, with a `+` when unknowns widen the reading), feature and feature-zero breakdown, and the unknowns list, each marked `recoverable` when a known input (`--specs`, `--common-libs`) would resolve it
- Cross-app hops: outbound HTTP hosts resolved through the property files and matched against the other applications in the same scan
- Per feature: `closure` (its flow and sub-flow names, including router wrappers and global-handler flows), `shared_flows` (the closure names also reached by another feature) and `outbound` (one row per distinct outbound call - `kind`, `config_ref`, and, for HTTP, the matched hop's `host`, `target_app` and `external`)
- `summary.outbound_system_count`: the distinct outbound hosts and connector configs across every feature
- `score.weights`: the full calibration the run used (bands, items, divisors, feature_zero, dataweave_labels, split_threshold, unknown_plus_threshold, family_prefix_segments, deployment_keys, and the catalogue as `{id, artefact_type, points, reason}` rows), embedded in every output so a reader never needs a copy of the weights file

### `mule-scan-policies`

Scan Anypoint Platform for API policies on deployed applications. Requires the `anypoint` extra.

```bash
pip install -e ".[anypoint]"

export ANYPOINT_CLIENT_ID=...
export ANYPOINT_CLIENT_SECRET=...
export ANYPOINT_ORG_ID=...
export ANYPOINT_ENV_ID=...

uv run mule-scan-policies
uv run mule-scan-policies --format json
```

### `mule-download-policies`

Download custom policies from Anypoint Exchange. Requires the `anypoint` extra.

```bash
export ANYPOINT_CLIENT_ID=...
export ANYPOINT_CLIENT_SECRET=...
export ANYPOINT_ORG_ID=...

uv run mule-download-policies --output-dir ./custom_policies
```

### `mule-download-specs`

Download every Exchange-hosted API spec an app estate's poms declare, into the layout `mule-discover --specs` reads. Requires the `anypoint` extra. See "Specs from Exchange" above for the full two-step and one-step forms.

```bash
export ANYPOINT_CLIENT_ID=...
export ANYPOINT_CLIENT_SECRET=...

uv run mule-download-specs /path/to/apps --output-dir ./exchange_specs
```

## Complexity Scoring

Each application is sized in **effort points**: the work in each feature,
plus a one-off foundation, plus a count of the things the scanner can see
but cannot size. The full model, including every catalogue entry and the
reasoning behind it, is in [docs/scoring-model.md](docs/scoring-model.md).

The shape of the number is:

```
app points = sum(feature points) + feature zero points
```

- A **feature** is one unit of migration work: one top-level resource
  family for an APIkit API (`/orders` covers `get:\orders`, `post:\orders`
  and `get:\orders\(id)\lines`), or one listener, scheduler or subscriber
  for everything else. Each feature owns the transitive set of flows its
  entrypoints reach through `flow-ref`.
- Every artefact a feature reaches - the flow XML itself, DataWeave, Java,
  scripts, SQL, WSDL and XSD, the RAML or OAS - is measured the same way:
  effective lines over a per-type divisor, plus one point per known-bad
  construct found in it.
- Fixed per-item points are added for each endpoint, outbound call,
  error-handler branch and cross-app hop.
- **Feature zero** is the one-off foundation: connector configs, property
  keys, the global error handler, the API contract, shared DataWeave
  modules and the parent POM chain, plus a fixed base added once per app
  for the skeleton, CI and connector clients a migration builds before its
  first feature. An app with no flow, sub-flow or connector config gets no
  base.

### Bands

The band is read straight off the app total:

| Band | Points |
|---|---|
| S | under 15 |
| M | 15 to under 40 |
| L | 40 to under 90 |
| XL | 90 and over |

A band gains a `+` (for example `L+`) when the unknowns count exceeds
`unknown_plus_threshold`, or more than a quarter of the feature count. The
plus means the reading is wider than the number: there are constructs the
scanner recognised but could not size.

### Unknowns

An unknown never adds points; it is listed with `file:line` and counted.
Dynamic `flow-ref`s, flow-refs that resolve nowhere, `http:request` hosts
that no property file resolves to a literal, DataWeave modules and Java
classes named but not found, unmeasured scripting engines, undefined
property keys and unresolved parent POMs all land here.

### Extended scenarios

Beside the points and the unknowns, each app carries a list of
extended-scenario markers: the capabilities the engagement terms set
apart, where the migration is likely to need a customer-provided target
component or gap closure outside the standard patterns.

| Marker | What it says |
|---|---|
| `sap` | the app talks to SAP - a `sap:` element, the connector as a dependency, or a flow named for an IDoc, BAPI or RFC beside one |
| `managed-file-transfer` | a file move made exactly-once: retried, deduplicated, transactional, locked, or written to a `.tmp` name and renamed |
| `b2b-edi` | an EDI or B2B module: `edi`, `x12`, `edifact`, `hl7`, `b2b`, `partner-manager`, `as2` |
| `transactional-messaging` | one commit spanning two systems: an XA transaction, a transactional queue or database operation, a manual JMS acknowledgement |
| `event-processing` | events handled as a stream: the CEP module, an aggregator, a Kafka consumer group with a window or a retry scope, a prefetching subscriber beside an object store |
| `custom-connector` | a config in a namespace no known module claims, or a `mule-plugin` dependency from neither Mule's nor MuleSoft's groupId |
| `stateful-batch` | a batch job that carries state or runs on past failures, or state declared to outlive the application |
| `platform-only` | a capability the Anypoint platform supplies rather than the application: the API gateway, clustering, CloudHub, Runtime Fabric, FIFO queues, a persistent cache, secure properties on an Enterprise runtime |

Each marker carries `file:line` evidence and the matched element or
dependency in plain words. A marker is never points and never moves a
band: it means an element of the application depends on a
customer-provided target component or on gap closure outside the standard
patterns. In the engagement terms' own words, "The presence of one of
these capabilities does not prevent us from migrating the application."

The markers are in every output as `score.extended_scenarios`, counted in
`summary.extended_scenario_count` and named in
`summary.extended_scenario_kinds`; the console summary prints
`Extended scenarios: sap, transactional-messaging` when an app carries
any. The element names, patterns and dependency tokens live in the
weights file's `extended_scenarios` section, so a customer-specific
signal is added with `--weights` and no code change. See
`docs/scoring-model.md`, "Extended scenarios".

### `--weights`

Every number the model uses - bands, divisors, per-item points,
feature-zero points and the whole known-bad catalogue - lives in one
versioned YAML file shipped with the tool. Override it per run:

```bash
uv run mule-discover /path/to/apps --weights ./my-weights.yaml -o ./inventory
```

An override file only needs the top-level keys it changes; every key it
omits falls back to the shipped default. The `weights_version` in force is
stamped into every output file and into `summary.weights_version`, and the
full calibration itself is embedded as `score.weights` so a reader never
needs a copy of the weights file. Two outputs are comparable only when
their weights versions match.

### DataWeave Classification

Each DataWeave inventory entry keeps its `complexity` and `classification`
fields, now read off the same measurement the score uses:

| Field | Value | Criteria |
|---|---|---|
| complexity | LOW / MEDIUM / HIGH | under `dataweave_labels.medium` points / under `dataweave_labels.high` / at or above it |
| classification | business_logic | a user-defined `fun`, or one of the trap constructs |
| classification | field_level_logic | any other catalogue hit (`lookup`, `readUrl`, `java!`, a non-core `import`) |
| classification | simple_mapping | no catalogue hit |

The two boundaries are points, so they only mean anything against the
DataWeave divisor they were drawn for. Both live beside that divisor in
`data/weights.yaml` under `dataweave_labels`, and a `--weights` override
moves them with it. The shipped pair is 0.15 and 0.4, which reads the
reference corpus as roughly 64 percent LOW, 28 percent MEDIUM and 8
percent HIGH.

## Package Structure

```
src/mule_discovery/
├── __init__.py                # Main discover_mule_app() orchestrator
├── constants.py               # XML namespaces, element classifications, connector weights
├── xml_helpers.py             # XML utility functions
├── models/                    # Data models (dataclasses)
│   ├── result.py              # DiscoveryResult (top-level container)
│   ├── flows.py               # FlowInfo, BatchInfo, ChoiceInfo, ScatterGatherInfo, ...
│   ├── connectors.py          # ConnectorInfo, SpringDependency
│   ├── dataweave.py           # DataWeaveInfo
│   ├── listeners.py           # HttpListenerInfo, ScheduledJobInfo
│   ├── dependencies.py        # ExternalDependencyInfo, SourceFiles, OutOfScopeItem
│   └── schemas.py             # ApiSpecInfo (OpenAPI, WSDL)
├── data/
│   └── weights.yaml           # Default scoring weights and known-bad catalogue
├── parsers/                   # File IO → models
│   ├── file_discovery.py      # find_mule_apps(), find_mule_xml_files()
│   ├── mule_xml.py            # Mule XML parsing (flows, listeners, jobs)
│   ├── pom.py                 # POM parsing (app name, version, connectors, parent-POM chain resolution)
│   ├── common_libs.py         # Indexing of customer-supplied common-libs (flows + parent POMs)
│   ├── http_auth.py           # HTTP auth config extraction
│   ├── dataweave.py           # DataWeave script parsing
│   ├── soap.py                # SOAP/WSDL service detection
│   ├── aws.py                 # AWS service detection (SQS, S3, DynamoDB)
│   ├── openapi.py             # OpenAPI spec detection
│   └── wsdl.py                # WSDL parsing utilities
├── analysis/                  # Models → models (pure functions)
│   ├── classification.py      # Flow type and source category constants
│   ├── complexity.py          # Script complexity, DataWeave inventory labels
│   ├── patterns.py            # Pattern detection (async, scatter-gather, choice, ...)
│   └── dependencies.py        # External dependency and out-of-scope extraction
├── scoring/                   # The effort-points model (docs/scoring-model.md)
│   ├── weights.py             # The versioned calibration file loader
│   ├── tokenize.py            # Effective lines, comment and string stripping
│   ├── measure.py             # The one rule: size points plus known-bad hits
│   ├── features.py            # Entrypoints, resource families, flow closure
│   ├── artefacts.py           # Closure to the files it depends on
│   ├── properties.py          # Property files and placeholder resolution
│   ├── hops.py                # Cross-app hops resolved through properties
│   ├── feature_zero.py        # The one-off foundation, sized per app
│   ├── unknowns.py            # Everything seen but not sizeable
│   ├── specs.py               # Exchange-hosted spec recognition + --specs resolution
│   └── assemble.py            # Points, bands, the embedded weights
├── output/                    # Models → formatted strings
│   ├── yaml_output.py         # YAML
│   ├── json_output.py         # JSON
│   └── text_output.py         # Human-readable text summary
├── anypoint/                  # Anypoint Platform integration (optional)
│   ├── policies.py            # Policy scanning
│   ├── exchange.py            # Custom policy download
│   └── specs.py               # Exchange-hosted API spec download
└── cli/                       # CLI entry points (thin wrappers)
    ├── discover.py            # mule-discover
    ├── scan_policies.py       # mule-scan-policies
    ├── download_policies.py   # mule-download-policies
    └── download_specs.py      # mule-download-specs
```

### Design Principles

- **No function does both IO and computation.** Parsers read files → return models. Analysis takes models → returns models. Output takes models → returns strings.
- **All data models are plain dataclasses** with typed fields - no methods with side effects.
- **All analysis functions are standalone** - no class methods, no inheritance.
- **Each output format is a separate module.**

## Testing

```bash
make test
```

Or directly:

```bash
uv run --extra dev python -m pytest
```

Coverage is enforced at 70% (branch coverage) via `pyproject.toml`.
