Metadata-Version: 2.4
Name: mainframe-modernization-toolkit
Version: 0.1.10
Summary: Deterministic COBOL/JCL analysis and mainframe modernization tooling
Author: Mainframe Migration Toolkit Contributors
License-Expression: Apache-2.0
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Code Generators
Classifier: Topic :: Software Development :: Quality Assurance
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: release
Requires-Dist: build>=1.2; extra == "release"
Requires-Dist: twine>=6; extra == "release"
Dynamic: license-file

# Mainframe Modernization Toolkit

Deterministic COBOL and JCL navigation, impact analysis, migration evidence,
and code-generation tools for Python modernization projects.

The package includes:

- A Python CLI with 17 deterministic analysis and generation commands.
- A bundled VS Code extension and COBOL/JCL language server.
- GitHub Copilot Language Model Tools for dependency-aware agent workflows.
- A packaged migration skill and specialized VS Code agent.

The parsers and graph operations do not use an AI model. Given the same source
and configuration, they produce the same result.

## Quick start

### 1. Install the VS Code extension

Python 3.10 or newer is required. The VSIX contains the Python toolkit, so no
pip package is needed for extension commands or language-model tools. Install
from the Marketplace:

```bash
code --install-extension mainframe-migration-toolkit.mainframe-migration-toolkit
```

You can also download the VSIX from the GitHub release and install it manually:

```bash
code --install-extension mainframe-migration-toolkit-0.1.10.vsix
```

The extension tries `mainframeMigration.pythonPath`, workspace virtual
environments, active virtual/Conda environments, common platform locations,
then `python3`, `python`, or `py`. Python versions older than 3.10 are skipped.
Installed CLI and module launches remain fallbacks. A non-default
`mainframeMigration.executablePath` is exact-only and bypasses auto mode.

### 2. Optional terminal CLI

Install the PyPI package only when you also want `mainframe-toolkit` in a
terminal:

```bash
pipx install mainframe-modernization-toolkit
# or
python -m pip install mainframe-modernization-toolkit
mainframe-toolkit --version
```

The wheel still contains the matching VSIX for export-based installation:

```bash
mainframe-toolkit vsix export --output mainframe-migration-toolkit.vsix
mainframe-toolkit vsix verify mainframe-migration-toolkit.vsix
code --install-extension mainframe-migration-toolkit.vsix
```

### 3. Initialize a mainframe workspace

Use **Mainframe Migration: Initialize Workspace** from the Command Palette, or
run the optional terminal CLI from the workspace containing COBOL, copybooks,
and JCL:

```bash
mainframe-toolkit workspace init .
```

This adds, without overwriting existing files:

- `mainframe-migration.json` and its schema.
- The relational IR schema.
- `.github/skills/mainframe-jcl-migration/`.
- `.github/agents/mainframe-jcl-migrator.agent.md`.

Edit `mainframe-migration.json` so its source libraries, extensions, encoding,
known external programs, and transport profiles match the workspace.

### 4. Verify the setup

```bash
mainframe-toolkit doctor --workspace .
mainframe-toolkit run migration_preflight -- . --jcl MYJOB --format json
mainframe-toolkit run migration_preflight -- . --jcl MYJOB --program MYPROG --scope program-only --format json
mainframe-toolkit run generate_program_capsule -- . --jcl MYJOB --program MYPROG --scope program-only --out-dir migration/MYJOB/capsules --format json
```

Preflight exits `0` when generation is unblocked and `2` when required source
or configuration is missing or ambiguous. Its JSON report includes detected
`capabilities`, a stable `capabilityDigest`, and `capabilityResolutions` that
show which user policy and adapter, if any, applies to each capability.
Unknown static COBOL `CALL` and JCL `EXEC PGM` targets are nonblocking
unknown-provenance `EXTERNAL_PROGRAM_ADAPTER` TODOs with explicit adapter
requirements; missing copybooks, missing PROCs, and duplicate/conflicting
sources remain `BLOCK` findings. Classify a verified target later through
`knownExternalPrograms` or
`knownExternalUtilities`.
Program scopes require both `--program` and `--jcl`; `program-only` keeps local
callees as boundary references, while `program-with-dependencies` includes the
transitive local static CALL closure. Supplying `--program` without `--scope`
selects the latter, so the packaged migration workflow always passes explicit
`--scope program-only`. It uses dependency closure only when the user requests
one cohesive downstream unit. For multi-program jobs, this writes one capsule
instead of sibling or downstream capsules, reducing generated artifact and
review size by an amount that depends on the job boundary.
Scoped JSON filters inventory, findings, and capabilities and emits a stable
`scopeDigest`; `boundaryReferences` preserve excluded local and unknown external
invocations for adapter review.

### 5. Use it in VS Code

The extension provides:

- F12 and hover for COBOL `CALL`, `COPY`, data items, and JCL `EXEC PGM=`.
- Context-aware copybook resolution.
- COBOL/JCL diagnostics, completion, and document symbols.
- Commands to reindex and inspect the dependency graph.
- Six deterministic Language Model Tools for Copilot agent mode.

Run **Mainframe Migration: Reindex COBOL/JCL Workspace** after changing source
library configuration.

Invoke the packaged workflow with a workspace root, JCL boundary, and optional
program selection:

```text
/mainframe-jcl-migration /path/to/workspace MYJOB MYPROG migration/MYJOB
```

When no program is supplied, the workflow selects the sole local candidate or
presents deterministic candidate/risk order and asks when multiple remain.
Alternatively, select the **Mainframe JCL Migrator** custom agent.

## How it works

The toolkit separates deterministic evidence collection from AI reasoning:

1. The language server indexes COBOL programs, copybooks, JCL jobs, calls,
   includes, data declarations, and execution edges.
2. Language Model Tools expose those indexed facts to Copilot.
3. Python commands persist graphs, warnings, contracts, rules, SQL, readers,
   fixtures, scaffolds, and migration reports.
4. The agent reasons over tool output instead of reconstructing dependencies
   from model memory.

Unresolved or unsafe constructs are never silently guessed. Tools emit
structured `BLOCK`, `TODO`, or informational findings for dynamic calls,
missing copybooks, ambiguous libraries, edited PIC clauses, ODO, `REDEFINES`,
transaction dialects, and unterminated SQL blocks.

## Running packaged tools

Prefer the umbrella command:

```bash
mainframe-toolkit run dependency_graph -- . --format json
mainframe-toolkit run impact_analysis -- . --changed ACCTREC --format text
mainframe-toolkit run business_rule_extractor -- app/cbl/VALIDATE.cbl --format markdown
mainframe-toolkit run copybook_to_contract -- app/cpy/ACCTREC.cpy --config mainframe-migration.json --format json
mainframe-toolkit run generate_file_readers -- . --out-dir migration/data --format text
```

The separator `--` ends arguments for `mainframe-toolkit`; everything after it
is passed to the selected tool.

The module form works when the console entry point is not on `PATH`:

```bash
python -m mainframe_modernization_toolkit run dependency_graph -- . --format json
```

Consumer workspaces do not need a `scripts/` directory. Do not locate or run
Python files inside site-packages directly.

## Tool catalog

| Tool | Purpose |
|---|---|
| `migration_preflight` | Validate configured inventory and migration blockers |
| `dependency_graph` | Build CALL, COPY, and EXEC dependency graphs |
| `impact_analysis` | Compute transitive upstream/downstream impact |
| `dead_code_finder` | Report unreferenced programs and copybooks with caveats |
| `sql_extractor` | Extract embedded SQL and host-variable evidence |
| `business_rule_extractor` | Extract reviewable IF/EVALUATE rules |
| `copybook_to_dataclass` | Generate Python models and optional DDL |
| `copybook_to_contract` | Build canonical physical and transport contracts |
| `generate_copybook_fixtures` | Generate deterministic ingestion-only fixtures |
| `generate_file_readers` | Generate binary-safe fixed-width readers |
| `cobol_to_python_skeleton` | Generate disposable traceability scaffolds |
| `jcl_flow_extractor` | Extract JCL steps, DDs, conditions, and flow |
| `migration_complexity_report` | Rank migration effort and risk |
| `characterization_test_scaffolder` | Scaffold golden-master harnesses |
| `generate_program_capsule` | Generate preflight-gated partial evidence capsules |
| `validate_relational_ir` | Validate reviewed relational IR |
| `ir_to_pyspark` | Compile executable relational IR to PySpark |

Each tool is also installed as an individual console entry point, but the
umbrella command is the stable form used by the VS Code extension and agent.

## Language Model Tools

The VS Code extension registers six deterministic tools:

| Tool | Example input |
|---|---|
| `mainframe_getDependencyGraph` | `{"focus":"MYPROG","direction":"both"}` |
| `mainframe_getCallers` | `{"program":"MYPROG"}` |
| `mainframe_resolveCopybook` | `{"copybook":"CUSTOMER-RECORD"}` |
| `mainframe_impactAnalysis` | `{"names":["MYPROG","CUSTOMER-RECORD"]}` |
| `mainframe_runMigrationScript` | `{"script":"migration_preflight","args":[".","--jcl","MYJOB","--program","MYPROG","--scope","program-only","--format","json"],"responseMode":"summary"}` |
| `mainframe_queryArtifact` | `{"runId":"<runId>","operation":"filter","pointer":"/findings","field":"/classification","equals":"BLOCK","select":["/code","/message"]}` |

`mainframe_runMigrationScript` invokes the pip-installed package. It does not
expect repository scripts in the user's project.

`mainframe_queryArtifact` performs deterministic, read-only JSON Pointer,
filter, page, keys, get, and summary queries over workspace or saved run
artifacts. Use a narrow query and pagination instead of grep, search, or reading
an entire JSON artifact into agent context.

Specify exactly one workspace-relative `path` or safe `runId`; `runId` defaults
to `stdout`. The pointer defaults to the document root, offset to `0`, and limit
to `20` with a maximum of `100`. Filters use exact primitive equality and allow
at most 20 projected `select` pointers. Inputs must be regular UTF-8 JSON files
no larger than 100 MiB; canonical real-path checks reject workspace and symlink
escapes. Follow `nextOffset` until it is `null`.

### Agent responses, artifacts, and preflight cache

The runner defaults to `responseMode: "summary"`, returning a compact envelope
with status, findings, counts, digests, `continuationAllowed`, and `nextActions`.
`responseMode: "preview"` adds bounded stdout/stderr excerpts. Complete stdout,
stderr, and result output is always spooled under
`.mainframe-toolkit/runs/<runId>` while the hard artifact limit is not exceeded.
Add `.mainframe-toolkit/` to the consumer repository's `.gitignore` unless run
evidence is intentionally committed.

Identical `migration_preflight` arguments reuse a cached envelope until COBOL,
copybook, JCL/PROC, configuration, artifact fingerprints, or explicit reindex
invalidate it. Agents should obey `continuationAllowed`, perform the listed
`nextActions`, and consume the full artifact instead of rerunning because a
preview was truncated.

`mainframeMigration.maxAgentResponseBytes` limits the serialized response only.
`mainframeMigration.maxArtifactBytes` defaults to a hard 100 MiB combined
stdout/stderr cap. Deprecated `mainframeMigration.maxOutputBytes` remains a
preview compatibility setting and does not terminate the process.

Preflight returns `environmentFacts` from schema-backed configuration such as:

```json
{
  "environment": {
    "cobolDialect": "Enterprise COBOL",
    "compiler": {
      "name": "IBM Enterprise COBOL",
      "version": "6.4",
      "options": ["RENT", "SSRANGE"]
    },
    "sourceFormat": "fixed",
    "runtime": "z/OS batch",
    "runtimeDependencies": {"DB2": "13"},
    "testCommands": ["./run-characterization-tests.sh"]
  }
}
```

Configured values are `KNOWN`. Missing values remain `UNKNOWN`; the toolkit
does not guess them or emit unrelated blockers. Packaged agent guidance allows
one targeted, capped search for each `UNKNOWN`, then requests user evidence.

## Configuration

`mainframe-migration.json` controls:

- Ordered primary and fallback COBOL, copybook, and JCL libraries.
- Source extensions and encoding.
- Known external programs, utilities, and copybooks.
- Generated TODO syntax.
- Physical-to-transport record representations.
- Target strategies and adapters under `targetCapabilities`.

Configuration is authoritative. The tools do not widen searches to guessed
directories when configured resolution fails.

### Capability discovery and target policy

Preflight uses two explicit stages. First, it deterministically discovers only
the capabilities evidenced by the selected JCL and its COBOL/files, then emits
the immutable `capabilities` inventory and `capabilityDigest`. Second, it applies
user-authored `targetCapabilities` policy; discovery never chooses a target
technology.

A concise policy can combine a class default, an exact discovered instance, and
a selector:

```json
{
  "targetCapabilities": {
    "schemaVersion": 1,
    "defaults": {
      "transform.sort_merge": {
        "strategy": "pyspark",
        "adapter": "builtin.pyspark_sort"
      }
    },
    "instances": {
      "storage.indexed_records:dataset:APP.ACCOUNTS": {
        "strategy": "relational_table",
        "adapter": "builtin.relational_keyed_store"
      }
    },
    "selectors": [
      {
        "id": "daily-bulk-loads",
        "capabilityClass": "load.bulk_records",
        "priority": 20,
        "match": {"job": "DAILY*"},
        "policy": {
          "strategy": "relational_bulk_load",
          "adapter": "builtin.bulk_load"
        }
      }
    ]
  }
}
```

Resolution precedence is exact instance, highest-priority matching selector,
capability-class default, then unresolved. Selector array order is irrelevant;
equal-priority selectors with different policies produce `BLOCK` instead of a
guess.

Discovery currently emits `orchestration.batch`, `transform.sort_merge`,
`load.bulk_records`, `storage.indexed_records`, `storage.sequential_records`,
`storage.versioned_generation`, `database.relational`, `transaction.online`,
`messaging.queue`, and `operations.audit`. For a selected JCL, COBOL SQL,
transaction, and queue evidence is limited to local programs in its transitive
CALL/literal dialect-link closure. Messaging requires an exact supported IBM MQ
CALL or CICS `READQ`/`WRITEQ`/`DELETEQ`; generic CALLs and generic CICS blocks do
not qualify. Audit discovery groups `SYSOUT=*`, `SYSPRINT`, and `SYSOUT` DDs per
job step. `security.authorization` is available in policy/schema menus but no
instance is emitted until explicit RACF/security command evidence is parsed.

Built-in adapter IDs are `builtin.relational_keyed_store`,
`builtin.key_value_store`, `builtin.lakehouse_table`,
`builtin.fixed_width_storage`, `builtin.pyspark_sort`, `builtin.sql_sort`,
`builtin.bulk_load`, and `builtin.batch_orchestration`. Preflight records their
declared guarantees in each resolution. Third-party adapters use the
`mainframe_modernization_toolkit.adapters` entry-point group and must pin
`adapterVersion`; unavailable, ambiguous, incompatible, or mismatched adapters
block.

Framework-only descriptors are also available as
`builtin.relational_database_framework`, `builtin.online_transaction_framework`,
`builtin.messaging_queue_framework`, `builtin.audit_framework`, and
`builtin.authorization_framework`. They validate strategy compatibility and
declare only configuration/source-traceability guarantees; they have no provider
implementation, so `generationEligible` remains `false`.

An unresolved policy, or a selected strategy without an adapter, emits a `TODO`
and leaves that capability ineligible for adapter-backed generation. Unaffected
analysis continues, and reviewed relational IR can preserve the missing target
decision as a deterministic code-local TODO using the configured prefix.

The extension invokes `mainframe-toolkit` by default. If VS Code cannot find the
entry point, set `mainframeMigration.executablePath` to its absolute path. Find
it with the Python environment where the package was installed:

```bash
python -c "import shutil; print(shutil.which('mainframe-toolkit'))"
```

## Safety boundaries

- Generated skeletons and capsules are evidence, not complete translations.
- Synthetic fixtures validate ingestion and field boundaries only. They are not
  authoritative expected program outputs.
- Semantic equivalence requires outputs captured from the mainframe or another
  verified implementation.
- Physical binary layouts and text transport layouts are separate contracts.
- PySpark generation requires reviewed, executable relational IR; the compiler
  does not invent business semantics.

## Troubleshooting

### VSIX verification fails on Windows with version 0.1.8

Version 0.1.8 can report a false negative when Windows prevents the verifier
from reopening its temporary VSIX file. Upgrade to 0.1.9 or newer and rerun
the same verification command:

```bash
python -m pip install --upgrade mainframe-modernization-toolkit
mainframe-toolkit vsix verify
```

### `mainframe-toolkit: command not found`

Use:

```bash
python -m mainframe_modernization_toolkit doctor --workspace .
```

Then configure the extension with the absolute entry-point path if needed.

### The agent tries to run `scripts/<tool>.py`

Upgrade the package, reinstall its bundled VSIX, and replace packaged workspace
customizations only after reviewing local changes:

```bash
python -m pip install --upgrade mainframe-modernization-toolkit
mainframe-toolkit vsix export --output mainframe-migration-toolkit.vsix
code --install-extension mainframe-migration-toolkit.vsix --force
mainframe-toolkit workspace init . --force
```

### The VSIX and Python package versions differ

```bash
mainframe-toolkit --version
mainframe-toolkit vsix verify
```

This check applies only to the optional wheel-export route. Export the VSIX
from the same Python environment where the package is installed.

## License

Apache-2.0.
