Metadata-Version: 2.5
Name: norcaster-discover
Version: 0.2.0
Summary: Find the AI systems a company already runs and plan their move behind the Norcaster gateway. Read-only, metadata-only, no runtime dependencies.
Author-email: "Norcaster (Navego AB)" <security@norcaster.com>
License: Apache-2.0
License-File: LICENSE
Keywords: ai-gateway,ai-governance,discovery,inventory,shadow-ai
Requires-Python: >=3.10
Provides-Extra: test
Requires-Dist: pytest>=7; extra == 'test'
Description-Content-Type: text/markdown

# norcaster-discover

Find the AI systems a company already runs, see which ones bypass Norcaster, and get a
draft plan for moving each one behind the gateway.

It runs **inside the customer's environment**, **reads only**, and has **no runtime
dependencies** (Python standard library). Everything it produces is a local file the customer
can review. Only `connect` and `sync` call the Norcaster API, and only to pair and to push those
files (see [Paired collector](#paired-collector)).

The quickest start, with a pairing code from the Discovery setup page (**Run in your network**):

```bash
uvx norcaster-discover connect NCD-XXXXX-XXXXX && uvx norcaster-discover sync --repo .
```

```
scan  ──► inventory.json ──┬─► report.md            (assessment readout)
                           ├─► ingest/*.json        (POST /coverage/ingest → estate-discovery triage)
                           ├─► upload.json          (POST …/estate-discovery/inventory → Discovery candidates)
                           ├─► plan.json            (draft Norcaster API calls, never executed)
                           └─► patches/             (git patch + PR description per system; repos untouched)
```

## Privacy contract

The inventory holds metadata only: file paths, line numbers, SDK/provider/model names,
environment-variable **names**, destination hosts and call counts.

It never records source code, prompts, payloads, key values, client IPs, user names or
URLs. Hard-coded keys are reported by location (`path:line`) only. From network logs,
only AI-related destinations survive; every other host and every other column is dropped
at read time. The ingest payload matches the strict schema of `POST /coverage/ingest`
(bare host, count, window, source) and nothing else.

## Install and run

```bash
pip install ./packages/norcaster-discover     # or: PYTHONPATH=src python -m norcaster_discover
```

```bash
# 1. Scan repos and network logs
norcaster-discover scan \
  --repo ~/src/support-chatbot --repo ~/src/platform \
  --egress proxy-2026-09.csv --egress-source proxy \
  -o inventory.json

# 2. Human-readable inventory for the readout
norcaster-discover report inventory.json -o report.md

# 3. Feed direct provider traffic into Norcaster's shadow-AI triage queue
#    (NORCASTER_INGEST_KEY: an org-bound key with only the coverage:ingest scope, see below)
norcaster-discover ingest-payload inventory.json -o ingest/
curl -X POST https://api.norcaster.com/coverage/ingest \
  -H "X-API-Key: $NORCASTER_INGEST_KEY" -H "Content-Type: application/json" \
  --data @ingest/coverage_ingest_001.json

# 3b. List each app-level system on the Discovery page. upload.json holds names, counts, env var
#     names and the AI provider hosts the code calls (no paths, lines, URLs or code); review it first. Needs the key of an owner,
#     admin or governance user (the ingest key is refused). Or upload inventory.json on the
#     Discovery page, which builds the same body in the browser.
norcaster-discover upload-payload inventory.json -o upload.json
curl -X POST https://api.norcaster.com/orgs/<org id>/estate-discovery/inventory \
  -H "X-API-Key: $NORCASTER_API_KEY" -H "Content-Type: application/json" --data @upload.json

# 4. Draft the migration
norcaster-discover plan inventory.json --org-id org_… --project-id prj_… -o plan.json

# 5. Patches and PR descriptions, one per system (the repos are not modified)
norcaster-discover patch inventory.json --repo ~/src/support-chatbot --repo platform=~/src/platform -o patches/
cd ~/src/support-chatbot && git checkout -b norcaster/support-chatbot-openai \
  && git apply ../patches/support-chatbot-openai.patch \
  && gh pr create --title "Route support-chatbot through the Norcaster gateway" --body-file ../patches/support-chatbot-openai.md
```

`patches/SUMMARY.md` lists every patch with its apply command, and every system that was
not patched with the reason.

## Paired collector

`connect` trades a pairing code from the Discovery setup page (one use, valid 15 minutes) for a
key that carries only `discovery:push`, bound to your organization. It saves the key in
`~/.config/norcaster/discover.json` (or `$XDG_CONFIG_HOME`, `%APPDATA%` on Windows,
`NORCASTER_DISCOVER_CONFIG` to choose), readable by you only.

`sync` takes the same options as `scan`, writes `inventory.json` and `upload.json` to
`.norcaster-discover/` for review, and pushes exactly those: the upload body to
`POST /estate-discovery/inventory` and the log counts to `POST /coverage/ingest`. The systems
appear on the Discovery page at once.

```bash
norcaster-discover sync --repo ~/src/support-chatbot --egress proxy.csv --egress-source proxy
norcaster-discover sync --repo . --dry-run          # write the files, push nothing
norcaster-discover sync --repo . --open-prs         # also open one pull request per system
norcaster-discover sync --kubernetes                # where the cluster's AI keys come from
```

- **The key can push and nothing else.** It cannot read the Discovery page, not even what it
  pushed, and cannot call the gateway or mint keys. It is a service-account key, so every push is
  attributed to the collector. Removing the collector on the setup page revokes it, and it stops
  working when the person who made the code leaves the organization.
- **Only four calls:** `POST /collectors/pair`, `POST /estate-discovery/inventory`,
  `POST /estate-discovery/secret-names` (with `--kubernetes`) and `POST /coverage/ingest`, over
  https (or to localhost).
- **In CI:** `connect <code> --print-key` prints the key instead of saving it. Store it as the
  secret `NORCASTER_DISCOVER_KEY`; `sync` reads it from there (and the API URL from
  `NORCASTER_API_URL`). The GitHub Action in this repository does that on a schedule.
- **`--open-prs`** applies each system's patch on its own branch, `norcaster/<system id>`, and
  opens a pull request with your own `git` and `gh`. It never force-pushes, never pushes another
  branch, and never merges. A repository with uncommitted changes, or a system whose branch
  already exists here or on the remote, is skipped and said so.
- **`--kubernetes`** reads the cluster's Deployments, StatefulSets, DaemonSets, Jobs and CronJobs
  with your own `kubectl` (its current context, or `--k8s-context`; every namespace, or
  `--k8s-namespace`). For every env var named like an AI key, it notes where the value comes
  from:
  - a Secret key, or a whole Secret through `envFrom`;
  - a ConfigMap, where a key is not secret at all;
  - a value written into the spec.

  It writes `k8s-secret-refs.json` and pushes it. Sent: the names of the Secrets or ConfigMaps,
  their keys, the env vars and the workloads, plus a count of the other references. Never a
  Secret, a ConfigMap or any value: those objects are never read, and a value in a spec is
  dropped in memory. The RBAC it needs is `get` and `list` on those workload kinds (and `list
  namespaces` for every namespace), not on Secrets. `--k8s-cluster-name` sets the name shown;
  the default is the context's name.

### The ingest key

Don't use an admin key for step 3. Mint a key that carries only the `coverage:ingest` scope
and is bound to your organization: in the portal under **Settings → API keys** (pick the
organization, tick only `coverage:ingest`), or as a key on a service account under
**Access → Service accounts & agents**. Over the API, with your own key:

```bash
curl -X POST https://api.norcaster.com/auth/api-keys \
  -H "X-API-Key: $NORCASTER_API_KEY" -H "Content-Type: application/json" \
  -d '{"name": "egress-ingest", "org_id": "<org id>", "scopes": ["coverage:ingest"]}'
```

With only that scope the key can call `POST /coverage/ingest` and nothing else. It has no
read access, can't reach the gateway and can't mint or rotate keys. Rotate it by minting a
new one and revoking the old one.

### Log formats

Auto-detected per file: CSV/TSV with a header row, JSON lines, or Squid native access
logs. Host and time columns are found by common names (`host`, `dest_host`, `query_name`,
`url`, …; `timestamp`, `time`, `@timestamp`, …); override with `--host-field` and
`--time-field`. A `count` column is honoured for pre-aggregated exports. Rows without a
parseable host or time are counted as unparsed, never guessed.

Traffic to `api.norcaster.com` (add your own gateway hosts with `--norcaster-host`) is
counted as governed and never sent as ungoverned. Consumer AI apps (chatgpt.com, claude.ai,
…) stay in the local inventory and are left out of the ingest payload unless
`--include-consumer-apps` is given. OpenAI's data-residency hosts (`eu.api.openai.com` and
the others) count as OpenAI traffic. The global Vertex endpoint `aiplatform.googleapis.com`
is kept and sent too, but Norcaster triages it as an unknown host: its Vertex provider can't
route the global location.

## What the plan contains

For each ungoverned system with a supported provider, the draft follows the portal's own
onboarding sequence (`portal/lib/onboarding-ai-setup.ts`):

1. `POST /providers` (once per provider type) with `residency_region`, plus the Azure base
   URL, Bedrock region or Vertex location when the scan saw them; credentials are a manual
   portal step and are never written to the plan. OpenAI gets one provider per OpenAI host
   the systems use: an app on one of OpenAI's data-residency hosts (`eu.api.openai.com`,
   `us.api.openai.com`, …) gets a provider with that host as its `base_url`, so moving it to
   Norcaster keeps its traffic in the region. `residency_region` is prefilled (`eu`, `us`,
   `me`) where OpenAI processes data in the region, and left required where it only stores
   it there. A system whose host the scan can't tell (its code names none and the logs show
   several) gets a required provider choice instead of the default host. The global Vertex
   endpoint (`aiplatform.googleapis.com`, `location="global"`) makes the location a required
   regional value: Norcaster's Vertex provider doesn't support the global location.
2. `POST /deploy` with `model_id`, `project_id` and a draft `purpose`. Data
   classification and legal owner are listed as required inputs.
3. `POST /providers/inference-targets` binding the deployment to the model seen in code.
4. `POST /deployments/{id}/apply-pack` with a suggested pack and the reason for it.
5. A manual step to set `contextual_policy.mode = monitor` with a
   `monitor_mode_deadline_iso` (default 14 days), after which the pack blocks on its own.
6. Code changes per call site: point the client at the gateway, add
   `X-Norcaster-Deployment-Id`, move keys to `NORCASTER_API_KEY` / `NORCASTER_DEPLOYMENT_ID`.

`<<REQUIRED …>>` marks a value only the customer can supply; `{deployment_id:…}` and
`{provider_model_id:…}` are resolved from earlier responses. Systems already routed through
Norcaster, dependency-only findings and unsupported providers are listed under
`not_migrated` with the reason.

### Where a client sends its traffic

An OpenAI or Anthropic client is attributed by its base URL, not by the SDK's name. The
scanner reads the `base_url` / `baseURL` / `openai_api_base` / `api_base` /
`anthropic_api_url` argument of `OpenAI`, `AsyncOpenAI`, `Anthropic`, `AsyncAnthropic`,
`ChatOpenAI`, `OpenAIEmbeddings`, `ChatAnthropic`, `createOpenAI`, `createAnthropic` and
`litellm.completion` / `embedding`. It uses a string literal as is. It resolves an
environment variable (`os.environ[…]`, `os.getenv(…)`, `process.env.…`, `import.meta.env.…`)
from the same component's `.env*`, YAML (including Kubernetes `name:` / `value:` pairs) or TOML
files, and falls back to the lookup's literal default. With no argument, it resolves
`OPENAI_BASE_URL`, `OPENAI_API_BASE` or `ANTHROPIC_BASE_URL` the same way. Only the host is
kept, never the URL.

| The base URL's host is | Provider label | `endpoint_kind` | Migration |
|---|---|---|---|
| unset, or the provider's own API (`api.openai.com`, `eu.api.openai.com`, …) | the SDK's provider (or the host's) | `provider` | as before |
| a Norcaster gateway (`api.norcaster.com`, `--norcaster-host`) | the SDK's provider, status `governed` | `norcaster` | none |
| a third-party AI gateway: `gateway.ai.cloudflare.com`, `api.portkey.ai`, `aigw.portkey.ai`, `ai-gateway.helicone.ai`, `gateway.helicone.ai`, `oai.helicone.ai`, `anthropic.helicone.ai`, OpenRouter, Vercel AI Gateway | `cloudflare_ai_gateway`, `portkey`, `helicone`, `openrouter`, `vercel_ai_gateway` | `third_party_gateway` | `manual_review` |
| private or cluster-local (`localhost`, private IPs, short names, `.local`, `.internal`, `.svc`; the same rule the API uses for provider base URLs) | `self_hosted_openai_compatible` / `self_hosted_anthropic_compatible` | `self_hosted` | `base_url_swap` (OpenAI wire) / `manual_review` (Anthropic wire) |
| any other public host | `openai_compatible_unknown` / `anthropic_compatible_unknown` | `unknown` | `manual_review` |
| built at run time, or an environment variable no file in the component sets | the SDK's provider | `unresolved` | as before |

A component with differently routed clients yields one system per label. The local inventory
records each system's `endpoint_kind`, its `base_url_hosts` and, for a self-hosted server on a
well-known port, an `engine_hint` (11434 → `ollama`, 8000 → `vllm`, 8080 →
`llama_cpp_or_tgi`). None of these three is uploaded: the upload carries only the provider
label.

In the plan, a self-hosted OpenAI-wire server becomes a `huggingface` provider. That adapter
serves any OpenAI-wire server: vLLM, TGI, llama.cpp, Ollama's `/v1`. Its `base_url` is a
required input that the plan never copies from the scan. Norcaster accepts a private origin only
on a self-hosted install or for an organization with a hybrid data plane, and the gateway must
be able to reach it. Gateway, unknown-host and self-hosted Anthropic-wire systems are listed
under `not_migrated`, and in `SUMMARY.md` under "Not patched", with the reason. For a gateway
the reason says: decide whether Norcaster replaces the gateway or sits in front of it.

### Migration types

| Type | Meaning |
|---|---|
| `base_url_swap` | OpenAI/Anthropic SDK in Python, JS/TS, Go, Java/Kotlin or .NET (or LangChain, Vercel AI SDK, LiteLLM): change base URL and headers |
| `client_swap` | Azure OpenAI client (Python/JS `AzureOpenAI`, .NET `AzureOpenAIClient`, Go `azure.WithEndpoint` / `DefaultAzureConfig`): swap to the plain OpenAI-compatible client, then as above |
| `client_rewrite` | Native cloud SDK (boto3 `bedrock-runtime`, AWS SDK): rewrite the client, or leave it listed as ungoverned |
| `manual_review` | The scanner won't guess (Google GenAI SDK, raw HTTP, Anthropic-on-Bedrock/Vertex) |
| `developer_config` | Coding assistants: base URL and headers on developer machines |

## What the patches change

Each rewrite replaces one client construction and nothing else: it points the client at the
gateway (`NORCASTER_BASE_URL`, default `https://api.norcaster.com`), authenticates with
`NORCASTER_API_KEY`, and sends `X-Norcaster-Deployment-Id` from `NORCASTER_DEPLOYMENT_ID`.
Other options (timeouts, retries, extra headers) are kept; a stale `X-Quantlix-*` header is
removed; Azure clients become the plain OpenAI-compatible client, and their endpoint and API
version move into the Norcaster provider config.

| Rewritten automatically | Python | JavaScript / TypeScript |
|---|---|---|
| OpenAI SDK | `OpenAI`, `AsyncOpenAI` | `new OpenAI({...})` |
| Azure OpenAI | `AzureOpenAI`, `AsyncAzureOpenAI` → `OpenAI` | `new AzureOpenAI({...})` → `new OpenAI` |
| Anthropic SDK | `Anthropic`, `AsyncAnthropic` | `new Anthropic({...})` |
| LangChain | `ChatOpenAI`, `OpenAIEmbeddings`, `ChatAnthropic`, Azure variants → OpenAI | — |
| Vercel AI SDK | — | `createOpenAI`, `createAnthropic` |

The official Go, Java / Kotlin and .NET SDKs are found (confidence `high`) and, for the shapes
below, rewritten too. A client counts only in a file that imports the SDK, so a local package or
class with the same name (`openaiformat.NewClient`, your own `ChatClient`) is not taken for one.

| Rewritten automatically | Client construction | What the rewrite does |
|---|---|---|
| openai-go, anthropic-sdk-go | `openai.NewClient(opts...)`, `anthropic.NewClient(opts...)` | Appends `option.WithBaseURL`, `WithAPIKey` (and `WithAuthToken` for Anthropic) and the `X-Norcaster-Deployment-Id` header; options apply in order, so these win. Adds the `cmp`, `os` and `option` imports. |
| go-openai | `openai.NewClient(key)`, key a literal or `os.Getenv("…")` | `openai.NewClientWithConfig` with `BaseURL`. go-openai has no header option, so the key must be bound to one deployment. |
| openai-java, anthropic-sdk-java (Java and Kotlin) | `OpenAIOkHttpClient(Async).fromEnv()` / `.builder() … .build()` in one chain | A builder with `.apiKey`, `.baseUrl` and the deployment header (`.authToken` too for Anthropic); other builder calls are kept. |
| OpenAI .NET | `OpenAIClient`, `ChatClient`, `EmbeddingClient` given a model and a key | An `ApiKeyCredential` and `OpenAIClientOptions { Endpoint = … }`. No header option, so the key must be bound to one deployment. |
| Anthropic .NET (official `Anthropic`) | `new AnthropicClient()` / `new()`, with or without an initializer | `ApiKey` and `BaseUrl` in the initializer; same bound-key rule. |

| Found, changed by hand | Client construction | Why |
|---|---|---|
| openai-go Azure, go-openai Azure | `azure.WithEndpoint(...)`, `openai.DefaultAzureConfig(...)` | Switching an Azure client changes its types. |
| go-openai config | `openai.DefaultConfig(...)` | The config is changed and used elsewhere. |
| OpenAI .NET Responses | `ResponsesClient`, `OpenAIResponseClient` | Its options type differs by SDK version (2.11 changed it). |
| Azure.AI.OpenAI | `AzureOpenAIClient` | As for Azure above. |
| Anthropic .NET, community | `AnthropicClient` with `using Anthropic.SDK;` | A different SDK; labelled `anthropic-sdk-dotnet`. |

The Go output is gofmt-clean. The rewrites were compiled against openai-go v3.69.0,
anthropic-sdk-go v1.78.0, go-openai v1.43.0, openai-java 4.73.0, anthropic-java 2.68.0, OpenAI
.NET 2.14.0 and Anthropic .NET 12.53.0 (2026-10-01).

Embeddings get their own deployment (`NORCASTER_EMBEDDINGS_DEPLOYMENT_ID`), because a
Norcaster deployment binds one model. A client pointed at a self-hosted OpenAI-wire server is
rewritten the same way; the server's origin moves into the Norcaster provider config.

It refuses rather than guesses. These stay with a person, listed in the PR description
under "Needs a human" or in `SUMMARY.md` under "Not patched":

- **Services that also call provider APIs the gateway does not route.** The gateway serves
  chat completions, messages, embeddings and — since the `/v1/responses` change of
  2026-09-29 — the OpenAI Responses API. A service that also uses Assistants, Realtime,
  batches, Anthropic files or Anthropic Managed Agents is not patched, because pointing its
  client at the gateway would break those calls. Against an older gateway, scan with
  `--gateway-apis chat.completions,messages,embeddings`.

OpenAI **files, vector stores, conversations and containers** are different: the same
gateway change forwards them once the deployment opts in
(`action_policies.forward_unchecked`). A service that uses them is patched and planned; the
inventory lists them under `opt_in_api_calls`, the plan adds a step that sets
`forward_unchecked` for exactly those APIs, and the PR description says what to enable and
what it means — the calls are recorded but their content is not policy-checked, and files or
vector stores created directly at the provider must be re-created through the gateway.
`--gateway-opt-in-apis ''` treats them as unsupported instead. The same names belong to
other libraries too (Docker or LXD containers, Slack or Twilio conversations, Gemini, Mistral,
Together or Groq files): a call made on one of their objects is taken to be theirs. What the
receiver holds is worked out where the call is — in its function, then the enclosing ones,
then the module (Python, read with `ast`), or its block and the outer ones (JavaScript /
TypeScript), `self.x` / `this.x` in its class — so one name can hold a Docker client in one
function and an OpenAI client in another. Every other call counts, including one through an
OpenAI client imported from another module; a module-level name assigned both kinds of client
counts (a function may run after either assignment), and so does a name assigned one kind in an
`if` and the other in its `else` before the call, or assigned an expression that may build
either (`docker.from_env() if local else OpenAI()`). A name rebound from another function through
`global`, and code inside JavaScript template strings, are not followed.
- **Clients sent through a third-party AI gateway, to an unknown public host, or to a
  self-hosted Anthropic-wire server** (see "Where a client sends its traffic").
- Positional arguments, `**kwargs` or spread options, options built elsewhere, imports it
  cannot resolve, and code that changed since the scan.
- A provider key that came from a variable rather than the environment (per-user or
  per-tenant keys): the patch is written, but flagged, because every call will now use
  `NORCASTER_API_KEY`.
- Go, Java / Kotlin and C# shapes outside the table above: an option slice (`opts...`), a
  go-openai key in a variable (dropping it would leave the variable unused; openai-go keeps such
  an option and appends its own), a Java builder built elsewhere, a .NET client already given
  options.
- Native cloud SDKs (boto3 `bedrock-runtime`), Google GenAI, LangChain `init_chat_model`,
  LiteLLM, other gateways (Vercel AI Gateway), notebooks, and test code (unless
  `--include-tests`).

Every Python result is re-parsed and every JS/TS result bracket-checked; a rewrite that
fails the check is dropped. Patches contain the customer's code, so unlike the inventory
they are meant to stay in the customer's environment.

## What it can't see

Clients built dynamically or hidden behind internal wrappers; SaaS AI features such as
Microsoft 365 Copilot; personal use of consumer AI apps outside the supplied logs; traffic
that leaves by a path the logs don't cover. The report says so in its last section.

Base URLs are only followed as far as the component's own files. A base URL built at run
time, or an environment variable set outside the component (a CI secret, a Helm value in
another repo), is reported as `unresolved`, and the client keeps its SDK's provider. A
gateway on a custom domain (Cloudflare custom domains, a self-hosted Portkey) is reported as
an unknown or self-hosted host, not as that gateway. Cloudflare's REST route on
`api.cloudflare.com` is reported as unknown: that host is all of Cloudflare's API.

Norcaster's provider usage reconciliation reads OpenAI's Usage API on `api.openai.com`.
OpenAI doesn't document whether usage of a data-residency project (one on
`eu.api.openai.com` or another regional host) is reported there, so reconciliation's coverage
of those projects is unverified.

In Go, Java / Kotlin and .NET, only calls to the four APIs the gateway routes are recognised
(chat completions, messages, embeddings, responses, in each SDK's own spelling). Assistants,
Realtime, batches, files and other provider APIs called from these languages are not seen, so
such a service can look ready to migrate when it isn't: check it by hand. Clients there are
recognised by the SDK's default package name and an import in the same file: a Go import under
another alias, or a .NET `global using` in a different file, is missed (the service then shows up
from its manifest, with confidence `low`).

## Tests

```bash
cd packages/norcaster-discover && python -m pytest
```

CI runs the suite in the `scanner-tests` job on Python 3.10 (the floor) and 3.14, with nothing
installed but pytest, so a dependency beyond the standard library fails there:

```bash
cd packages/norcaster-discover && uv run --isolated --no-project --python 3.10 --with "pytest>=7" python -m pytest -q
```

`test_signatures.py` checks the host mapping against the API's `classify_endpoint` when the
API is importable, and skips otherwise. The suggested-pack rules live in `pack_rules.py`,
which is copied byte for byte to `api/governance/pack_rules.py` so the hosted recommender
suggests the same pack; `test_pack_rules.py` compares the two copies when the API is in the
checkout, and the API's tests fail when they differ.
