Metadata-Version: 2.4
Name: semantr
Version: 1.0.0
Summary: Discover what your organisation is actually using AI for, from Bedrock invocation logs.
License: Apache-2.0
Requires-Python: >=3.13
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: litellm>=1.95.0
Requires-Dist: boto3>=1.34
Requires-Dist: numpy>=1.26
Requires-Dist: jinja2>=3.1
Requires-Dist: click>=8.1
Requires-Dist: pydantic>=2.6
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == "dev"
Requires-Dist: httpx>=0.27; extra == "dev"
Dynamic: license-file

# Semantr

**Semantic analysis of your AI logs — turned into intelligence.**

Cost tools read your logs and tell you what you spent. Semantr reads the same logs and
tells you what the money *bought*: what kinds of work your organisation is doing with AI,
which of it is automated, which is people experimenting, and where it could cost less.

The difference is semantics. Grouping by API key is arithmetic. Working out that four
hundred conversations are all customer support triage — and that nobody knew a team was
running document extraction — takes reading what was actually said.

```bash
uvx semantr
```

That is the whole thing. It asks AWS where your Bedrock invocation logs are, prices the
run, and analyses them — no arguments, no configuration, and no account.

No `uvx`? It ships with [uv](https://docs.astral.sh/uv/)
(`curl -LsSf https://astral.sh/uv/install.sh | sh`), or use `pip install semantr && semantr`.

Runs where your logs already are. Raw logs, completions and whole conversations never
leave your machine — the one exception is naming, which sends short snippets to a model,
and `--classifier-model` points that at your own.

---

## What it does that a cost dashboard cannot

| | |
|---|---|
| **Reconstructs conversations** | Bedrock logs are stateless. Semantr rebuilds the threads, so cost is per conversation rather than per request |
| **Discovers the work** | Groups by meaning and names each group from its own contents. No taxonomy to configure, and no `other` bucket to hide in |
| **Tags what it finds** | Coding vs not, and repeatable vs experimental — measured from the data, not guessed |
| **Prices the finding** | Spend against each discovered use case, with quantified improvements |
| **Costs cents** | Naming runs ~150 model calls whatever the corpus size. Under $1 for a million invocations |

## The idea, in three steps

```
1. SESSIONISE   group invocations into conversations
2. DISCOVER     find what kinds of work the org is doing, and name them
3. AGGREGATE    roll those use cases up to teams, apps, cost, time
```

Everyone does cost. Nobody does semantic analysis. Cost attribution falls out of the same
data almost for free — the reason to run this is that it can tell you *what the AI is
being used for*.

**Discovery, not classification.** There is no fixed taxonomy and therefore no `Other`
bucket. Conversations are clustered by what they have in common and each cluster is named
from its own contents, so the categories come from what your org is actually doing —
including the things nobody thought to ask about.

---

## Try it in one minute

```bash
uvx semantr
```

```
semantr 1.0.0 — looking for your invocation logs
  found invocation logs at s3://acme-bedrock-logs/AWSLogs/ (ap-southeast-2)

Semantr will read those logs, rebuild the conversations, and name the
kinds of work it finds. Reading is free; naming costs about a cent.
Continue? [Y/n] y
```

**Nothing to look up.** Bedrock records its own logging destination, so the bucket comes
from `GetModelInvocationLoggingConfiguration` rather than from you. If invocation logging
is switched off, or delivering to CloudWatch instead of S3, it says exactly that — those
are the two reasons the tool cannot help yet, and neither should arrive as "no records
found".

Naming the clusters is the one step that calls a model. The first run is served on
Semantr's, capped, with no account. To keep the snippets in-house instead, pass
`--classifier-model` — see [Pointing it at a model](#pointing-it-at-a-model).

Reading another account's logs:

```bash
uvx semantr --profile prod
uvx semantr --role-arn arn:aws:iam::123456789012:role/SemantrRead
```

---

## What it costs to run

Embeddings run about two orders of magnitude cheaper per token than generation, and
clustering is local compute. Naming happens **once per cluster** — around 150 calls
whether your corpus is one million invocations or fifty million.

| Component  | Scales with       | 1M invocations | 50M invocations |
|------------|-------------------|----------------|-----------------|
| Embedding  | Distinct snippets | ~$0.10         | ~$2             |
| Naming     | Cluster count     | ~$0.50         | ~$0.50          |
| **Total**  |                   | **under $1**   | **~$3**         |

Naive per-invocation classification would be roughly $600 and $30,000 respectively.

*Figures are Anthropic first-party rates. Bedrock is partner-priced separately — check
before quoting these.*

**You never have to take that on trust:**

```bash
semantr analyse ./logs --dry-run
# 48,200 distinct snippets to embed (from 611,203 sessions, 92% collapsed),
# ~150 clusters to name, est. $0.60
```

`--dry-run` makes no inference calls at all.

---

## Pointing it at a model

Naming the discovered clusters is the one step that uses a model. It runs **once per
cluster** — about 150 calls whether your corpus is a million invocations or fifty million —
so this costs cents, not dollars.

There are two ways to supply that model, and no third:

- **Semantr's**, the default. The first run needs no account at all — it is served from a
  capped free allowance, which is what makes `uvx semantr` work on its own. An API key
  raises the limit. Short snippets — a truncated system prompt, tool names, the first turn
  — are sent to it, and that is the only prompt text that ever leaves your machine.
- **Yours**, via `--classifier-model`. The snippets go to your provider instead of ours;
  point it at a local model and nothing leaves your network at all.

There was once an offline mode that named clusters from term frequency alone. It was
removed because it did not work: measured against the same corpus, clustering was
identical but naming produced nine near-duplicate names out of twenty-one, with seven
distinct kinds of work all labelled "Acme Internal Helpful" — term frequency latches onto
shared boilerplate in the system prompts. A model gets the same corpus down to one
near-duplicate. Shipping the offline path would have meant shipping a report that looked
authoritative and said nothing.

### Bedrock

You already have credentials if you have these logs. `semantr` uses your ambient AWS
credentials — the same ones `aws` uses — and asks for nothing else.

```bash
export AWS_PROFILE=your-profile        # or AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY
export AWS_REGION=us-east-1

semantr analyse ./logs \
  --classifier-model bedrock/anthropic.claude-3-5-haiku-20241022-v1:0 \
  --embedding-model  bedrock/amazon.titan-embed-text-v2:0
```

You need `bedrock:InvokeModel` on those two models, and model access enabled for them in
that region.

### Anthropic

Put the key in a `.env` file where you run from, or export it. `semantr` reads `.env`
automatically, and existing environment variables win.

```bash
echo 'ANTHROPIC_API_KEY=sk-ant-...' > .env

semantr analyse ./logs --classifier-model anthropic/claude-haiku-4-5
```

Anthropic has no embeddings endpoint, so embeddings run **locally** and only the ~150
naming calls leave your machine. That is also the better posture if you would rather keep
prompt snippets in-house.

### OpenAI, Azure, Vertex — or your own hardware

```bash
echo 'OPENAI_API_KEY=sk-...' > .env
semantr analyse ./logs --classifier-model openai/gpt-4o-mini

# entirely on your own hardware
semantr analyse ./logs \
  --classifier-model ollama/llama3 --embedding-model ollama/nomic-embed-text
```

Any [litellm](https://github.com/BerriAI/litellm) model string works and its credential
conventions apply, so if a provider works with litellm it works here. Cost figures come from
litellm's maintained pricing registry, so they stay current without us shipping a release.

### Check the bill before you spend it

```bash
semantr analyse ./logs --classifier-model anthropic/claude-haiku-4-5 --dry-run
# 48,200 distinct snippets to embed (from 611,203 sessions, 92% collapsed),
# ~150 clusters to name, est. $0.60
```

`--dry-run` makes no inference calls, and errs high rather than low.

---

## The report

A clinical header — source, window, total spend, use cases, conversations, invocations,
cost per conversation — then two charts that answer at a glance what a table cannot: how
spend divides across use cases, and a 2x2 of **coding vs not** against **repeatable vs
experimental**, which is where "what is this org automating" becomes visible.

Under that, five tabs:

1. **Use cases** — one filterable, sortable row per discovered use case with its summary,
   tags and spend. Open a row for models, tools, tokens, dates, cost improvements and
   sample prompts
2. **Details** — the inventory: models, tools, applications and IAM principals, with counts
3. **Risk flags** — PII, secrets and anomalies, as counts and labels
4. **Conversations** — the costliest, session-level rather than request-level
5. **Run** — what was read, what it cost, what was skipped

Leading with total spend would make this a FinOps tool with a novelty feature. Leading
with use cases makes the spend figures supporting evidence.

The HTML is self-contained: inline CSS, embedded data, **zero external requests**. It
renders in light and dark and can be emailed as-is. Because it is meant to be forwarded,
tabs never lose content — the whole report is visible with JavaScript disabled, and
printing forces every tab and every expanded row open.

See [`examples/sample-report.html`](examples/sample-report.html).

---

## Hosted history and trends

The CLI and the HTML file are free and always will be. The hosted control plane adds what
a file cannot: history, trends between runs, and a shareable link.

Generate an API key in the Semantr web UI, then:

```bash
semantr configure --api-key smtr_...
semantr analyse ./logs --json-out report.json    # pushes automatically from now on
```

**Once a key is configured, runs push by default.** Configuring the key is the consent;
having to remember a flag every time is not a privacy control, it is a papercut. Every run
says what it is about to send, and `--no-push` keeps a run entirely local.

**`--push` sends aggregates only.** Prompt and completion text is stripped before
anything leaves the machine. `--include-examples` opts in to sending sample prompts; it
is off by default and the stored report is marked either way.

### Sharing

`semantr serve report.json` opens the report locally with a **Share** button. Pressing it
publishes that snapshot and hands back a public URL you can send to anyone.

**Share works with no account.** Without an API key it publishes through the anonymous
endpoint — rate limited, and the link expires. You should not have to sign up before you
can show a colleague what you found.

The button is deliberately *not* in the written HTML file. Publishing needs your API key,
and a key inside a file designed to be forwarded would travel to everyone who received
it — where any of them could press Share and publish your data. Served locally, the key
stays in your config and the browser only ever talks to your own machine.

The control plane is a separate project: **[semantr-home](https://github.com/allanitis/semantr-home)**.
Run it yourself with `docker compose up`, or use the hosted one.

---

## What crosses the boundary

This is the whole egress schema. It is here, in the README, rather than buried in a
privacy policy.

**Crosses (only with `--push`, or an API key configured):**

| | |
|---|---|
| Run metadata | run id, timestamp, duration, config |
| Ingest counts | records read, skipped, skip reasons |
| Use cases | discovered names, descriptions, categories, confidence |
| Volumes | session and invocation counts, token counts |
| Cost | spend totals and per use case, model, app, day |
| Attribution | execution-role **names** (account ids are removed) |
| Risk | finding labels and counts |

**Never crosses:**

- Prompt text
- Completion text
- Sample prompts
- The opening turn of any conversation

Verify it yourself before sending anything:

```bash
semantr telemetry --show --payload-file payload.json
```

**Call-home is opt-in.** With no API key configured, `semantr` makes no network calls at
all — not a first-run ping, not a version check, nothing. Improving the classifier is a
separate, explicitly consented flag and never a side effect of use.

---

## Commands

| | |
|---|---|
| `semantr analyse SOURCE` | The main command. A directory or `s3://bucket/prefix` |
| `semantr analyse … --dry-run` | Estimate the cost, spend nothing |
| `semantr generate-fixtures DIR` | A synthetic corpus to try it against |
| `semantr telemetry --show` | Print the exact call-home payload |
| `semantr configure --api-key …` | Store control-plane settings |
| `semantr serve payload.json` | Open the report locally with a Share button |
| `semantr share payload.json` | Publish anonymously, no account, get a URL |

---

## How sessionisation works

Bedrock logs are stateless — a twenty-turn conversation is twenty unrelated records. But
they are *self-describing*: invocation N+1's `messages` array contains N's messages, plus
N's output, plus the new turn. So this is an exact join, not a heuristic:

```
state_after(X)  = hash(X.messages + [X.output])
state_before(Y) = hash(Y.messages[:-1])
parent(Y)       = the X where state_after(X) == state_before(Y)
```

Index every invocation by `state_after`, then look up each invocation's `state_before`.
The result is a forest; each tree is a conversation.

Four cases are deliberately not handled: retries appear as sibling branches, compacted
history starts a new conversation, branching is kept as branches, and a missing parent is
a root. All four **over-count conversations and none of them mis-attributes cost or use
case** — that is the right direction to be wrong in, and it keeps the algorithm one
paragraph long.

---

## Development

```bash
python3.13 -m venv .venv && .venv/bin/pip install -e '.[dev]'
.venv/bin/pytest
```

227 tests, about fifteen seconds. The whole suite runs with no credentials, no network
and no services — a deterministic fake provider stands in for inference, which is why it
is safe to run anywhere. That fake is a test fixture, not a product mode.

Requirements live in [`requirements.md`](requirements.md); decisions that shaped them are
in [`.claude/audit/decisions.md`](.claude/audit/decisions.md).

### Layout

```
semantr/
  adapters/     bedrock.py — S3 and local; fixtures.py — synthetic corpora
  core/         schema, normalise, cost, attribution
  inference/    provider seam: litellm, Semantr-hosted, test fake, cache warm-up
  pipeline/     sessionise → snippet → dedup → cluster → name → aggregate
  report/       payload.py (JSON, the contract) → render.py (self-contained HTML)
  cloud/        client for the control plane (the service lives in semantr-home)
```

The load-bearing decision: `report/payload.py` emits JSON and the frontend renders from
it, so the static file, the local server and the hosted app share one renderer.

---

## Status

Version 1. Bedrock invocation logs only.

Known limits, carried from the design and needing real data to close:

- Requires that Bedrock invocation records carry the caller's IAM ARN — assumed,
  unverified against a live account
- Cost figures use Anthropic first-party rates; Bedrock partner pricing is separate
- Cluster granularity has not been validated against a real corpus. The acceptance test
  is that a human recognises their own company in the cluster list
- `head` vs `head_tail` snippet extraction has not been A/B'd on real data

## Licence

Apache 2.0.
