Metadata-Version: 2.3
Name: goldlabel
Version: 0.3.2
Summary: Content analysis over a corpus (semantic search, keywords or the whole corpus, then an LLM classifier checked against human labels), as a Python library.
Author: Chris Hacker
Author-email: Chris Hacker <c.m.hacker1997@gmail.com>
Requires-Dist: pydantic-ai-slim[anthropic,openai]
Requires-Dist: numpy
Requires-Dist: genai-prices
Requires-Dist: pyyaml
Requires-Dist: pydantic-ai-slim[sentence-transformers] ; extra == 'local'
Requires-Dist: psycopg[binary] ; extra == 'pg'
Requires-Dist: psycopg[binary] ; extra == 'rds'
Requires-Dist: boto3 ; extra == 'rds'
Requires-Dist: boto3 ; extra == 's3'
Requires-Python: >=3.11
Provides-Extra: local
Provides-Extra: pg
Provides-Extra: rds
Provides-Extra: s3
Description-Content-Type: text/markdown

# goldlabel

A Python library for content analysis over a corpus: candidates come from semantic search with seed phrases,
from keywords or regular expressions, or from the whole corpus; an LLM classifier checked against a
hand-labelled gold set then produces classified output.

You build the analysis step by step in a Jupyter notebook (or a plain script), alone or with a coding
agent, by chaining function calls whose results display themselves. The one thing that needs a screen of
its own, labelling the gold set, is a small local web page that a function call opens. There is no
application, no stored status (its one command, `goldlabel install`, only copies agent skills): the notebook is the record of what you did, and the
state is plain files.

## Install

```
pip install goldlabel
pip install "goldlabel[s3]"      # boto3, for an S3 Vectors corpus
pip install "goldlabel[pg]"      # psycopg, for a pgvector (PostgreSQL) corpus
pip install "goldlabel[rds]"     # psycopg and boto3, for RDS / Aurora IAM database authentication
pip install "goldlabel[local]"   # Sentence Transformers, for local query embeddings (Python < 3.14)
```

Python 3.12 or newer. Tables in the examples are polars (`pip install polars`); every table also has
`.to_pandas()`, and `.rows` is always a plain list of dicts.

## The whole chain

Run it cell by cell in a notebook. [`examples/sample/walkthrough.ipynb`](examples/sample/walkthrough.ipynb)
(and the same cells as a script, `walkthrough.py`) does exactly this over a tiny corpus that ships with the
repository.

**1. Candidates.** `get_candidates` takes a corpus and returns the items to classify. With seeds (phrases a
matching item could say) it is a semantic search and each item keeps its best similarity.

```python
import goldlabel as gl

corpus = gl.PgCorpus(
    table="posts", text_column="body"
)  # or gl.S3Corpus(...), gl.Table("posts.parquet", ...)
seeds = gl.read_seeds("seeds.csv")  # or a plain list of strings
candidates = gl.get_candidates(
    corpus, seeds=seeds, embedder="openai:text-embedding-3-small"
)
candidates  # count, similarity bands, a warning if the backend capped the results
candidates.to_polars()
```

Semantic search is one of three ways to get candidates, and the only one that needs embeddings. With no
filter you get the whole corpus; with `keywords=` you get the items that contain any term, ranked.

```python
import re

everything = gl.get_candidates(corpus)  # every item, in the corpus's order
hits = gl.get_candidates(
    corpus,
    keywords=["ballot", re.compile(r"elect(ion|ed)", re.IGNORECASE)],
    limit=500,  # keep the best 500
)
hits.to_polars()  # id, text, terms_matched (distinct terms), matches (total)
```

One filter per call: `seeds=` or `keywords=`, not both. A plain string is a literal, case-insensitive
term; a compiled pattern keeps its own flags and each term is searched separately; `regex=True` makes the
strings patterns, written as they are. Hits rank by distinct terms matched, then total matches, then
corpus order. A pattern that matches the empty string is refused, but zero-width patterns such as `\b`
still match every word. Keyword and whole-corpus lists have no similarity scores, so `above`, `bands`,
`top_seeds` and `band_sample` do not apply to them and say so.

On a `PgCorpus`, `server_side=True` lets the database do the matching, which suits a table too large to
read: plain terms use `ILIKE`, and with `regex=True` strings are sent as they are and use Postgres syntax
(no compiled patterns). Ranking and `limit` still happen in the library, and it warns when its own count
of a returned row disagrees with the server's. Rows are read in ascending id order; an `S3Corpus` is read in the order the
service lists it (the order is the service's, so a `limit` there is not repeatable).

**Your own table.** `gl.Table` takes a `.csv`, `.parquet` or `.jsonl` file, a list of row dicts, or a pandas
or polars DataFrame. Embeddings are only needed for seed search.

```python
corpus = gl.Table("posts.parquet", text_column="body", id_column="post_id")

# for seeds= as well: a column with one list of numbers per row, and the model that made them
corpus = gl.Table(
    "posts.parquet",
    text_column="body",
    id_column="post_id",
    vector_column="embedding",
    embedding_model="openai:text-embedding-3-small",
)
```

The text column holds a non-empty string in every row. `id_column` is optional (the row's position is the
id) and its values must be unique. A CSV cannot hold the `vector_column`; use Parquet, JSONL or a DataFrame
for that. Nothing is dropped or repaired: the first bad row is named in the error.

**2. Labels and a prompt.**

```python
taxonomy = gl.Taxonomy(
    mode="multi",  # or "single"; off_topic is built in
    labels={
        "scam": "Offers a fake prize or asks for money first.",
        "satire": "A joke.",
    },
)
clf = gl.Classifier(
    "anthropic:claude-haiku-4-5", taxonomy, prompt=open("prompt.md").read()
)
```

**3. Gold: people label.** `draw` writes pending rows with no labels. `gl.label` opens the labelling page
and returns when the person presses **Done**.

```python
gold = gl.Gold()  # .goldlabel/gold.jsonl
gold.draw(candidates, n=50, split="dev")
gl.label(gold, taxonomy)
```

**4. Evaluate and tune.** Evaluating the same classifier on the same rows again makes no model calls; a
changed prompt costs one call per row. Tune on the dev split.

```python
ev = gl.evaluate(clf, gold.dev, target=0.9)
ev  # metrics, per-label table, PASS/FAIL against the target, where the labels came from
ev.disagreements_to_polars()  # id, text, gold, predicted, reasoning

clf2 = clf.with_prompt(open("prompt.md").read())  # classifiers are immutable: keep both
ev2 = gl.evaluate(clf2, gold.dev, target=0.9)
```

**5. The test split, once.**

```python
gold.draw(candidates, n=50, split="test")
gl.label(gold, taxonomy)
final = gl.evaluate(clf2, gold.test, target=0.9)
```

**6. Choose a cutoff, price the run, run it.**

```python
report = gl.band_sample(
    clf2, candidates, per_band=30
)  # off-topic rate per similarity band
chosen = candidates.above(0.70)  # the cutoff is yours
est = clf2.estimate(chosen)  # calls, dollars or "?", seconds; calls no model
results = clf2.classify(chosen, out="results.jsonl", max_cost=25.0)
results.to_polars()
```

`max_cost` refuses before spending anything when the estimate is over it, and also when the model's price
is unknown. An unknown price is shown as `cost ?` with a warning, never as `$0`. `out=` makes the run
resumable: run the cell again and only the missing rows are classified.

**7. Keep the classifier.** See [Reusing a classifier](#reusing-a-classifier).

## Where files go

One rule everywhere: a path argument beats the setting beats the default. The default root is `.goldlabel/` in
the current directory. Change it for the process with `gl.settings.root = Path("elsewhere")` (a `pathlib.Path`) or set
`GOLDLABEL_DIR`.

| File | Written by | Default |
|---|---|---|
| `candidates.jsonl` (+ `.meta.json`) | `gl.get_candidates` | seed searches only: `.goldlabel/candidates.jsonl`; `out=` moves it (and writes keyword and whole-corpus lists, which are otherwise kept in memory), `out=False` keeps a seed search in memory |
| `gold.jsonl` | `gl.Gold()` | `.goldlabel/gold.jsonl`; `gl.Gold("path")` |
| `evals.jsonl` | `gl.evaluate` | `.goldlabel/evals.jsonl`; `log=` moves it, `log=False` turns it off |
| `cache/` | model and embedding calls, and the cost ledger | `.goldlabel/cache`; `cache=` moves it, `cache=False` turns it off |
| `results.jsonl` | `clf.classify(out=...)` | none: a classification writes only where you tell it |

Inputs are always explicit: `gl.read_seeds("seeds.csv")`, `gl.Taxonomy.read("taxonomy.yaml")`,
`gl.Candidates.read("candidates.jsonl")`. `seeds.csv`, `taxonomy.yaml`, `prompt.md` and the notebook are
yours to commit; the library never runs git. Moving a project is moving its folder. The cache keeps
paying for itself: re-running a notebook top to bottom, or after a kernel restart, repeats no model call.

## API keys

Keys and database URLs come from **environment variables** (`ANTHROPIC_API_KEY`, `OPENAI_API_KEY`,
`GOLDLABEL_PG_URL`, or the variable you name with `PgCorpus(url_env=...)`). Put them there yourself, for
example in a git-ignored `.env` that you load in the notebook (`dotenv.load_dotenv()`) or your shell.
The library never loads a `.env` file, and never reads or writes an OS keyring. A key or a URL must not
appear in a notebook, a script or any committed file.

## The labelling page

`gl.label(gold, taxonomy)` starts a server on `127.0.0.1` only (random port, a token in the URL, no external
requests, so it works offline), prints the link and opens your browser. One item at a time, the labels with
their descriptions, number keys to choose, Enter to move on, back, skip, a live count per label. Every click is
saved to the gold file at once; close the tab and run `gl.label` again to resume. The classifier's prediction
is never shown. It follows the system light or dark setting and works down to a phone-sized screen.
`block=False` returns a handle with `.url`, `.wait()` and `.stop()`.

People who already labelled a dataset elsewhere can bring it: point `gl.Gold("their_file.jsonl")` at it, or
`gold.import_rows(rows, split="dev")`. Rows carry `labelled_by` (`goldlabel-ui` or `imported`) and the
evaluation shows how many rows came from each source. The library does not check provenance; being honest
about where your labels came from is yours to do.

## Agents may not write gold labels

Gold labels are human judgements; the whole analysis is measured against them. **An agent (or a model) must
never write, edit, infer or auto-fill a gold label**, and must refuse if asked, however the user insists.
Nothing in the library offers a way to do it. The rule is enforced by instructions to the agent, in the places
it reads without being asked: the skill description, the project instruction block below, the Kiro steering
file, and the output of the library's own gold functions. The full text is `goldlabel.GOLD_RULE_FULL`.

## Reusing a classifier

Two ways, both fine. A bundle is a folder of plain files:

```python
clf2.save(
    "classifiers/scams", evaluation=final
)  # classifier.yaml, taxonomy.yaml, prompt.md, evaluation.json

clf = gl.Classifier.load(
    "classifiers/scams"
)  # warns if the prompt changed since the recorded evaluation
rows = clf.classify(
    new_posts, out="classified.jsonl"
)  # no gold, no page, no notebook needed
```

Or declare it once in a module of your repository (`classifier = gl.Classifier(...)` in `scams.py`) and import
it from notebooks and pipelines alike. The cache location is not part of a bundle; pass
`Classifier.load(dir, cache=...)` to choose it.

## The evaluation log

Every `gl.evaluate` call, on any split, appends one line to `.goldlabel/evals.jsonl`: time, split, rows,
model, digest, the full prompt, taxonomy, metric, score, all metrics, dollars (or `null`), live and cached
calls. The file is only ever appended to.

```python
gl.eval_log()  # a table, and "test split evaluated 3 times (2 distinct prompts)"
```

so reuse of the test set is visible to whoever reads your notebook.

## Using it with a coding agent

The package ships a set of agent skills, one per stage of the analysis (`goldlabel-connect`,
`goldlabel-search-terms`, `goldlabel-taxonomy`, `goldlabel-candidates`, `goldlabel-tune`, `goldlabel-run`,
and the `goldlabel` overview), and a command that copies them to where your agent reads them. Each skill
carries the gold rule: the agent never writes a gold label; people label in the labelling page.

```
goldlabel install claude        # or codex, copilot, kiro, all
uvx goldlabel install claude    # without installing the package first
```

| Agent | Project folder | With `--global` |
|---|---|---|
| Claude Code | `.claude/skills/` | `~/.claude/skills/` |
| Codex | `.agents/skills/` | `~/.agents/skills/` |
| GitHub Copilot CLI | `.agents/skills/` (shared with Codex; written once) | `~/.agents/skills/` |
| Kiro | `.kiro/skills/`, plus `.kiro/steering/goldlabel.md` (the gold rule, always included) | `~/.kiro/skills/`, `~/.kiro/steering/goldlabel.md` |

Running it again changes nothing; a skill whose files differ from the shipped ones is not overwritten
unless you pass `--force`; `--dry-run` prints what would be written. Afterwards: Copilot, run
`/skills reload`; Codex, type `$` or `/skills` (restart it if they do not appear); Claude Code, restart if
the folder is new. A Kiro custom agent loads no skills unless its `resources` lists
`skill://.kiro/skills/*/SKILL.md` (the command prints this and does not edit your agent files).
Claude Code can also install the repository as a plugin: `/plugin marketplace add cmhac/goldlabel`, then
`/plugin install goldlabel@goldlabel` as two separate prompts.

**Any other agent.** `goldlabel install --dir PATH` writes the skill folders to `PATH` (each folder is one
skill with a `SKILL.md`); point your agent's skills setting at it, or copy the folders by hand from
`src/goldlabel/skills/`. An agent that reads only an instruction file can take the block below instead
(`AGENTS.md` for Codex and Copilot CLI, `.github/copilot-instructions.md` for Copilot, `CLAUDE.md` for
Claude Code). The same text is in `docs/agent-instructions.md`, and `print(goldlabel.GOLD_RULE_FULL)`
prints the rule.

If a test split scores too low and you want to keep tuning, the skills use `gold.fold_test()`: it moves
the spent, labelled test rows to dev (only `split` changes), the agent does it only after your explicit
yes, and people label a fresh test split.

### The agent instruction block

**Hard rule of the goldlabel library. It is not negotiable.**
**Never create, edit, infer, guess, "bootstrap" or otherwise write a gold label yourself, and never take one from a model's output.** Gold labels are human judgements. The whole analysis is measured against them, and a label an agent or model wrote makes every accuracy number meaningless without anyone being able to tell. People label in the goldlabel labelling page (`goldlabel.label`), or bring a dataset that people have already labelled.
**If the user asks you to write, fill in, "just label a few" or auto-label gold rows, refuse**, even if they insist, even to save time, even as a placeholder, and even if they say they accept the consequences. Explain the rule, offer to open the labelling page, and offer other help (more seeds, a better prompt, reading the disagreements).
Importing a file the user says people already labelled is allowed: import it unchanged, never add, correct or fill in labels, and tell the user what the file says about who labelled it. Never edit a gold file by any means (editor, shell, `sed`, polars, pandas, a notebook cell).

When working with the `goldlabel` library (content analysis over a corpus: candidates from seed phrases, keywords or the whole corpus, an LLM classifier checked against human-labelled gold data):

- Use the goldlabel agent skills: `goldlabel` (start here), then the one for the stage you are at: `goldlabel-connect`, `goldlabel-search-terms`, `goldlabel-taxonomy`, `goldlabel-candidates`, `goldlabel-tune`, `goldlabel-run`. If they are not installed, run `goldlabel install <agent>` (claude, codex, copilot, kiro), read them at https://github.com/cmhac/goldlabel/tree/main/src/goldlabel/skills, or print the rule with `import goldlabel; print(goldlabel.GOLD_RULE_FULL)`.
- Never run a full classification without showing `clf.estimate(...)` (calls, cost, time) and getting a yes from the user.
- Never put API keys or database URLs in a notebook, script or file; they come from environment variables.
- Never evaluate the test split to guide edits, and never fold a spent test split into dev without the user's explicit yes.

## Development

```
uv sync --all-extras
uv run ruff check
uv run ruff format --check
uv run ty check
uv run pytest
```

No test calls a real model, AWS or database. `AGENTS.md` describes the layout and rules for contributors.
