Metadata-Version: 2.4
Name: pennsieve-ai-utils
Version: 0.2.0
Summary: Shared blackboard contract, governor LLM client, and literature tools for the Pennsieve AI Co-Scientist stages.
License: MIT License
        
        Copyright (c) 2026 Pennsieve
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
        
Project-URL: Repository, https://github.com/Pennsieve/pennsieve-ai-utils
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pennsieve-llm<0.7,>=0.6.1
Requires-Dist: httpx>=0.27
Requires-Dist: json-repair>=0.30
Requires-Dist: numpy>=1.26
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == "dev"
Requires-Dist: ruff>=0.5; extra == "dev"
Dynamic: license-file

# pennsieve-ai-utils

Shared utilities for the Pennsieve AI Co-Scientist stages:

1. `pennsieve-ai-scout`
2. `pennsieve-ai-hypothesize`
3. `pennsieve-ai-analyze`
4. `pennsieve-ai-write`

Each stage used to carry its own copy of the blackboard schema, the governor
LLM client, the model-role resolver, the agent helpers and the literature
tools. They now import them from here. The package is published to PyPI as
`pennsieve-ai-utils` (same release flow as `pennsieve-llm`: push a `vX.Y.Z`
tag), so the Pennsieve App Store build of each stage can `pip install` it
without GitHub credentials.

See [RUNBOOK.md](RUNBOOK.md) for local stage supervision, stopping, review
bundles and the remaining Pennsieve Test deployment work.

```bash
pip install pennsieve-ai-utils
```

## What's here

| Module | Owns | Used by |
|---|---|---|
| `pennsieve_ai_utils.blackboard` | Canonical `Blackboard` JSON contract, `schema_version` migrations, unknown-field preservation | all stages |
| `pennsieve_ai_utils.llm` | `LLMClient`: governor transport via `pennsieve-llm`, role-based model selection, preflight, self-healing on 403/404, `usage.jsonl` logging | all stages |
| `pennsieve_ai_utils.models` | Role table (`reasoning`, `synthesis`, `bulk`), capability ranking, allow-list discovery, operator pins | `llm` |
| `pennsieve_ai_utils.agents` | `call_json`, `extract_json`, `first_text`, `format_datasets`, attack-tag helpers, `load_prompt` | every agent |
| `pennsieve_ai_utils.processor` | Pennsieve processor env contract (`INPUT_DIR`/`OUTPUT_DIR`/run IDs), SIGTERM handling, upstream blackboard loading, output persistence | every `main.py` |
| `pennsieve_ai_utils.tools` | CrossRef, DataCite, DOI lookup, JATS/Markdown chunkers, Bedrock embeddings + reranker, RAG store reader | scout, hypothesize, write |
| `pennsieve_ai_utils.workflow` | Local four-stage supervisor, handoff gates, review bundles (`python -m pennsieve_ai_utils.workflow`) | operators |

Stage-specific science — pipelines, prompts, Discover/DANDI/OpenNeuro
inspection, notebook execution, manuscript rendering — stays in each app.

## Blackboard contract

```python
from pennsieve_ai_utils import Blackboard

bb = Blackboard.load("blackboard.json")
bb.record_model_assignment(
    stage="hypothesize",
    role="reasoning",
    assignment={"model_id": "us.anthropic.claude-sonnet-4-6"},
)
bb.save("blackboard.json")
```

- One superset schema carries fields produced by every stage.
- `schema_version` enables explicit migrations; documents without a version
  are treated as legacy version 0.
- Unknown fields are preserved at their original object level on a
  load/save round trip.
- Model provenance is cumulative by stage (`model_assignments[stage][role]`)
  rather than overwritten by the next application.

## LLM client

```python
from pennsieve_ai_utils.llm import LLMClient
from pennsieve_ai_utils.agents import call_json

client = LLMClient(stage="hypothesize", roles=("reasoning", "synthesis"))
assignments = client.preflight()          # raises ModelNotAvailable if a floor can't be met
for role, assignment in assignments.items():
    bb.record_model_assignment(stage="hypothesize", role=role, assignment=assignment)

verdict = call_json(client, agent="critic", system=..., user=..., run_id=bb.run_id,
                    model="reasoning")
```

Agents address models by **role**, never by vendor tier. The legacy names
`opus` / `sonnet` / `haiku` still resolve as aliases of
`reasoning` / `synthesis` / `bulk`, and `VP_MODEL_<ROLE>` pins keep working.

Preflight asks the governor's `GET /v1/models` for the allow-list first and
falls back to tripping a `model_not_allowed` 403 on older governors. Each
assigned model is then verified with a one-token call so an IAM block or a
retired Bedrock model fails at startup rather than twenty minutes in.

The client needs `LLM_GOVERNOR_FUNCTION_NAME` (platform-injected) or
`PENNSIEVE_LLM_MOCK=1` for offline tests — see
[LLM access on compute nodes](https://docs.pennsieve.io/docs/llm-access-on-pennsieve-compute-nodes).

## Processor helpers

```python
from pennsieve_ai_utils.processor import (
    ProcessorEnv, install_sigterm_handler, load_upstream_blackboard, write_outputs,
)

install_sigterm_handler()
env = ProcessorEnv.from_environ()
bb = load_upstream_blackboard(env.input_dir / "blackboard.json",
                              stage="analyze", upstream="Hypothesize",
                              run_id=env.execution_run_id)
...
write_outputs(bb, env.output_dir, client.usage_log)
```

These follow the
[Pennsieve processor contract](https://docs.pennsieve.io/docs/pennsieve-processors):
read from `INPUT_DIR`, write to `OUTPUT_DIR`, exit non-zero on failure.

## Releasing

```bash
# bump version in pyproject.toml, commit, then:
git tag v0.2.0 && git push origin v0.2.0
```

`.github/workflows/publish-pypi.yml` builds and publishes via PyPI Trusted
Publishing. One-time setup on pypi.org: add this repository as a trusted
publisher for the `pennsieve-ai-utils` project (GitHub environment `pypi`).
Apps pin `pennsieve-ai-utils>=0.2,<0.3` and pick up patch releases on rebuild.

## Development

```bash
python -m pip install -e ".[dev]"
python -m ruff check src tests
python -m pytest
```
