Metadata-Version: 2.4
Name: audit-closed-ai-scientist
Version: 0.2.0
Summary: Benchmark and audit protocol for statistically governed autonomous research systems.
Author: K. Takahashi
License-Expression: CC-BY-4.0
Project-URL: Homepage, https://github.com/kadubon/audit-closed-ai-scientist
Project-URL: Documentation, https://github.com/kadubon/audit-closed-ai-scientist#readme
Project-URL: Repository, https://github.com/kadubon/audit-closed-ai-scientist
Project-URL: Issues, https://github.com/kadubon/audit-closed-ai-scientist/issues
Keywords: ai scientist,autonomous research,auditability,benchmark,e-process,sequential inference
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy<3,>=1.26
Requires-Dist: scipy<2,>=1.11
Requires-Dist: matplotlib<4,>=3.8
Dynamic: license-file

[![DOI](https://zenodo.org/badge/1173085548.svg)](https://doi.org/10.5281/zenodo.18870260)
[![PyPI](https://img.shields.io/pypi/v/audit-closed-ai-scientist.svg)](https://pypi.org/project/audit-closed-ai-scientist/)
# audit-closed-ai-scientist

Audit-Closed AI Scientist is a benchmark for evaluating the statistical validity of autonomous research systems.

The repository demonstrates failure modes of naive AI scientist pipelines (p-hacking, optional stopping, hypothesis shopping) and provides a reproducible implementation of an audit-closed protocol based on transparency logs and sequential e-process inference.

This benchmark is intended for developers of AI scientist systems, autonomous research agents, and self-driving laboratories.

## Start here

An autonomous research system can repeatedly try hypotheses, stop when a promising result appears,
or choose a favorable experimental design after seeing earlier results. These adaptive choices can make
ordinary p-values look more convincing than they are. This project is a **benchmark and protocol
implementation** for measuring and constraining those failure modes.

In the protocol, a candidate set and the evidence used to judge it are committed to a tamper-evident
transparency log. The update decision is then a deterministic function of that declared log. Sequential
e-processes and explicit alpha accounting make the stated benchmark tests valid under continuous
monitoring. The simulations compare that approach with deliberately naive discovery policies.

This is not a tool that establishes a scientific theory, physical-laboratory safety, or deployment
readiness. A successful run supports only the declared synthetic benchmark configuration and its
recorded audit artifacts.

### Install and run

The published distribution works on Windows, macOS, and Linux with Python 3.10 or later.
You only need Python and an internet connection; no Git checkout, conda environment, or API key is
required to run the synthetic benchmark.

```bash
python -m pip install --upgrade audit-closed-ai-scientist
```

Run a small integrated benchmark first. The output is a JSON artifact, so it can be archived or compared
across runs.

```bash
audit-closed-ai-scientist benchmark --runs 20 --output results/benchmark.json
```

Run every simulation with the lightweight reproducible profile:

```bash
audit-closed-ai-scientist run --profile quick --output results/experiment_results.json
```

`benchmark` runs the integrated false-discovery, replication, sequential-evidence, and adversarial
checks. `run` runs the complete simulation suite and writes plots to `figures/`. Use `--profile standard`
for the larger reference configuration. The same command is also available as
`python -m audit_closed_ai_scientist`.

### Choose the right command

| Goal | Command | Expected use |
| --- | --- | --- |
| Check that the installation works | `audit-closed-ai-scientist benchmark --runs 20` | A fast smoke test; its small sample size is not suitable for comparing systems. |
| Compare a system during development | `audit-closed-ai-scientist benchmark --runs 320 --seed 2030` | A fixed, documented integrated benchmark run. |
| Reproduce all repository simulations | `audit-closed-ai-scientist run --profile quick` | A practical local run with saved JSON and figures. |
| Produce the larger reference run | `audit-closed-ai-scientist run --profile standard` | A longer run for a declared reference configuration. |

Every command writes a JSON result file. Keep it together with the command, package version, and any
external-agent adapter revision used to produce it.

### Read the result correctly

The most useful fields in an integrated benchmark result are:

| Field | Meaning in this benchmark |
| --- | --- |
| `false_discovery_rate.audit_closed` | How often the audit-closed policy accepts in a synthetic no-signal setting. Lower is better, subject to the declared configuration. |
| `replicability.audit_replication_success_rate` | How often accepted synthetic signal-world findings reproduce on held-out data. |
| `evidence_stability_under_sequential_testing` | Difference from the declared alpha level under the configured monitoring scheme. |
| `robustness_to_adversarial_experiments` | False-acceptance and tamper-detection behavior under the repository's adversarial simulation. |

Compare results only when the seed, profile/configuration, package version, and external-agent adapter
are recorded. A low benchmark false-discovery rate is not a universal guarantee and does not validate an
unmodeled laboratory, dataset, or agent behavior.

### Use it with your own AI scientist

Provide a callable (or an object with `evaluate_trial`) that returns at least `accepted: bool` and
`winner: str | None`. The harness counts malformed responses as interface failures rather than silently
repairing them.

```python
from benchmarks import benchmark

report = benchmark.evaluate(my_ai_scientist)
print(report["null_world"]["false_discovery_rate"])
```

The complete adapter contract and a working integration guide are in
[`docs/integration_with_ai_scientist_systems.md`](docs/integration_with_ai_scientist_systems.md).

## Agent Skill

This repository includes an Agent Skills-compatible workflow at
[`.agents/skills/audit-closed-ai-scientist/SKILL.md`](.agents/skills/audit-closed-ai-scientist/SKILL.md).
Compatible agents can discover it from this repository, or copy it to a supported user skills directory
such as `~/.agents/skills/audit-closed-ai-scientist/`. The canonical implementation remains this repository.

It is designed for:
- AI scientist developers
- autonomous research agents
- automated scientific discovery pipelines
- self-driving laboratories
- agentic AI research frameworks

The benchmark studies where naive autonomous discovery fails, and how an audit-closed protocol mitigates those failures using:
- transparency logs
- deterministic replay
- sequential e-process inference
- explicit alpha accounting

## Why this benchmark exists

Naive autonomous research loops are statistically unsafe under adaptive search. In this benchmark, naive pipelines are stress-tested under:
- optional stopping
- p-hacking / many-hypothesis search
- candidate and design shopping
- adversarial experiment submissions

The benchmark compares:
- naive discovery policies (publish when significance appears)
- audit-closed policies (`Accept_t = f(Log_0:t)`)

## Reference paper

- Takahashi, K. (2026). *Audit-Closed AI Scientist Protocol*. Zenodo. https://doi.org/10.5281/zenodo.18728589
- Repository manuscript: [`paper/audit_closed_ai_scientist_protocol.tex`](paper/audit_closed_ai_scientist_protocol.tex)

## What is implemented

Core simulations:
1. `simulations/p_hacking_simulation.py`
2. `simulations/candidate_shopping.py`
3. `simulations/optional_stopping.py`
4. `simulations/power_curve.py`
5. `simulations/adversarial_agents.py`

Protocol-extension benchmark modules:
1. `audit_protocol/physical_sentinels.py` + `simulations/sentinel_hierarchy.py`
2. `audit_protocol/drift_localization.py` + `simulations/drift_localization_simulation.py`
3. `audit_protocol/certificate_schema.py` + `simulations/certificate_schema_validation.py`

Integrated benchmark:
- `benchmarks/discovery_validity_benchmark.py`
  - false discovery rate
  - replication success
  - sequential evidence stability
  - adversarial robustness

## Reproduce from source

For contributors or users who need the exact repository state:

```bash
git clone https://github.com/kadubon/audit-closed-ai-scientist
cd audit-closed-ai-scientist
python -m pip install .
audit-closed-ai-scientist run --profile standard
```

### Conda

```bash
conda env create -f environment.yml
conda activate audit-closed-ai-scientist
python run_all_experiments.py --profile standard
```

### Make targets

```bash
make reproduce
make benchmark
make figures
make test
```

Output artifacts:
- `results/experiment_results.json`
- `figures/*.png`
- `result_summary.md`

Figure regeneration from raw JSON:

```bash
python regenerate_figures.py --input results/experiment_results.json --output-dir figures
```

## External agent API

Use the benchmark as an evaluation harness for your own AI scientist:

```python
from benchmarks import benchmark

# my_ai_scientist: callable or object with evaluate_trial(...)
report = benchmark.evaluate(my_ai_scientist)
print(report["null_world"]["false_discovery_rate"])
print(report["signal_world"]["replication_probability"])
```

Adapter contract (minimum return fields):
- `accepted: bool`
- `winner: str | None`

See [`docs/integration_with_ai_scientist_systems.md`](docs/integration_with_ai_scientist_systems.md).

## Security and audit integrity

Implemented safeguards:
- append-only hash-chained transparency log
- Merkle checkpoint verification
- fail-closed transcript checks
- candidate-set commitment before evaluation
- deterministic replay validation
- tamper tests for payload/hash/commitment/alpha rewrites

Security tests:

```bash
python -m unittest discover -s tests -p "test_*.py"
```

## Scientific integrity statement

This repository is a benchmark-focused implementation.
It supports benchmark-level claims about statistical governance and replayability.
It does not claim full deployment implementation of all physical protocol components in the paper.

Detailed audits:
- [`docs/paper_alignment_audit.md`](docs/paper_alignment_audit.md)
- [`docs/security_audit.md`](docs/security_audit.md)
- [`docs/reproducibility.md`](docs/reproducibility.md)
- [`docs/repository_audit_report.md`](docs/repository_audit_report.md)
- [`result_summary.md`](result_summary.md)


## Citation

Software (current release):
- Takahashi, K. (2026). *audit-closed-ai-scientist* (v0.2.0). GitHub. https://github.com/kadubon/audit-closed-ai-scientist/releases/tag/v0.2.0

Archived software record:
- The Zenodo DOI https://doi.org/10.5281/zenodo.18870261 identifies the archived v0.1.0 software record. It is not presented as a DOI for v0.2.0.

Protocol paper:
- Takahashi, K. (2026). *Audit-Closed AI Scientist Protocol*. Zenodo. https://doi.org/10.5281/zenodo.18728589

Machine-readable metadata is in [`CITATION.cff`](CITATION.cff).

## Maintainer release procedure

PyPI Trusted Publishing is configured for this exact identity: repository
`kadubon/audit-closed-ai-scientist`, workflow file `.github/workflows/workflow.yml`, and GitHub
environment `pypi`. No PyPI API token is stored in the repository.

1. Update the version consistently in `pyproject.toml`, `audit_closed_ai_scientist/__init__.py`, and
   `CITATION.cff`.
2. Commit the release, tag it as `v<version>` (for example, `v0.2.0`), and create a GitHub Release from
   that tag.
3. The workflow tests the installed package on Windows, macOS, and Linux; only after those tests and
   the version/tag check pass does its isolated `pypi` job upload the built wheel and source distribution.

The publish job uses OpenID Connect with the required `id-token: write` permission, which matches
PyPI's Trusted Publisher flow. Configure GitHub Environment protection for `pypi` if a human approval
gate is required before publication.
