Metadata-Version: 2.4
Name: refuta
Version: 0.0.3
Summary: Refuta — Refutation-driven Agentic Code Security Platform for DGX Spark
Author: Refuta contributors
Author-email: "DAPADA Inc." <contact@dapada.co.kr>
License-Expression: Apache-2.0
Project-URL: Homepage, https://github.com/memtomem/refuta
Project-URL: Repository, https://github.com/memtomem/refuta
Project-URL: Issues, https://github.com/memtomem/refuta/issues
Keywords: security,code-audit,llm,agent,sast,refutation,sandbox,bubblewrap
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Security
Classifier: Topic :: Software Development :: Quality Assurance
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Dynamic: license-file

# Refuta — Refutation-driven, self-hosted agentic code security platform

**Refuta** (`refuta`) combines local LLM inference (developed on an NVIDIA DGX Spark with `nvidia/Qwen3.6-35B-A3B-NVFP4` served by vLLM on `:8000`) with deterministic code, scanners, and a `bwrap` isolation sandbox. It is an agentic security audit platform that **forms a hypothesis, tries to refute it, and reports only the vulnerabilities and patches that survive**.

The model's output is treated as a claim, not as evidence: every quote it cites is matched against the original file text, line numbers are computed by code, and a finding is only marked `refuted` when the model points to a guard that verifiably exists in the snapshot.

---

## 1. Requirements

- Linux, Python 3.11+. No runtime dependencies (standard library only).
- [bubblewrap](https://github.com/containers/bubblewrap) (`bwrap`) for isolated patch verification. Without it, tests fall back to a plain subprocess and the result records `sandbox_backend: subprocess`.
- For `--runner local`: an OpenAI-compatible endpoint with JSON-schema structured output (e.g. vLLM). Set `QWEN_BASE` (default `http://127.0.0.1:8000/v1`); the model is discovered from `/models`.

```bash
pip install refuta          # or, from a source checkout: uv sync --group dev
```

---

## 2. Package layout

```text
refuta/
├── pyproject.toml          # Package settings and the `refuta` CLI entry point
├── refuta/
│   ├── __init__.py
│   ├── schemas.py          # FindingStatus (9 states), RepoPolicy, ScannerCandidate, EvidenceQuote, CrossReviewRecord, ...
│   ├── repo_provider.py    # Immutable snapshots, .git credential stripping, untrusted-instruction-file quarantine, Loop 1 diff scope
│   ├── scanners.py         # Static/secret/authorization scanners, Semgrep/Gitleaks/OSV/Syft parsers, Loop 4 attack-surface analyzer
│   ├── sandbox.py          # bwrap (--unshare-all) isolated runner for pytest and scanners, SHA256 supply-chain integrity checks
│   ├── verifier.py         # Quote and line-number verifier, refutation adjudicator, cross-model review (option D), bwrap patch verifier
│   ├── runners.py          # LocalRunner (:8000), ClaudeRunner, CodexRunner, FixtureRunner, RunnerRouter, memory guard
│   ├── store.py            # SQLite (WAL) store for evidence, state, cache, and accepted risk + code/deployment/SBOM graph
│   ├── loops.py            # Loop 3 (new-CVE impact), Loop 4 (attack-surface reduction), Loop 6 (post-deploy re-verification), quiet-window guard
│   ├── publisher.py        # GitHub webhook HMAC verification, outbound secret redaction, PR/SARIF publish payloads
│   └── cli.py              # `refuta scan | surface | audit | eval` CLI
├── sample/repo/            # Multi-file mini service for evaluation (includes a prompt-injection trap AGENTS.md)
├── patches/                # CASE-01-IDOR fix patch and verification manifest
├── fixtures/               # Canned responses for offline regression runs
├── expected.json           # Ground truth used only by the independent evaluator (`refuta eval`)
├── results/                # fixture / qwen / qwen-cached run outputs and metrics
└── tests/
    ├── test_refuta.py      # Unit, integration, and security-boundary regression tests
    └── test_local_diagnostics.py # Local-response failure and diagnostic privacy regressions
```

---

## 3. Benchmark and verification results

On a real host vLLM (`nvidia/Qwen3.6-35B-A3B-NVFP4`, `:8000`), the recorded run (`--samples 3`) found **4/4 real vulnerabilities (including 1 `fixed_verified` through bwrap-isolated patch verification)**, **refuted 2/2 safe cases protected by upstream guards (`refuted`, 0 false positives)**, with **0 quote or line-number evidence errors**, **0 raw secret leaks**, and the **untrusted `AGENTS.md` quarantined (`quarantine_verified: true`)**. At recording time, all **22 unit and regression tests passed** (`1.46s`).

| Mode | Vulnerabilities found | Safe cases refuted | False positives | Evidence errors | Secrets exposed | Model calls | Time |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Baseline scanners only (reference) | 4/4 | 0/2 | **2** | — | 0 | 0 | 0.005 s |
| Canned responses (`fixture`) | 4/4 (`fixed_verified` 1) | 2/2 | **0** | 0 | 0 | 5 (stub) | 0.354 s |
| Real Qwen3.6 audit (`local`, `--samples 3`) | 4/4 (`fixed_verified` 1) | 2/2 | **0** | 0 | 0 | 15 | 114.173 s |
| Re-run from cached investigations (`qwen-cached`) | 4/4 (`fixed_verified` 1) | 2/2 | **0** | 0 | 0 | 0 (15 cache hits) | 0.296 s |

The `local` row is one recorded run. vLLM output is not bit-exact at temperature 0, so a single investigation can flip: on 2026-10-26, single-sample runs (`--samples 1`) matched the ground truth in 6 of 8 runs, and both failures left `CASE-02-GUARDED-ADMIN` `confirmed` instead of `refuted` (1 false positive). With `--samples 3` (strict-majority vote over independently adjudicated investigations), 10 of 11 runs matched; the other had no majority on CASE-02 and escalated it to `needs_review`. No run in either mode missed a vulnerability, produced an evidence error, or exposed a secret.

- Real Qwen run: [report](https://github.com/memtomem/refuta/blob/main/results/qwen/report.md), [structured result](https://github.com/memtomem/refuta/blob/main/results/qwen/result.json), [SARIF](https://github.com/memtomem/refuta/blob/main/results/qwen/sarif.json), [metrics](https://github.com/memtomem/refuta/blob/main/results/qwen/metrics.json)
- Cached re-run: [report](https://github.com/memtomem/refuta/blob/main/results/qwen-cached/report.md) and [metrics](https://github.com/memtomem/refuta/blob/main/results/qwen-cached/metrics.json)
- Canned responses: [report](https://github.com/memtomem/refuta/blob/main/results/fixture/report.md) and [metrics](https://github.com/memtomem/refuta/blob/main/results/fixture/metrics.json)

---

## 4. CLI usage

Run from a source checkout (`uv sync --group dev` creates `.venv`). After `pip install refuta`, use `refuta` instead of `.venv/bin/refuta`.

```bash
# 1) All unit and integration tests
.venv/bin/pytest -q

# 2) Static, secret, and authorization scanners only
.venv/bin/refuta scan \
  --repo sample/repo \
  --out /tmp/refuta-scan

# 3) Loop 4 attack-surface analysis (unauthenticated endpoints, debug routes, privileged routes)
.venv/bin/refuta surface \
  --repo sample/repo \
  --out /tmp/refuta-surface

# 4) Full security audit on the local vLLM (Qwen3.6 :8000) + bwrap patch verification
.venv/bin/refuta audit \
  --repo sample/repo \
  --runner local \
  --samples 3 \
  --patches patches/manifest.json \
  --out results/qwen

# 5) Evaluate against the independent ground truth (expected.json)
.venv/bin/refuta eval \
  --result results/qwen/result.json \
  --expected expected.json \
  --repo sample/repo

# Offline, no model needed: --runner fixture --fixture fixtures/responses.json
```

`audit` defaults to 3 independent investigations per candidate for `--runner local`, with a strict-majority vote; other runners default to 1. Set `--samples N` to override either default (`N >= 1`). On an uncached audit, the local default uses about three times as many model calls as `--samples 1`. Existing sample-0 cache entries are reused; only missing samples require new calls.

Local investigations run sequentially, and remaining samples are still attempted after a failure. The HTTP timeout (`RepoPolicy.max_timeout_sec`, default 120 seconds) applies to each request rather than setting a deadline for the entire audit. A stalled endpoint can therefore cause three consecutive timeout waits per candidate with the local default. Use `--samples 1` to reduce the number of requests when troubleshooting the endpoint.

`audit` exits 0 when the run completes and 2 when any investigation sample fails (`partial`), even if the other samples reach a strict majority and decide the finding. Check `result.json.errors` for failed samples and each finding's `sample_statuses` for the vote. `eval` exits 1 when the result does not match the ground truth.

### Local-response diagnostics

When a local model response has invalid UTF-8, malformed JSON, an invalid response envelope, or investigation content that is not a JSON object, the audit automatically saves a diagnostic under `<out>/diagnostics/<run_id>/<candidate_id>-sample<N>.json`. The sample number is zero-based. Each entry in `result.json.errors` points to its diagnostic; the same error list is saved in the audit's SQLite record. Failed responses are not cached. Successful responses create no diagnostic files, and a diagnostic write failure is recorded while the audit continues as `partial`.

Diagnostics contain the model, failure stage, response text, completion reason and token usage when available, and candidate/rule/sample identity. JSON errors also include the exact parser input after redaction, with character position and one-based line/column coordinates. Secrets in that parser input use equal-length `*` masks so the coordinates remain accurate after code-fence or reasoning-block removal and fallback extraction. Other diagnostic text uses the existing hashed secret markers, including for tokens split by Unicode escapes inside nested JSON or following short escapes such as `\n`. Diagnostic masking conservatively recognizes token prefixes within surrounding text and extends masking beyond scanner length caps; it can redact more text than the source scanner. The serialized diagnostic is checked again before writing.

Diagnostic directories are restricted to the owner (`0700`), files use `0600`, and symlink directories are rejected. Diagnostics are ignored by Git. Redaction covers the existing GitHub-token, `sk-`, and AWS-access-key families; truncated tokens or tokens interrupted by invalid encoding may not match. A field with unresolved escape obfuscation is replaced entirely with a redaction marker; parser input instead uses stars while preserving its length and line breaks. Diagnostics can still contain source code or other sensitive values and should be handled as sensitively as the audited snapshot. These permissions apply to diagnostics, not to the audit's other output files. HTTP errors and timeouts retain their existing error handling without response-body capture.

---

## 5. Security and verification boundaries covered by tests

1. **Untrusted instruction-file quarantine and credential stripping (`RepoProvider`)**: when a snapshot is created, tokens in `.git/config` are removed and `AGENTS.md`, `CLAUDE.md`, `.claude/`, `.codex/`, and `.github/copilot-instructions.md` are quarantined automatically.
2. **Loop 1 diff scope (`compute_diff_scope`)**: computes exactly which files were added, modified, or deleted between two snapshots.
3. **Immediate secret masking (`scanners.py`)**: `ghp_...`, `sk-...`, and `AKIA...` values are replaced with `REDACTED_SHA256:<12 hex>` as soon as they are detected; tests verify that no raw value reaches prompts, SQLite, reports, or SARIF.
4. **External scanner parsers (`parse_semgrep_json`, `parse_gitleaks_json`, `parse_osv_scanner_json`, `parse_syft_cyclonedx_json`)**: normalize Semgrep, Gitleaks, OSV-Scanner, and Syft CycloneDX SBOM output into the standard schema, masking secrets automatically.
5. **Deterministic evidence verifier (`verifier.py`)**: checks that the `vulnerable_quotes` and `guard_quotes` the model cites exist verbatim in the original files, and computes line numbers itself. Missing files, path escapes (`../`), and altered code quotes are rejected immediately as `invalid_evidence`.
6. **Strict refutation**: even if the model claims `is_refuted: true`, the refutation is rejected (`needs_review`) unless it presents guard-code quotes (`guard_quotes`) verified against the snapshot files.
7. **Option D cross-model review (`adjudicate_cross_review`)**: adjudicates agreement and disagreement between the primary analyst and a second refuting reviewer, switching to `refuted` only when the refuting side presents a valid guard-code quote.
8. **Sample voting (`adjudicate_samples`, `audit --samples N`)**: runs N independent investigations per candidate, adjudicates each one separately, and keeps a status only when a strict majority reaches it; anything else escalates to `needs_review`. A sample counts as `refuted` only if its guard quote verified, so a single-sample refutation without evidence cannot win the vote. Per-sample statuses are kept in `sample_statuses`, and each sample has its own cache entry.
9. **`bwrap` (bubblewrap) isolated patch and scanner runner (`sandbox.py`)**:
   - Isolates test and scanner processes with `/usr/bin/bwrap` (`--unshare-all --die-with-parent --new-session --ro-bind / / --tmpfs $HOME --ro-bind <venv> <venv> --ro-bind <base interpreter under $HOME, e.g. uv Python> --tmpfs /tmp --ro-bind <work_repo> /tmp/work --clearenv`).
   - Demonstrated inside the sandbox: (1) `$HOME` is not accessible, (2) the source under `/tmp/work` cannot be written (`Read-only file system`), (3) neither the host's `127.0.0.1:8000` nor external networks are reachable (`Network is unreachable`).
   - A finding is promoted to `fixed_verified` only when all four conditions hold: (1) test files (`tests/`) are unmodified, (2) the security test fails before the patch (the vulnerability reproduces), (3) the security test passes after the patch, (4) the functional regression tests pass after the patch.
   - Execution is blocked when an external binary or model weight does not match its pinned SHA256 digest (`verify_artifact_sha256`).
10. **Default deny for sending code off-host, and a memory guard (`RunnerRouter`)**: with `local_only=True` (the default), `ClaudeRunner` and external `CodexRunner` calls are blocked; local inference is refused with `InsufficientMemoryError` when `MemAvailable` is below the threshold (`min_mem_available_gb`, default 8).
11. **Accepted-risk expiry and invalidation on file change (`RefutaStore`)**: an `accepted_risk` exception is released and the finding restored to `confirmed` when the exception expires or the SHA256 of the target file changes.
12. **Continuous security loops and publishing safeguards (`loops.py`, `publisher.py`)**:
    - **Loop 3 (new-CVE response)**: queries the deployment/SBOM graph to identify affected services, environments, and runtime images immediately, without any code change.
    - **Loop 4 (attack-surface reduction)**: automatically identifies unauthenticated public routes, debug endpoints, and privileged admin paths.
    - **Loop 6 (post-deploy re-verification)**: fails while a pre-patch commit is still running in production, and completes only once the patched commit is deployed and passes its health check.
    - **Quiet-window guard (`QuietWindow`, `is_in_quiet_window`)**: tells heavy scheduled audits when to yield the host, e.g. to another model server sharing the GPU. Windows are configured by the caller (time zone, start/end, weekdays); the test uses a DGX Spark schedule (Mon/Thu 03:50–09:00 KST, daily 06:50–07:30 KST) as its example.
    - **Publishing safeguards**: GitHub webhook HMAC-SHA256 signature verification, blocking publication on a commit SHA mismatch (`StaleCommitPublishError`), and automatic redaction of secret tokens in outbound text.
13. **Local-response diagnostics (`tests/test_local_diagnostics.py`)**: redacts supported tokens (including overlapping patterns and Unicode escapes), preserves parser coordinates, separates runs, restricts file permissions, rejects symlink directories, and keeps partial audits usable when diagnostic writes fail. Non-object local responses are rejected before caching, and malformed legacy cache entries are re-investigated.

---

## License

[Apache License 2.0](https://github.com/memtomem/refuta/blob/main/LICENSE). Copyright 2026 DAPADA Inc. and Refuta contributors.
