# Secure FL Benchmarks

This directory contains the maintained, reproducible benchmark suite for evaluating the Secure FL trade-off between model quality and cryptographic overhead.

## Maintained Scripts

- `canonical_benchmark.py`: reproducible baseline-vs-secure benchmark pipeline
- `run_all_benchmarks.py`: runs the maintained benchmark suites and writes a summary

Legacy benchmark scripts were removed to avoid drift and conflicting results.

## What This Suite Answers

The benchmark is designed to answer:

1. How much accuracy changes when ZKP is enabled (`low`, `medium`, `high` rigor)
2. How much runtime overhead ZKP introduces compared to baseline
3. Whether runs are stable across repeated seeds (mean/std/95% CI)
4. Whether benchmarked ZKP runs used real proof payloads (optional strict mode)

## Quick Start

```bash
# Fast smoke run (2 clients, 2 rounds)
python experiments/canonical_benchmark.py --quick

# Reproducible run with repeats and strict proof requirement
python experiments/canonical_benchmark.py \
  --datasets mnist synthetic_small \
  --num-repeats 5 \
  --seed 42 \
  --seed-step 17 \
  --require-real-proofs

# Run all maintained suites
python experiments/run_all_benchmarks.py --quick
```

## Output Files

Each run writes two artifacts:

- `benchmark_results.json`: full per-seed/per-config runs with metadata
- `tradeoff_report.json`: aggregated baseline-vs-secure trade-off report

Both files include metadata for reproducibility (timestamp, git commit, run args, torch version).

## Strict Proof Mode

Use `--require-real-proofs` to fail runs when ZKP configs do not emit real proof payloads.

This protects against accidentally benchmarking fallback paths where proof generation is unavailable.

## Notes

- Benchmarks are standalone scripts and are not shipped in the package CLI.
- For publication-grade claims, run with repeats (`--num-repeats >= 5`) and strict proof mode.
