Metadata-Version: 2.5
Name: robotensor-horizon-engine
Version: 0.2.0
Summary: The Robotensor Horizon competition: epochs, demonstration pools, evaluation, scoring and a signed store.
License: Apache-2.0
License-File: LICENSE
Requires-Python: >=3.10
Requires-Dist: numpy>=1.23
Requires-Dist: pynacl>=1.5
Requires-Dist: pyyaml>=6.0
Requires-Dist: robotensor-horizon-protocol
Provides-Extra: dev
Requires-Dist: pytest>=8; extra == 'dev'
Requires-Dist: ruff==0.16.7; extra == 'dev'
Provides-Extra: mirror
Requires-Dist: huggingface-hub>=0.23; extra == 'mirror'
Description-Content-Type: text/markdown

# horizon-competition

The Robotensor Horizon competition: participants fine-tune
[Zero-WAM](https://huggingface.co/robbyant-research/zero-wam-posttrain-robotwin), and every week
each submission is evaluated on the **same demonstrations in the same scenes**, across several
benchmark axes at once. A model has to get better without getting worse anywhere.

This repository runs the contest. It holds **no benchmark and no model**: benchmarks are forks with
their own simulators, driven as subprocesses, and a model is served behind
[horizon-protocol](https://github.com/robotensor/horizon-protocol).

## An epoch

```bash
horizon-competition config check --config competition.yml
horizon-competition store init   --store store/ --keys keys/

horizon-competition epoch open      --config competition.yml --epoch epochs/2026-W38 \
    --register register/ --store store/ --keys keys/
horizon-competition epoch pool      --epoch epochs/2026-W38
horizon-competition epoch evaluate  --epoch epochs/2026-W38 --key <key> --policy 127.0.0.1:7100 \
    --screen-only
horizon-competition epoch shortlist --epoch epochs/2026-W38 --leader <key>
horizon-competition epoch evaluate  --epoch epochs/2026-W38 --key <key> --policy 127.0.0.1:7100
horizon-competition epoch score     --epoch epochs/2026-W38
                                    # the shortlist, over the frozen pool; --dry-run is a
                                    # rehearsal, and only `close --dry-run` publishes it
horizon-competition epoch close     --epoch epochs/2026-W38 --store store/ --keys keys/
horizon-competition store verify    --store store/ --public-key $HORIZON_PUBLIC_KEY
horizon-competition epoch derive    --secret <revealed> --out /tmp/check \
    --record store/records/epoch-2026-W38.json --public-key $HORIZON_PUBLIC_KEY
```

1. **open** draws a secret, publishes only its hash — signed, and with the window this epoch
   scores, `opens_at` to `closes_at`, computed from `epoch.days` — and derives every unit from it,
   so nobody, including us, can pick the units after seeing a submission. The epoch scores the
   entries whose `submitted_at` falls inside that window, and nothing carries over from the epoch
   before: a model competes again by being submitted again.
2. **pool** builds each demonstration once and freezes it. Every model then runs from those files.
3. **evaluate** runs a submission over the pool. Each unit also fixes the model's sampling seed, so
   two models never differ by luck of the draw. `--screen-only` runs one episode per task.
4. **shortlist** takes what screening showed and picks who is worth a full evaluation. One episode
   per task can tell a model that works from one that does not, and nothing finer, so it only
   shortlists; the current leader is kept when it entered this epoch too.
5. **score** takes each submission's mean and ranks them (below).
6. **close** publishes the epoch and reveals the secret. **derive** then re-derives the units from
   the published record on any machine — no fork, no config, no GPU — so anyone can check that the
   week's scenes are the ones the commitment named. Pin `--public-key` there too: a record checked
   against nothing but itself says only that it agrees with itself.

The store only grows: every record is signed, the index of them is signed too, and a name already
published is never published over. Verify against the organiser's key, pinned from outside the
store — without `--public-key`, a store that was rewritten and re-signed verifies against the
rewriter's own key.

Every step is resumable: a bundle that exists is not rebuilt, a unit with a result is not re-run.
An epoch is a week of GPU time, and machines die in the middle of one.

## Publishing

The dashboard reads a published competition from a Hugging Face dataset repository, so a mirror is
what makes a result public:

```bash
pip install "horizon-competition[mirror]"
HF_TOKEN=... horizon-competition store mirror --store store/ --epochs epochs/ \
    --repo robotensor/horizon-results               # --public to create it public
```

It uploads the signed index, every record the index names, and — for each epoch the store has a
record for — `epoch.json`, `units.jsonl`, each bundle's `demo.json` and `demo.mp4`, and each run's
`result.json` and `rollout.mp4`. Nothing else: not `private/` (the demonstrator's own trajectory),
not `demo_frames.npz` (the arrays a policy is shown, 78 MB a unit), not `secret.txt`, and not a
record nobody signed — the index decides what is carried, never the directory. The store is
verified before anything is uploaded, files are replaced rather than deleted, and `--dry-run`
prints the plan without a token.

The repository's visibility is set when it is created and never changed by a later mirror. A
private one is read by the site with `HF_TOKEN` set server-side.

## Scoring

- A task's score is its success rate; an axis's score is the mean over its tasks.
- **A submission's score is the mean of its axis scores**, over the epoch's axes (2026-09-19). An
  axis dropped for voids is in nobody's mean; the record still publishes what went unmeasured.
- **Rank:** that score, descending. A tie goes to the earlier submission, then to the key, so the
  order is total and a near-copy of the leader gains nothing.
- There is no base model: every key an epoch scores is a submission made inside its window
  (2026-09-19, decision Q27). Nothing is paid out either: the epoch publishes a score and a place,
  and no record names a share or a payout.

## Configuration

`competition.yml` says what the contest is: the axes, the episodes per task, how long an epoch's
submission window stays open (`epoch.days`) and the submission shape. There is no `scoring` block:
a score is the mean of the axis scores and the rank follows from it, so nothing about scoring is
configurable. **Randomization is not here** — it lives in each benchmark's own task config,
so one file describes the environment and this one describes the contest. Changing an axis starts a
new score series; results from different configs are never mixed.

Its paths hang from `${WORKSPACE}`: the environment's, or else the directory the repositories were
cloned into (`scripts/bootstrap.sh`'s rule). They are made absolute when the file is read.
`config check` refuses any key it does not know or that is given twice, and any value the code does
not honour (seven known keys are checked but not read by any code yet: `config.UNREAD`). It says
which interpreter, checkout or dataset this machine lacks, and asks each fork whether it takes the
axis's task config and has its tasks. An epoch keeps a copy of the config it was opened with and
runs every step under it.

## Continuing this on another machine

```bash
gh repo clone robotensor/horizon-competition
./horizon-competition/scripts/continue.sh     # sets the box up, smokes it, prints the prompt
```

See [start here](docs/start-here.md).

## Documentation

- [Start here on a new machine](docs/start-here.md) — one command, then one paste.
- [The plan](docs/plan.md) — what this is, what is built, what is measured and what is not.
- [Setting up a new machine](docs/bootstrap.md) — a GPU box from nothing to a green smoke test.
- [P0: run the model](docs/p0-spike.md) — the one thing nobody has done yet.
- [Running an epoch](docs/runbook.md) — the weekly sequence, and what to do when a step goes wrong.
- [Entering the competition](docs/entering.md) — what a participant submits and how it is scored.

## Development

```bash
uv venv --python 3.10 .venv && uv pip install -e ".[dev]" -e ../horizon-protocol
ruff check . && ruff format --check .
pytest -q
```

Licensed under [Apache-2.0](LICENSE).
