Metadata-Version: 2.4
Name: simready-benchmark
Version: 2026.7.1
Summary: SimReady Benchmark, the engine-agnostic core for validating SimReady USD assets
Author: NVIDIA Corporation
License-Expression: Apache-2.0
Project-URL: Homepage, https://www.nvidia.com
Keywords: nvidia,simready,test,validation
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Requires-Python: >=3.12
Description-Content-Type: text/markdown
License-File: LICENSE.txt
Requires-Dist: filelock<4.0,>=3.13.0
Requires-Dist: usd-core>=23.5
Provides-Extra: kit
Requires-Dist: simready-benchmark-engine-kit; extra == "kit"
Dynamic: license-file

# simready-benchmark

Engine-agnostic Python framework for benchmarking SimReady USD assets.
Discovers applicable tests, executes them across one or more engine sessions
in parallel, and produces HTML and JSON reports.

Part of the [SimReady Python Library Suite](https://developer.nvidia.com/simready)
alongside `simready-validate` and `simready-search`.

---

## Requirements

- **Python 3.12 exactly.** The supported engine, NVIDIA Isaac Sim, requires
  Python 3.12, so the virtual environment must use 3.12.
- **A 64-bit x86 machine (x86_64, also called AMD64)** running Windows 10 or 11,
  or Linux (Ubuntu). `usd-core`, a core dependency, publishes no ARM (aarch64)
  wheels, so `pip install` fails while resolving `usd-core` on ARM machines,
  including ARM cloud instances. On Linux, confirm your architecture with
  `uname -m`, which must report `x86_64`.
- **A CUDA-capable NVIDIA GPU and about 30 GB of free disk**, primarily for
  [Isaac Sim](https://docs.isaacsim.omniverse.nvidia.com/latest/index.html),
  the engine that runs the tests.

---

## Installation

Always work inside a dedicated virtual environment, because the framework and
the Isaac packages can conflict with other USD or Omniverse installations.

### Windows

```powershell
mkdir C:\simready
cd C:\simready
python -m venv .venv
.\.venv\Scripts\Activate.ps1
pip install "simready-benchmark[kit]"
```

If PowerShell reports that running scripts is disabled, run
`Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser` one time,
then retry the activate command.

### Linux (Ubuntu)

Ubuntu 22.04 ships Python 3.10 and has no `python3.12` package, so add the
[deadsnakes PPA](https://launchpad.net/~deadsnakes/+archive/ubuntu/ppa) first:

```bash
sudo apt install -y software-properties-common
sudo add-apt-repository -y ppa:deadsnakes/ppa
sudo apt update
sudo apt install -y python3.12 python3.12-venv
```

Then create the environment and install:

```bash
mkdir -p ~/simready
cd ~/simready
python3.12 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
pip install "simready-benchmark[kit]"
```

Do not install `python-is-python3` as a substitute; it only aliases `python` to
the system Python (3.10 on Ubuntu 22.04), which Isaac Sim rejects.

### Verify

```bash
simready-benchmark --version
```

These commands respect the default index, mirror, and credentials already
configured for `pip`. To install a local repository build, run
`repo.bat python_package` from the `simready-explorer` root and install the
`simready_benchmark-*.whl` and
`simready_benchmark_engine_kit-*.whl` files from `_build/packages/dist/`.

The `[kit]` extra pulls in `simready-benchmark-engine-kit`, the Kit and Isaac Sim
engine plugin, along with `usd-core`, `numpy`, and `Pillow`. Install
`simready-benchmark` without the extra for the engine-agnostic core and CLI only.

Specification tiers may also be installed as Python packages:

```bash
pip install simready-foundation-tier-core
```

Benchmark discovers their catalogs and optional runtime tests through the
`simready.tier` descriptor. Independent test-only wheels use
`simready_benchmark.tests`.
An explicit Foundation checkout remains available for source-tree development:

```bash
simready-benchmark --foundations-path /path/to/simready_foundations --list-tests
```

This discovers the checkout's tier catalogs and runtime-test package. The flag
applies to that command only; it is not required for an installed-tier
workflow.

To use the same checkout on every command without repeating
`--foundations-path`, create or update
`~/.simready-benchmark/engines.toml` (`%USERPROFILE%\.simready-benchmark\engines.toml`
on Windows) and persist the checkout's project configuration:

```toml
[paths]
project_config = "<foundations>/sample_content/project_config.toml"
```

Use forward slashes in the TOML path on Windows. Benchmark then obtains the
checkout's content root, tier catalogs, and tier-owned runtime tests from that
single project configuration, so subsequent commands need neither
`--foundations-path` nor `--project-config`:

```bash
simready-benchmark --show-config
simready-benchmark --list-tests
simready-benchmark --assets /path/to/assets
```

`SIMREADY_PROJECT_CONFIG` provides the same project-config selection through an
environment variable. Explicit CLI arguments take precedence over environment
and `engines.toml` defaults.

---

## Setup

Install a tier with its Benchmark extra, then validate the environment:

```bash
pip install "simready-foundation-tier-core[benchmark]"
simready-benchmark --setup --install-isaac
```

The tier advertises catalogs and its optional `runtime_tests_path` through
`simready.tier`. Independent test-only distributions use
`simready_benchmark.tests`. Setup checks those installed providers and the active
Python 3.12 environment. With `--install-isaac`, it installs pip Isaac from its
configured NVIDIA package index when absent. It creates no source checkout,
does not persist `--foundations-path`, and does not create a configuration
file. A source-checkout-only workflow does not use `--setup` to register the
checkout; select it per command with `--foundations-path`, or persist its
`sample_content/project_config.toml` under `[paths]` in `engines.toml` as shown
above.

When upgrading from the retired `simready-foundation-runtime-tests-kit`,
uninstall it and then force-reinstall `simready-foundation-tier-core`. The old
and new distributions otherwise claim the same test-package files; `--setup`
rejects that ambiguous state with an actionable migration message.

Pip Isaac is discovered automatically as two runtimes: `isaac_sim` for PhysX
and `isaac_sim_newton` for Newton. An optional `engines.toml` can persist path
defaults such as `project_config` and `output_dir`; it is also used for a
custom/standalone executable, custom experience, or remote worker.

### Verify the Configuration

```bash
simready-benchmark --show-config
```

This prints installed tier catalogs, runtime-test packs, automatic engines, and
any explicit project or `engines.toml` override.

---

## First Test

List the discovered tests, then run the full pipeline (plan, run, and report)
against one demo asset. With an installed tier, or after persisting the
Foundation project config in `engines.toml`, run:

```bash
simready-benchmark --list-tests
simready-benchmark --assets <foundations>/sample_content/common_assets/props_general/apple_a01
```

For an unconfigured source checkout, include `--foundations-path` on both
commands:

```bash
simready-benchmark --foundations-path <foundations> --list-tests
simready-benchmark --foundations-path <foundations> \
  --assets <foundations>/sample_content/common_assets/props_general/apple_a01
```

Replace `<foundations>` with your clone path, for example
`<drive>:\path\to\simready_foundations` on Windows or
`/path/to/simready_foundations` on Linux. Isaac Sim launches and the
applicable tests run against the apple asset. Benchmark prints the allocated
run directory and the exact HTML report path when it finishes. Open that
printed `index.html`; do not assume a fixed `_testing` directory because an
existing requested directory causes Benchmark to allocate a fresh
`benchmark_testing[_N]` child.

The exit code is `0` when every test passes and `1` when any test fails. Refer to
[Exit Codes](#exit-codes) for the full list.

### Known Issues (Isaac Sim First Run)

- **The first run on a machine where Isaac Sim has never launched can be very
  slow or appear to fail** until Isaac Sim compiles its shader cache. Warm the
  cache once: launch Isaac Sim directly, wait for the main viewport, open any USD
  file (the sample apple works), then close it. Re-run the benchmark. After one
  successful run the cache is warm and later runs complete normally.
- **A test can fail with an entirely white (blank) comparison frame.** Isaac Sim
  and Kit sometimes do not render a frame in time, which fails the comparison.
  This is an engine rendering issue, not an asset defect. Re-run only the failed
  tests with `simready-benchmark --rerun failed`; if they pass on the second
  run, the asset is good. If your original run used a custom `--output-dir` or
  `--project-config`, pass the same flag to `--rerun failed` so it finds the
  results.
- **The Isaac Sim install stalls or fails on a network read timeout.** Re-run
  `simready-benchmark --setup --install-isaac`; setup uses extended pip timeouts
  and retries for the multi-gigabyte download.
- **An installed provider or Isaac is missing.** Run
  `simready-benchmark --show-config`, then
  `simready-benchmark --setup --install-isaac`. Setup reports separately when
  the tier, test pack, or pip Isaac launcher is missing.

---

## Day-to-Day Use

The virtual environment is per-terminal. In every new terminal, activate it
before running the tool:

```bash
# Windows
cd C:\simready
.\.venv\Scripts\activate

# Linux
cd ~/simready
source .venv/bin/activate
```

To refresh the test content and check for engine and framework upgrades, run
`simready-benchmark --update`.

---

## Output Layout

The requested output location resolves in this order: `--output-dir`,
`engines.toml [paths].output_dir`, `<project_root>/_testing`, then
`<cwd>/_testing`. A new run allocates a fresh `benchmark_testing[_N]` child when
that requested location already exists. The resolved directory contains:

- `plan.json`. The resolved plan
- `run_summary.json`. Terminal run, readiness, and failure summary
- `state/events.jsonl`. Live JSON-Lines event stream
- `state/work_pool.json`. Work-item and per-test execution state
- `results/<asset-dir>/<asset>`. Root-layer report copy; writable layers are
  stamped, while USDZ packages remain byte-identical
- `results/<asset-dir>/.simready/validation.json`. Wrapped
  `SimReady_Metadata.runtime_testing` receipt
- `results/<asset-dir>/.simready/runtime/<test-output-name>/<engine>/result.json`.
  Unique tests keep their raw name; provider qualification and a deterministic
  digest make overlapping names collision-safe. Per-engine
  test result and media artifacts
- `logs/<engine>_<n>.log`. Per-engine subprocess logs
- `index.html` plus `assets/<slug>.html` (HTML index and per-asset details) and
  `report/test_results_index.json`. Aggregated reports

The original asset directory is never modified. This is intentional: source
assets may be read-only, remote, version controlled, or require a separate
checkout/publish workflow before they can be changed.

The results tree mirrors the source asset's relative path to prevent output
name collisions; it does not mirror the complete asset directory. Stamping
copies only the root USD layer and writes the benchmark receipt beside it.
Referenced layers, payloads, `materials/`, `textures/`, and other relative
dependencies are not copied or rewritten. Consequently, the stamped USD under
`results/` is a report/metadata artifact and may not open with complete
composition or visual fidelity. Open the original asset for inspection, or use
an authorized downstream packaging/publishing step if a portable stamped asset
is required.

Benchmark never clears the requested output directory. It creates and marks a
fresh exclusive run directory; when the requested directory already exists
(including a filesystem or drive root), it writes only to a
`benchmark_testing[_N]` child. It never writes run artifacts directly into the
root or clears existing content. Read the resolved output and report paths from
the final CLI events. The deprecated `--keep-outputs` flag is accepted for
compatibility but does not change the always-preserve behavior.

---

## Key Concepts

### Plan / Run / Stamp / Report Pipeline

A single `simready-benchmark` invocation runs four phases by default:

1. **Plan.** Scan installed test packs and the active `sr_specs`; match assets to
   applicable tests; write `plan.json`.
2. **Run.** Execute the plan across one or more engine sessions in parallel;
   stream JSON events; write per-test `result.json`.
3. **Stamp.** Unless suppressed with `--no-stamp`, write matching benchmark
   metadata into writable root-layer mirrors and `.simready/validation.json`.
   USDZ mirrors are byte-identical and sidecar-only. These copies are receipts,
   not self-contained asset packages.
4. **Report.** Aggregate per-test results and stamp information into HTML and
   JSON reports.

### Engines and Test Packs

Engines are registered through `simready_benchmark.engines`. Independent test
packs use `simready_benchmark.tests`; tier-owned packs are advertised through a
`simready.tier` descriptor's optional `runtime_tests_path`. No per-install path
configuration is needed for installed providers.

Out of the box, `simready-benchmark` registers an experimental `nvcf` engine
plugin for NVIDIA Cloud Functions execution. The NVCF plugin, remote-worker
path, and local emulator are **Technology Preview** features and are not the
supported release-validation path. They require a separately provisioned
compatible worker endpoint and credentials; Benchmark does not provide an NVCF
deployment, worker URL, API key, or test account.
`simready-benchmark-engine-kit` adds the supported `kit` engine plugin for Kit
and Isaac Sim. Third-party engines and test packs slot in through the same
mechanism.

### Engine Configuration

Pip Isaac in the active environment is discovered automatically as
`isaac_sim` (PhysX) and `isaac_sim_newton` (Newton). No config file is needed.

An optional `engines.toml` overrides automatic local discovery for a
standalone/custom runtime. The framework reads it using a 4-tier lookup (first
match wins):

1. `--engines-toml <FILE>` CLI flag
2. `$SIMREADY_ENGINES_TOML` environment variable (a missing file is a hard error)
3. `engines.toml` in the current working directory
4. `~/.simready-benchmark/engines.toml`

The run output uses its own precedence: `--output-dir`, then
`[paths].output_dir` from the resolved `engines.toml`, then
`<project_root>/_testing`, and finally `<cwd>/_testing`. Benchmark allocates a
fresh `benchmark_testing[_N]` child when the requested location already exists.

For a standalone install, two engine configurations can share one executable
and differ only in the Kit experience:

```toml
[kit.isaac_sim]
executable_path = "C:/simready/.venv/Scripts/isaacsim.EXE"
version = "6.0.1.0"
tags = ["isaac", "kit"]

[kit.isaac_sim_newton]
executable_path = "C:/simready/.venv/Scripts/isaacsim.EXE"
version = "6.0.1.0"
tags = ["isaac", "kit"]
experience = "isaacsim.exp.full.newton"

# Optional: fixed output location (otherwise <CWD>/_testing).
[paths]
output_dir = "/path/to/_testing"
```

Select an engine per run with `--runtime`: `--runtime isaac_sim_newton` runs
Newton only, `--runtime isaac_sim` runs PhysX only, and a run with no `--runtime`
runs every eligible test on all configured engines. Runtime-specific physics
tests are eligible only when the asset validation record contains the matching
feature variant, such as `FET_003_NEWTON` or `FET_003_PHYSX`;
`FET_003_STANDARD` alone does not authorize either concrete backend when the
test has runtime-specific variants. Runtime-neutral tests can still run on both
engines. The report attributes each result to the feature matching its engine.

To launch the raw Isaac Sim application in Newton mode, without the benchmark,
pass the experience as the first launcher argument:
`isaacsim.EXE isaacsim.exp.full.newton` (`isaacsim` without the `.EXE` on Linux).

---

## CLI

This README summarizes common operations. The project documentation's
`docs/cli-reference.md` contains every public option and subcommand. The exact
help for an installed build is available with `simready-benchmark --help` and
`simready-benchmark <subcommand> --help`.

The default invocation runs the full pipeline. Mutually exclusive mode flags
select alternate top-level workflows, and four subcommands provide staging,
stamping, and preview-emulator operations.

### Mode Flags

| Flag | Behavior |
|---|---|
| (default) | Plan, run, and report in one pass |
| `--plan-only` | Generate `plan.json` and stop |
| `--edit-plan ...` | Edit an existing `plan.json` (no run, no report) |
| `--rerun {plan,failed,report}` | Operate on an existing run: `plan` runs and reports against the existing `plan.json`, `failed` re-runs only the tests that failed in a previous run, `report` regenerates the report from existing per-test `result.json` files |
| `--list-tests` | Print the registered features and tests |
| `--show-config` | Print the resolved configuration and exit |
| `--setup` | Verify installed tiers, runtime-test packs, and pip Isaac; add `--install-isaac` to install Isaac when absent |
| `--update` | Check pip Isaac for an update and report the Benchmark upgrade command |

Subcommands are `stamp`, `stamp-one`, `stage`, and the Technology Preview
`nvcf-emulator`. For example:

```bash
simready-benchmark stamp --output-dir _testing
```

### Common Flags

| Flag | Description |
|---|---|
| `--project-config <PATH>` | Load content roots, `sr_specs`, profiles, features, and `[tests].paths` from `project_config.toml` |
| `--foundations-path <DIR>` | Load tier catalogs and runtime tests from a `simready_foundations` source checkout; cannot be combined with `--project-config` |
| `--sr-specs <PATH>` | Path to `sr_specs` (overrides project config and `engines.toml`) |
| `--assets <PATH> [<PATH>...]` | One or more files or directories of assets |
| `--engines-toml <FILE>` | Override the `engines.toml` lookup chain |
| `--tests-path <PATH> [<PATH>...]` | Extra directories scanned for `@test`-decorated functions |
| `--output <FILE>` | Override the plan path; normally unnecessary because `--output-dir` writes `<DIR>/plan.json` automatically |
| `--output-dir <DIR>` | Override the default `_testing/` location |
| `--features <ID> [<ID>...]` | Match feature ID prefixes; also bypasses the validation gate for matches |
| `--tests <SELECTOR> [<SELECTOR>...]` | Raw names keep every provider match; `provider:name` selects one exact provider |
| `--max-concurrent <N>` | Up to N engine sessions in parallel (default 2) |
| `--session-mode {batch,per_asset}` | Reuse sessions across assets, or start a fresh session for each asset |
| `--workers <url1,url2,...>` | **Technology Preview:** separately provisioned NVCF worker endpoints (additive to local Kit engines) |
| `--no-local-engines` | Skip local engines; with `--workers`, selects the **Technology Preview** NVCF-only path |
| `--runtime <NAME> [<NAME>...]` | Pin execution to one or more discovered engines (for example `isaac_sim_newton` or `isaac_sim`); with no flag every test runs on all matching engines. Unknown names hard-fail |
| `--no-stamp` | Skip the stamp phase after a successful run |
| `--keep-outputs` | Deprecated compatibility flag; existing output is always preserved |
| `--format {html,json,both}` | Select report output; default `both` |

`--setup` accepts `--install-isaac` or `--no-install-isaac`. Run
`simready-benchmark --help` for the complete list.

### Exit Codes

| Code | Meaning |
|------|---------|
| `0` | All tests passed |
| `1` | One or more tests failed, or a runtime error |
| `2` | CLI usage error |
| `3` | `--rerun plan` or `--rerun report` invoked but no `plan.json` or results found |
| `4` | Setup or environment not ready (no engine could run part of the plan, or an engine failed preflight). `4` outranks `1`. |
| `130` | Interrupted by user (Ctrl-C) |

---

## Python API (Advanced)

The CLI is the recommended entry point. The Python surfaces are low-level and
intended for advanced integrations:

- `simready_benchmark.Planner`. Static-method class;
  `Planner.plan(sr_specs_path, test_dirs, discovery, validation_source, ...) -> Plan`.
- `simready_benchmark.Runner`. Static-method class;
  `Runner.start(plan, output_dir, ...) -> RunHandle`.
- `simready_benchmark.Reporter`. Static-method class;
  `Reporter.generate(output_dir, fmt="both") -> dict`.
- `simready_benchmark.PlanEditor`. Wraps a `Plan`; appends overrides. Load and
  save through the public `Planner.load` and `Planner.save` methods.
- `simready_benchmark.test`. The `@test` decorator. Functions MUST be
  `async def`. Required keyword arguments: `features`, `name`, `description`,
  `expected_video`, and `version`. Optional: `engine`, `config_defaults`,
  `max_duration` (default 300), and `enabled` (default `True`).
- `simready_benchmark.RunContext`, `simready_benchmark.SceneHandle`. Runtime
  types used by tests.
- `simready_benchmark.EngineConfig`, `EnginePlugin`, `EngineSession`, and
  `ValidationResult`. Public engine-plugin contracts.

The package ships six skills under `src/simready_benchmark/skills/` with
ready-to-use recipes: `install-environment`, `run-tests`,
`embed-in-pipeline`, `diagnose-failures`, `write-test-pack`, and
`write-engine-plugin`.

---

## Metadata Ownership

`simready.test` owns only the `runtime_testing` sub-key of `SimReady_Metadata`
in USD `customLayerData` and `.simready/validation.json`. It never reads or
writes `asset_id`, `validation`, or any other section.

```
customLayerData["SimReady_Metadata"]
├── asset_id          <- simready.create
├── validation        <- simready.validate
└── runtime_testing   <- simready.test
```

```
.simready/validation.json
├── schema_version, profile, passed, issues, ...   <- simready.validate
└── SimReady_Metadata.runtime_testing.tested_features
                                                    <- simready.test
```

---

## License

Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
SPDX-License-Identifier: Apache-2.0
