Metadata-Version: 2.4
Name: evostudio-inference-sdk
Version: 0.7.0b19
Summary: Inference integration SDK for EvoStudio simulation benchmarks
Author: EvoMind Team
License-Expression: Apache-2.0
Project-URL: Homepage, https://studio.evomind-tech.com/
Project-URL: Documentation, https://studio.evomind-tech.com/docs/%E6%A8%A1%E5%9E%8B%E8%AF%84%E6%B5%8B%E5%AE%98%E6%96%B9%E9%95%9C%E5%83%8F%E4%B8%8E%E8%87%AA%E5%AE%9A%E4%B9%89%E9%95%9C%E5%83%8F
Keywords: robotics,simulation,evaluation,inference
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy>=1.24
Dynamic: license-file

# EvoStudio Inference SDK

`evostudio-inference-sdk` is the model-integration boundary for EvoStudio's official simulation
benchmarks. Users implement one `Policy` lifecycle and do not install simulator packages or configure
benchmark scheduling and autotune parameters.

Install the SDK from PyPI:

```bash
python -m pip install evostudio-inference-sdk
```

```python
import numpy as np

from evostudio_inference_sdk import ActionChunk, BenchmarkName, Policy, PolicyCapabilities, run


class UserPolicy(Policy):
    def load(self, context):
        self.model = load_model(context.checkpoint_path)
        return PolicyCapabilities(
            supported_benchmarks=(BenchmarkName.LIBERO,),
            required_features=("images.primary", "state.proprioception"),
            supported_suites={BenchmarkName.LIBERO: ("libero_10",)},
            supported_action_chunk_sizes=(1,),
        )

    def act(self, observation):
        return ActionChunk(np.asarray(self.model.predict(observation.features), dtype=np.float32))


run(UserPolicy())
```

EvoStudio launches this entrypoint once per selected inference replica, passes the staged checkpoint and
benchmark contract to `Policy.load()`, and calls it through the local SDK protocol. The user process does
not start a simulator or select evaluator concurrency.

The SDK also contains EvoStudio's platform benchmark runtime, durable work queue, process lifecycle,
resource measurement, and automatic K/g selection under `evostudio_inference_sdk.runtime`. Those modules
are platform internals and are not exported from the user-facing package API. Installing the PyPI package
provides the public Policy API and protocol client; it does not install simulators, benchmark datasets,
model frameworks, or the platform image dependencies needed to execute those internal runtime modules.

Earlier source releases exposed a `libero` extra, but it installed only `imageio[ffmpeg]`; it never
installed a runnable LIBERO simulator or dataset. The extra has therefore been removed. Existing
environments keep their already-installed packages, but consumers that use `imageio` directly must now
declare `imageio[ffmpeg]>=2.34,<3.0` in their own image or application dependencies. This is only a video
dependency migration and is not a supported way to provision any Benchmark runtime.

The platform runtime uses three core scheduling concepts:

- **Server**: an inference process that loads one user `Policy` replica and serves action requests over a
  local socket. A Server does not run a simulator. Multiple Workers may share one Server, which can batch
  their concurrent requests. Autotune calls the selected Server replica count `g`.
- **Worker**: an evaluator process that owns a benchmark simulator lane. It claims Work Items, runs their
  environments and episodes, sends observations to a Server, applies the returned actions, and publishes
  results. A Worker does not load a separate copy of the user model. Autotune calls the selected Worker
  concurrency `K`.
- **Work Item**: the durable, atomically claimed unit of benchmark work. Its exact size is adapter-specific;
  for example, a RoboTwin Work Item is one manifest-sized episode window from a task. Windows preserve the
  task's canonical seed-cache indices while allowing idle Workers to share long-horizon tasks. A failed
  Worker returns its claimed Work Item to the pending queue, while a completed Work Item is recorded
  idempotently so it cannot be scored twice.

RoboTwin and RoboCasa multi-episode work items publish each completed episode as an observable partial
result. Product progress therefore advances while the item is still running, while official aggregation
and durable completion continue to use only the item's final result. A non-OOM RoboTwin native evaluator
failure with a positive exit code is recorded as a terminal failed item (episodes completed before the
crash are kept and the remainder of that window counted as failures), so the worker continues with the next item instead
of aborting the evaluation pass; OOM and signal-terminated exits still restart the worker and replay its
claimed item, with one retry allowed per runtime pass. Observable progress is recomputed from the durable
queue snapshot and published through the atomic `testlog.txt` status file.

Runtime errors are collected in the append-only `/root/testresult/errors.jsonl` journal. It includes
recoverable Worker failures as well as terminal checkpoint, Policy Server, simulator, configuration, and
other evaluation errors. A failed RoboTwin window keeps the native log excerpt in its worker and native
logs and counts that window's missing episodes as failures in the official aggregate, so Quick Test users see the
simulator or Policy failure instead of only a Worker exit code. Component logs remain available for
detailed diagnosis.

`PolicyCapabilities` declares the supported benchmarks, maximum inference batch size, and safe
action-chunk sizes. Evaluator concurrency and replica count are platform-owned autotune parameters rather
than user-declared capabilities. SDK-native LIBERO scored evaluations measure real Policy-backed episodes
across K/g candidates and select the last candidate that improves throughput while resources remain safe.
Candidates that lose Workers, report a Worker failure or OOM, or leave probe episodes incomplete are not selected.
The platform stops a faulty probe immediately, limits each candidate to 15 minutes and the complete tune to
30 minutes, then continues with the last healthy K/g pair. `/root/testresult/autotune/selection.json` is
updated atomically when every candidate starts, advances, or stops, so running, accepted, rejected, and timed-out trials
remain visible to platform log synchronization without contributing episodes to the formal score.

## API stability and versioning

The names exported by `evostudio_inference_sdk.__all__` are the public SDK API. Modules below
`evostudio_inference_sdk.runtime` are platform implementation details unless this document explicitly
states otherwise; applications should not depend on their internal classes or command-line arguments.

The package follows Semantic Versioning. During the current `0.x` Beta series, a minor release may make an
incompatible change to `__all__`, the public Policy lifecycle, or the wire protocol, while patch releases
remain compatible bug fixes. Every evaluation image pins one exact SDK version. User images should also
pin the version they validated, and review release notes before upgrading to another minor version. The
first `1.0.0` release will freeze the public Policy lifecycle and compatibility policy for stable use.

The installed version comes from distribution metadata and can be read with
`importlib.metadata.version("evostudio-inference-sdk")`; `evostudio_inference_sdk.__version__` is not part
of the public API. The release owner updates the version in `pyproject.toml` and `uv.lock`, moves the
`CHANGELOG.md` Unreleased entries into a dated release section, and records every user-visible API,
Policy-lifecycle, and wire-protocol change in the same release merge request.

`CHANGELOG.md` is included in the source distribution, but is intentionally not copied into the wheel's
runtime package. Wheel users can read the same release notes in the repository or PyPI source archive.

## Episode-scoped state

The runtime calls `reset(EpisodeContext)` before an episode, sends its `episode_id` on each observation,
and calls `end_episode(episode_id)` when the episode finishes. The SDK forwards this lifecycle but does not
store or synchronize mutable Policy state.

Stateful Policies should isolate mutable state by `episode_id` and release it from `end_episode()`.
Stateful and stateless Policy examples are provided in the EvoStudio user manual.

## Model-owned input preprocessing

Each Benchmark sends the canonical observation defined by its Benchmark contract without model-specific
resize, normalization, or channel/layout conversion. The user Policy performs those transformations before
calling its model. Model input dimensions therefore do not belong in the public Benchmark or Policy API.

## Current benchmark status

The repository contains the following platform execution chains. Their manifests describe dependencies
that EvoStudio's Benchmark images or shared storage must provide; they are not dependencies of the PyPI
distribution. A chain being present here does not guarantee that every deployment has provisioned its
external artifacts.

| Benchmark | Platform execution chain | Image/shared-storage dependencies |
| --- | --- | --- |
| LIBERO | SDK-native adapter and persistent LIBERO worker | Pinned LIBERO Python environment, simulator source, dataset and initial states |
| LIBERO+ | SDK-native adapter that reuses the LIBERO worker | Pinned LIBERO+ Python environment, simulator archive, dataset and initial states |
| RoboTwin | SDK-native adapter and worker that invoke the pinned RoboTwin launcher | RoboTwin environment and source, deploy policy, checkpoint assets and validated seed cache |
| RoboCasa | SDK-native adapter and persistent RoboCasa worker | Pinned RoboCasa environment, simulator/task source and dataset assets |
| CALVIN | SDK-native adapter and persistent CALVIN worker | Pinned CALVIN environment, benchmark/config source, dataset and evaluation-sequence manifest |

LIBERO and CALVIN convert their native observations to the canonical SDK feature names and send native
7-dimensional actions back to their simulators. CALVIN's task-D-to-task-D path implements the official
1,000 long-horizon sequences with static/gripper images, proprioception and language. The suite list and
runtime paths for every chain remain owned by its installed Benchmark image manifest.
- Evaluation has two product paths. Custom-image users provide their own SDK `Policy` and saved image.
  StarVLA users may instead upload a standard run snapshot and directly select the official StarVLA image.
  Both paths use platform Benchmark environments and evaluators from `/autodl-fs`.

The complete LIBERO entrypoint and scoring contract is documented in
`LIBERO_EVALUATION_PROTOCOL.md` in the Evaluation Kit repository.
The LIBERO document defines the `/root/startbenchmark.sh` arguments, checkpoint staging, canonical
observation/action mapping, episode initialization and horizons, success detection, output files, error
codes, and required validation sequence.
The SDK-native CALVIN manifest, observation/action mapping, official long-horizon evaluator, aggregation,
and validation procedure are documented in
`CALVIN_EVALUATION_PROTOCOL.md` in the Evaluation Kit repository.

The SDK integration, autotune, and benchmark-runtime throughput results are maintained in the Evaluation
Kit repository. The report keeps LIBERO and RoboTwin results separate and includes the StarVLA official
four-GPU RoboTwin launcher baseline used for the runtime comparison.

## Policy HIL model components

The SDK supplies checkpoint discovery, immutable verification, Pi0.5 loading,
processor construction, and the pinned LeRobot RTC configuration used by
`policy_hil_continuous`. It does not provide a robot-control service and never
owns CAN, cameras, recording, progress, or uploads.

The separately packaged `evostudio-hil-runtime` service in the EvoStudio Client
repository imports `load_pi05_runtime`. That service keeps the official
`RTCInferenceEngine`, model, processor pipelines, four arms, three cameras, and
pedal in one Python process. There is no action-chunk wire protocol or Client
queue adapter.

Local checkpoints and referenced model resources must remain inside explicitly
allowlisted roots. The caller obtains a `VerifiedCheckpoint` from
`CheckpointCatalog`, freezes its hashes into the collection session, and passes
it to the component factory. The factory also requires an explicit local
tokenizer path because deployed checkpoints can contain training-host paths that
are not valid on the collection machine.

The runtime requires the pinned LeRobot 0.6.0 installation used by the validated
native Pi0.5 RTC deployment. Archived `evostudio-engine` checkouts are not a
runtime source. The `policy_hil_runtime/v1` contract belongs to the Client HIL
service; this package exposes model components rather than a socket server.

## Direct evaluation with the official StarVLA image

Custom images contain the user's SDK `Policy`, dependencies and `/workspace/evostudio-policy.json`.
The separate official StarVLA image path uses an EvoStudio-published image with a pinned StarVLA runtime
and a preconfigured evaluation entrypoint:

```text
/opt/evostudio/starvla/starvla_official_policy.py
```

Cloud stages the user-selected StarVLA run directory. The run keeps StarVLA's native layout:

```text
<run>/
├── config.yaml
├── dataset_statistics.json
├── checkpoints/
│   └── <selected full checkpoint>
└── <local resources referenced by config.yaml>
```

The staged evaluation snapshot contains one selected full checkpoint. `SessionContext.checkpoint_path`
may point to that checkpoint file or to the run directory. The official loader deterministically locates
the checkpoint and adjacent config/statistics, builds the framework from `config.yaml`, performs strict
state-dict loading, and initializes training-time normalization.

The official Policy follows the pinned StarVLA reference client for the active Benchmark. RoboTwin,
RoboCasa, and LIBERO+ use OpenCV `INTER_AREA`; LIBERO uses PIL bilinear resize; CALVIN uses the upstream
aspect-preserving resize-and-pad and uint8 conversion. All produce the pinned clients' `224x224` StarVLA
model input without changing the Benchmark's canonical Observation. The Policy does not derive this
client-side preprocessing from, or rewrite, the uploaded StarVLA configuration. Tokenization, inference
kwargs and dtype are read from the StarVLA configuration or
its framework defaults. Benchmark environments, datasets, evaluator workers and native action application
are provided by `/autodl-fs`. Both custom images and the official StarVLA image support an independent real
`K=1/g=1` quick test and full formal evaluation.

`config.yaml` must carry an `inference` section. It names the Observation features the Policy consumes and
declares the per-dimension action transforms:

```yaml
inference:
  dtype: bfloat16                    # required: bfloat16, float16 or float32
  unnorm_key: franka                 # optional: training normalization statistics key
  inputs:                            # required
    images:                          # required, non-empty; order is the camera order
      - images.primary
      - images.wrist
    language: language.instruction   # required
    state: state.proprioception      # optional, declare only if the model consumes it
  kwargs:                            # optional, forwarded to the StarVLA framework
    do_sample: false
    use_ddim: true
    num_ddim_steps: 10
  action_transforms:                 # optional, applied in order to the model output
    - index: 6
      operation: threshold
      threshold: 0.5
      above: -1.0
      below: 1.0
```

`inputs` values are Benchmark Observation feature names; LIBERO exposes `images.primary`, `images.wrist`,
`state.proprioception` and `language.instruction`. A missing `inputs` or malformed `action_transforms`
entry fails Policy startup. A syntactically valid but semantically wrong camera order or action transform
cannot be detected automatically, so both must match the training-time definitions.

`action_transforms` supports `threshold` (`index`, `threshold`, `above`, `below`) and `affine`
(`index`, `scale`, `offset`). The example above reproduces the LIBERO gripper binarization: dimension 6
becomes `-1.0` when the raw value exceeds `0.5`, and `1.0` otherwise.

The image ships an exporter that produces a portable run snapshot:

```bash
/root/miniconda3/envs/starVLA/bin/python \
  /opt/evostudio/starvla/export_starvla_eval_run.py \
  --run-dir <training run> \
  --checkpoint <training run>/checkpoints/<selected checkpoint> \
  --output <export directory> \
  --resource-root <root of the local construction resources>
```

The exporter copies every local resource referenced by the `framework` and `inference` sections into the
export directory, rewrites those configuration values to relative paths, and runs a strict-load preflight.
A hand-assembled snapshot that keeps training-host absolute paths (for example `framework.qwenvl.base_vlm`)
cannot be resolved inside the evaluation instance.

The official image Policy process contract is:

```json
{
  "schema_version": 1,
  "command": [
    "/root/miniconda3/envs/starVLA/bin/python",
    "/opt/evostudio/starvla/starvla_official_policy.py"
  ],
  "cwd": "/root/starVLA"
}
```

The LIBERO platform entrypoint loads the Policy process contract from the fixed image path
`/workspace/evostudio-policy.json`:

```json
{
  "schema_version": 1,
  "command": ["/opt/user/bin/python", "/workspace/user_policy.py"],
  "cwd": "/workspace"
}
```

EvoStudio selects the Policy runtime and validates the file independently during quick test and scored
evaluation. End users do not configure the path, K/g, or benchmark runtime flags.

## License

Copyright 2026 EvoMind Team. Licensed under the Apache License, Version 2.0. See `LICENSE`.
