Metadata-Version: 2.4
Name: x4d-devkit
Version: 0.34.0rc2
Summary: Official X-4D external SDK and CLI for datasets, platform APIs, training bridges, and automation
Author: windzu
License-Expression: Apache-2.0
Project-URL: Homepage, https://github.com/windzu/x4d-devkit
Project-URL: Repository, https://github.com/windzu/x4d-devkit
Project-URL: Issues, https://github.com/windzu/x4d-devkit/issues
Keywords: autonomous-driving,dataset,evaluation,annotation,lidar,camera,cli,sdk
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy>=1.24
Requires-Dist: httpx>=0.27
Requires-Dist: rfc8785==0.1.4
Requires-Dist: zstandard<1,>=0.23
Provides-Extra: inference
Requires-Dist: pydantic>=2.0; extra == "inference"
Requires-Dist: fastapi>=0.110; extra == "inference"
Requires-Dist: uvicorn>=0.27; extra == "inference"
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"
Requires-Dist: pytest-cov; extra == "dev"
Requires-Dist: ruff==0.15.21; extra == "dev"
Requires-Dist: tomli>=2.0; python_version < "3.11" and extra == "dev"
Provides-Extra: docs
Requires-Dist: mkdocs-material>=9.5; extra == "docs"
Requires-Dist: mkdocstrings[python]>=0.27; extra == "docs"
Provides-Extra: release
Requires-Dist: build>=1.2; extra == "release"
Requires-Dist: twine>=6.0; extra == "release"
Dynamic: license-file

# x4d-devkit

Canonical contract library and `x4d-devkit` CLI for consuming X-4D data and
public platform capabilities.

`x4d-devkit` owns X-4D data semantics, identity, coordinate/schema contracts,
platform IO adapters, and official evaluation primitives. It is the stable
interpretation layer between X-4D platform state and consumers such as training
platforms, X-Points, Clip preannotation services, evaluation scripts, and
automation.

It is not a downstream training, annotation-tool UI, experiment, deployment, or
framework-specific cache-policy package. It may call public platform APIs or
read standard X-4D files, but it must not import backend-private `app.*`
modules or mutate DB/MinIO internals.

Design decisions for this boundary are tracked in GitHub issues, starting with:

- [#42](https://github.com/windzu/x4d-devkit/issues/42) - official external SDK and CLI boundary
- [#43](https://github.com/windzu/x4d-devkit/issues/43) - historical private-source runtime integration
- [#44](https://github.com/windzu/x4d-devkit/issues/44) - capability registry and CLI sync

## Installation

```bash
pip install x4d-devkit==0.34.0rc1
```

Devkit is released independently as an immutable wheel. X-4D, XPoints, and
other consumers must install one exact package version during image builds.
Runtime source copies, submodule installs, editable installs, VCS `main`
installs, and fallback dependency paths are not supported deployment modes.
Use a published prerelease for coordinated cross-repository development.

Optional heavy dependencies:

```bash
# Clip preannotation service SDK
pip install x4d-devkit[inference]
```

The platform API client and `x4d-devkit` CLI are base package features.
See [the release process](docs/release-process.md) for the immutable artifact
and consumer cutover rules.

## Scope

See [docs/module-boundary.md](docs/module-boundary.md) for the module ownership
table and command boundary.

Internal consumers should use the current-state identity contract for
platform-to-platform data flow. See
[docs/current-state-identity.md](docs/current-state-identity.md) for the
generic identity model, and
[docs/training-current-state-data.md](docs/training-current-state-data.md) for
work-session materialization and conditional annotation writeback. Every
Dataset 0.17 Clip carries its authoritative Label Schema snapshot at root
`label_schema.json`.
Shared validation rules for platform submit, X-Points preflight, local clip
validation, and training consumers are documented in
[docs/shared-contract-validation.md](docs/shared-contract-validation.md).
The Dataset 0.17 native contract, exact nanosecond time, Multi-LiDAR layout,
and model materialization are documented in
[docs/dataset-0.17.md](docs/dataset-0.17.md). The single-file annotation
authority, semantic revision, and
derived track views are documented in
[docs/annotation-contract.md](docs/annotation-contract.md).
Portable export producers and consumers share the exact `0.17` manifest
parser documented in
[docs/export-bundle-contract.md](docs/export-bundle-contract.md).
`examples/current_state_identity_consumers.py` provides an offline downstream
fixture for read-only, read-write, and materialization consumers.

Included:

- Local X-4D dataset loading, validation, transforms, annotations, calibration,
  ego pose, and evaluation helpers.
- Platform API client and public `x4d-devkit` CLI commands for external
  workflows such as clip listing/download, screening preview, model/checkpoint
  registration, status queries, and capability discovery.
- Typed, authenticated start/reuse, task polling, current-result discovery,
  and opaque asset access for canonical Clip static-RGB reconstruction.
- Typed semantic-scene preparation over canonical static RGB reconstruction.
- Strict scene/instance point-authority validation with independent input and
  output revisions; voxel OCC remains a downstream derivation.
- Typed point-level semantic-scene authoring sessions, dirty semantic-chunk upload, and
  complete compare-and-set submit without a per-point mutation API.
- Strict validation of `x4d-training-identity-v4` manifests.
- Strict producer-option and manifest parsing for export bundle `0.17`.
- Clip preannotation protocol, source-only inference adapter, and service
  wrapper for algorithm projects.

Excluded:

- Backend migrations, DB/MinIO repair scripts, cache rebuild internals,
  emergency operations, and one-off delivery importers.
- Any code that imports backend-private `app.*`.
- Downstream experiment taxonomy, `CLASS_NAMES`, model-class mappings, sampler
  policy, model acceptance thresholds, deployment policy, and framework-specific
  artifacts such as OpenPCDet infos/dbinfos or mmdetection3d pickle views.

Promotion rule: an internal operation becomes an `x4d-devkit` command only
after it is productized as an external SDK/API workflow with explicit
permissions, side effects, idempotency, and failure modes. Data deletion, clip
production, and internal service dispatch stay out of devkit.

Training-specific convenience utilities must remain framework-neutral. Devkit
may expose raw category stats, identity fingerprints, coordinate/schema
validation, cache completeness checks, and official X-4D metrics; downstream
training repos own taxonomy mapping, training-format generation, sampler
configuration, and experiment lifecycle.

## Quick Start

### Connect to an X-4D platform

The CLI can authenticate with a bearer token passed explicitly, through
environment variables, or from the user config written by `x4d-devkit login`.

```bash
x4d-devkit --api-url http://host:8000 --token <token> projects list
x4d-devkit --api-url http://host:8000 --token <token> clips list --project-id 1
x4d-devkit --api-url http://host:8000 --token <token> clips download --clip-ids clip_a,clip_b --output /data/x4d/clips
```

```bash
export X4D_API_URL=http://host:8000
export X4D_TOKEN=<token>
x4d-devkit projects list
x4d-devkit clips list --project-name demo-project --has-archive true --format json
x4d-devkit clips download --project-name demo-project --output /data/x4d/demo-project/clips --workers 8
x4d-devkit clips download --project-name demo-project --archive-profile keyframes_only --output /data/x4d/demo-project/keyframes --workers 8
```

```bash
x4d-devkit --api-url http://host:8000 login --username <user> --password <password>
```

`login` stores `api_url` and the returned access token as `token` in
`~/.config/x4d/config.toml`. Archive downloads go through the platform file
proxy and extract standard X4D clip directories; training containers do not
need database or MinIO access.

Clip archive downloads are profile-aware. `--archive-profile keyframes_only` is
the default and requests the smaller archive with sweeps removed and metadata
filtered accordingly. Use `--archive-profile full` when a consumer needs the
complete clip payload including `sweeps/`. If the requested profile has not been
built, is stale, or failed to build, the devkit reports that profile-specific
reason instead of falling back to another tar.

### Validate a training identity manifest

The `manifest` command has one contract and one operation. It validates an
exact `x4d-training-identity-v4` record; it does not create datasets, download
clips, or translate earlier manifest shapes.

```bash
x4d-devkit manifest validate \
  --manifest training_identity.json \
  --format json
```

Unsupported versions, removed aliases, and malformed clip/schema identities
fail validation. Producers own manifest construction and must emit the current
contract directly.

### Consume the project label schema

X-4D is the only source of truth for label-schema bodies. Consumers resolve the
effective project record from the platform instead of maintaining their own
category list or copying the record into each clip.

```bash
x4d-devkit label-schemas project-get --project-id 1 --format json
x4d-devkit label-schemas get \
  --schema-id object_3d_low_speed_roadside \
  --version 1.0.0 \
  --format json
```

```python
from x4d_devkit import X4DClient
from x4d_devkit.label_schema import compare_project_label_schema_snapshots

client = X4DClient.from_config()
project = client.projects.resolve(project_name="demo-project")
project_schema = client.label_schemas.resolve_project(project_name="demo-project")
project_schema_view = client.label_schemas.get_project_view(project["id"])
label_schema = project_schema.label_schema

print(project["id"], project["name"])
print(project_schema.binding.to_dict())
print([category.id for category in label_schema.classes_for_view("object_3d")])
print([category["id"] for category in project_schema_view.to_dict()["classes"]])

label_schema.validate_category("object_3d", "truck")

current = client.label_schemas.resolve_project(project_id=project["id"])
comparison = compare_project_label_schema_snapshots(project_schema, current)
assert comparison.status == "same"
```

The native Dataset 0.17 Clip stores the immutable identity in
`meta.label_schema` and the complete authoritative snapshot in root
`label_schema.json`. Registry records are used when creating/binding Clips, not
when loading or validating their current contents.

Training code may still define experiment-specific class mappings, but those
mappings must be validated against the resolved project record before use.

### Inspect or diff current state

Dataset identity clients, local manifests, portable cache identities, remote
identity manifests, and update plans use one schema field only:
`label_schema`, containing the complete
`x4d-label-schema-identity-v2` object. `label_schema_fingerprint` and
`clip_label_schema` are not aliases and are rejected. A ready platform record
without the complete identity is also rejected.

`x4d-devkit dataset inspect` writes `x4d-remote-identity-v5`, and
`x4d-devkit dataset diff` writes `x4d-update-plan-v7`. The v7 plan records the
requested materialization policy, each cache's reviewed local materialization
policy, and every Clip's reviewed target and local predecessor identities.
Applying it is fail-closed and idempotent: completed Clips are reported as
already satisfied, while local or remote identity drift requires a new plan.
An explicit `upgrade_materialization_policy` action upgrades a keyframe cache
to `full` by reusing present assets and downloading only missing sweeps. A full
cache also satisfies later keyframe-only requests and is never downgraded.
Older plan contracts are rejected rather than migrated.

### Load a clip

```python
from x4d_devkit import ClipLoader

loader = ClipLoader("/path/to/clip")
print(loader.meta)

for sample in loader.samples:
    for sd in loader.sample_data_for_sample(sample.token):
        print(sd.channel, sd.file_path)
```

### Open a platform work session

Internal tools such as X-Points can open the platform's current clip state
without waiting for a prebuilt archive:

```python
from x4d_devkit import ClipLoader, X4DClient, materialize_work_session_clip
from x4d_devkit.client import WorkSessionConflictError
from x4d_devkit.core.loader import CLIP_WORLD_FRAME_ID

client = X4DClient.from_config()
manifest = client.clip_work_sessions.open(project_id=1812, clip_id="clip-id")
materialize_work_session_clip(
    manifest,
    output_dir="/data/x4d_cache/clips/clip-id",
    client=client,
    materialization_policy="keyframes_only",
)

loader = ClipLoader.from_work_session(manifest, api_url=client.api_url)
anns_world = loader.annotations_for_sample("sample-token", frame=CLIP_WORLD_FRAME_ID)
lidar_asset = loader.point_cloud_asset("sample-token", "configured_lidar_channel")

try:
    response = client.clip_work_sessions.submit_annotations(
        manifest=manifest,
        annotations=manifest.annotations,
    )
    print(response["annotation_revision"])
except WorkSessionConflictError as exc:
    print(exc.current_identities)
    raise
```

The work-session carries the same complete Clip schema snapshot stored at root.
Materialization writes both it and the matching identity in
`meta.label_schema`, so the resulting Clip remains self-contained. A mismatch
fails closed; there is no registry lookup or legacy fallback.

Submit is conditional on the manifest's `source_revision`,
`annotation_revision`, and `base_label_schema_fingerprint`. If the platform state
changed after the session was opened, the client raises
`WorkSessionConflictError` with machine-readable `conflicts`,
`base_identities`, and `current_identities`.

### Coordinate frame transforms

Frames are real frame_id strings: any node in the calibration tree (e.g.
`"LIDAR_TOP"`, `"base_link"`, `"cam_front"`) plus the constant
`"clip_world"` (the SLAM-anchored clip-local world). The legacy aliases
`"sensor"`, `"ego"`, `"world"` are not accepted.

Training converters should read the clip's self-described frame contract
rather than assume fixed channel names. The native annotation frame is the
explicit `meta.annotation_frame_id`; `clip_world` is local to one clip and is
not a global frame across clips.

The installed package exposes the training coordinate contract for logs and
debugging:

```bash
x4d-devkit dataset coordinate-contract
x4d-devkit dataset coordinate-contract --format json
```

```python
from x4d_devkit import COORDINATE_CONTRACT_VERSION, get_training_coordinate_contract
```

```python
from x4d_devkit import ClipLoader
from x4d_devkit.core.loader import CLIP_WORLD_FRAME_ID

loader = ClipLoader("/path/to/clip", label_schema=label_schema)
sd = loader.sample_data_for_channel("LIDAR_TOP")[0]

# Load mixed native records without losing uint64 timestamp_ns
records_sensor = loader.load_point_records(sd)
records_ego = loader.load_point_records(sd, frame=loader.ego_pose_frame_id)
records_world = loader.load_point_records(sd, frame=CLIP_WORLD_FRAME_ID)

# Get annotations transformed to clip-local world
anns_world = loader.annotations_for_sample(sample.token, frame=CLIP_WORLD_FRAME_ID)

# Or to a specific sensor's frame
anns_lidar = loader.annotations_for_sample(sample.token, frame="LIDAR_TOP")

# Get the transform matrix directly (sd is required when clip_world is involved)
T = loader.get_transform(loader.sensor_frame_id(sd), CLIP_WORLD_FRAME_ID, sd=sd)
```

### Validate a clip

```bash
x4d-devkit validate /path/to/clip --label-schema project_schema_record.json
```

```python
from x4d_devkit import validate_clip

report = validate_clip("/path/to/clip", label_schema=label_schema)
print(report)
```

For a remote clip, call `validate_clip_payload(..., asset_sizes=...)` with a
mapping of clip-relative object paths to byte sizes. This applies the same
asset existence and point-cloud layout checks without downloading the assets.

### Serve a Clip preannotation pipeline

Third-party model projects should use `x4d-devkit[inference]` instead of
hand-writing the X-4D preannotation protocol. The pipeline receives an
annotation-isolated `InferenceClip` and returns one complete Dataset 0.17
annotation document.

```python
from x4d_devkit.inference import PreannotationPipeline, PreannotationService


class MyPipeline(PreannotationPipeline):
    profile_id = "centerpoint-track-v1"
    display_name = "CenterPoint + tracking v1"
    pipeline_identity = "sha256:" + "1" * 64
    checkpoint_identity = "sha256:" + "2" * 64
    config_schema = {"type": "object", "additionalProperties": False}

    def predict_clip(self, clip, config, context):
        # Replace with detector -> tracker -> aggregation -> QC.
        # Existing Clip annotations are not exposed through ``clip``.
        return clip.build_annotations([])


PreannotationService(
    service_name="my-preannotation-service",
    service_version="1.0.0",
    pipelines=[MyPipeline()],
    clip_resolver=resolve_clip,
    service_token=SERVICE_TOKEN,
).run(port=9000)
```

See [docs/clip-preannotation-service.md](docs/clip-preannotation-service.md)
and `examples/preannotation_service.py` for the standard integration flow.

### Detection evaluation

```python
from x4d_devkit.eval import DetectionEval, DetectionConfig

config = DetectionConfig(
    class_names=["car", "pedestrian", "bicycle"],
    dist_thresholds=[0.5, 1.0, 2.0, 4.0],
)
evaluator = DetectionEval(config, gt_clips=[...], pred_clips=[...])
result = evaluator.evaluate()
print(f"mAP: {result.mAP:.3f}, NDS: {result.NDS:.3f}")
```

For detectors that intentionally do not predict velocity, construct 7D boxes
and evaluate with `with_velocity=False` so velocity error is excluded from the
primary detection score:

```python
from x4d_devkit.eval import Box, DetectionConfig, evaluate

pred = {
    "sample_1": [
        Box.from_xyzlwhyaw([0, 0, 0, 4, 2, 1.5, 0.0], category="car", score=0.9)
    ]
}
config = DetectionConfig(
    class_names=["car"],
    dist_thresholds=[0.5, 1.0, 2.0, 4.0],
    dist_th_tp=2.0,
    min_recall=0.1,
    min_precision=0.1,
    max_boxes_per_sample=500,
    class_range={"car": 50.0},
    with_velocity=False,
)
result = evaluate(gt, pred, config)
assert result.with_velocity is False
```

## Development

Development and verification run in the repository-owned Docker environment.
A host Python installation is not part of the project workflow.

```bash
make verify
```

Use `make test`, `make lint`, `make docs`, `make package`, or `make shell` for
individual tasks. Dependency changes are resolved and locked with `make lock`.
See [the development environment guide](docs/development.md) for the ownership
boundary and complete workflow.

## Modules

| Module | Description |
|--------|-------------|
| `core` | Data models, token generation, coordinate transforms, clip loader |
| `eval` | Detection evaluation (mAP, TP metrics, NDS) |
| `training_identity` | Strict `x4d-training-identity-v4` parsing and validation |
| `validation` | Clip structure and data validation |
| `client` | X-4D platform API client |

## License

Apache License 2.0
