Metadata-Version: 2.4
Name: deepen-grade
Version: 0.4.1
Summary: Free, local CLI that grades robot-learning datasets (MCAP/rosbag/LeRobot) on hygiene, episode quality, calibration sanity, and cross-episode consistency -- every metric traced to a named paper.
Project-URL: Homepage, https://github.com/mmusa/deepen-grade
Project-URL: Repository, https://github.com/mmusa/deepen-grade
Project-URL: Issues, https://github.com/mmusa/deepen-grade/issues
Project-URL: Deepen AI, https://deepen.ai
Project-URL: Full Audit, https://deepen-robograde.pages.dev
Author-email: Deepen AI <oss@deepen.ai>
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: data-quality,imitation-learning,lerobot,mcap,robot-learning,robotics,rosbag
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Quality Assurance
Requires-Python: >=3.10
Requires-Dist: click>=8.1
Requires-Dist: numpy>=1.24
Requires-Dist: rich>=13.7
Provides-Extra: all
Requires-Dist: huggingface-hub>=0.24; extra == 'all'
Requires-Dist: mcap-ros2-support>=0.5.3; extra == 'all'
Requires-Dist: mcap>=1.1.1; extra == 'all'
Requires-Dist: pandas>=2.0; extra == 'all'
Requires-Dist: pyarrow>=14.0; extra == 'all'
Requires-Dist: rosbags>=0.10.4; extra == 'all'
Provides-Extra: cluster
Requires-Dist: scikit-learn>=1.3; extra == 'cluster'
Provides-Extra: dev
Requires-Dist: huggingface-hub>=0.24; extra == 'dev'
Requires-Dist: mcap-ros2-support>=0.5.3; extra == 'dev'
Requires-Dist: mcap>=1.1.1; extra == 'dev'
Requires-Dist: pandas>=2.0; extra == 'dev'
Requires-Dist: pyarrow>=14.0; extra == 'dev'
Requires-Dist: pytest-cov>=5.0; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: rosbags>=0.10.4; extra == 'dev'
Requires-Dist: ruff>=0.6; extra == 'dev'
Requires-Dist: scikit-learn>=1.3; extra == 'dev'
Provides-Extra: lerobot
Requires-Dist: huggingface-hub>=0.24; extra == 'lerobot'
Requires-Dist: pandas>=2.0; extra == 'lerobot'
Requires-Dist: pyarrow>=14.0; extra == 'lerobot'
Provides-Extra: mcap
Requires-Dist: mcap-ros2-support>=0.5.3; extra == 'mcap'
Requires-Dist: mcap>=1.1.1; extra == 'mcap'
Provides-Extra: ros
Requires-Dist: rosbags>=0.10.4; extra == 'ros'
Description-Content-Type: text/markdown

# deepen-grade

A free, local, `pip install`-able CLI that answers **"is this robot dataset
training-grade?"** in one graded report.

`deepen grade` reads `.mcap` (ROS 2), `.bag` (ROS 1), `.db3` (ROS 2 sqlite), or
a [LeRobot](https://huggingface.co/docs/lerobot) dataset (local path or a
HuggingFace `repo-id`), and runs four layers of checks -- **hygiene**,
**episode quality**, **calibration sanity**, and **cross-episode analysis**
(trajectory consistency, duplicate detection, behavioral clustering, coverage,
drift vs. a previous run) -- entirely on your machine. Nothing uploads by
default. Every number in the report is traced to a named, public source: an
open standard, an ROS REP, or a peer-reviewed paper.

```
$ deepen grade my_episode.mcap

╭─ deepen-grade report ──────────────────────────────╮
│ my_episode.mcap                                    │
│ format: mcap   episodes: 1                          │
╰──────────────────────────────────────────────────────╯
Overall grade: A (100/100)   grade_schema 1.0.0 -- DEFECT-class findings only, see training_value below
Calibration:   NOT ASSESSED
               no calibration metadata found anywhere in this dataset -- calibration
               quality is unknown, not good; verification requires the deep audit

         Per-episode grades
┏━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━┓
┃ Episode     ┃ Grade ┃ Score ┃ Task ┃
┡━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━┩
│ my_episode  │ A     │ 100   │ -    │
└─────────────┴───────┴───────┴──────┘

           Checks -- my_episode
┏━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━┳━━━━━━━━━━━━━━┳...
│ Check                   ┃ Res. ┃ Claim        ┃ Summary
│ Topic frequency & drops │ PASS │ DEFECT       │ 4 topics, no drops detected
│ Cross-modal sync skew   │ WARN │ risk         │ max skew 28.4ms exceeds 15ms budget
│ tf-tree integrity       │ PASS │ DEFECT       │ 6 frames, single-parent tree
│ Idle / stall detection  │ PASS │ risk         │ 3.1% of episode idle
│ LDJ smoothness          │ PASS │ risk         │ LDJ = -14.2
│ Joint-limit saturation  │ PASS │ risk         │ worst dim 'joint_3' saturated 4.2%
│ Gripper chatter         │ PASS │ risk         │ 2 direction reversals (0.31 Hz)
│ Action-state consistency│ N/A  │ not assessed │ action-space semantics undeclared
│ Calibration sanity      │ INFO │ not assessed │ NOT ASSESSED -- no camera calibration metadata found
└──────────────────────────────────────────────────────┘
   Claim column: only DEFECT rows can move "Overall grade" above -- risk/
   characteristic/not assessed/by design never do (see training_value).

    Training-value profile (RISK -- never graded)
┏━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━┳━━━━━━┳━━━━━━┳━━━━━┓
┃ Check                   ┃ Pass ┃ Warn ┃ Fail ┃ N/A ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━╇━━━━━━╇━━━━━━╇━━━━━┩
│ Cross-modal sync skew   │ 0    │ 1    │ 0    │ 0   │
│ Idle / stall detection  │ 1    │ 0    │ 0    │ 0   │
│ LDJ smoothness          │ 1    │ 0    │ 0    │ 0   │
│ Joint-limit saturation  │ 1    │ 0    │ 0    │ 0   │
│ Gripper chatter         │ 1    │ 0    │ 0    │ 0   │
└─────────────────────────────────────────────────────┘

╭─ Funnel ─────────────────────────────────────────────╮
│ What this can't tell you locally:                    │
│   - Whether camera calibration is geometrically       │
│     correct (not just present) -- targetless          │
│     verification is Deepen's sealed, patented audit.  │
│   - A signed acceptance certificate for a data buy.   │
│   - Robot Dataset Quality Index leaderboard entry.    │
│   - Influence-function trainability scoring.          │
│                                                        │
│ Run the full sealed audit: https://deepen-robograde.pages.dev │
╰────────────────────────────────────────────────────────╯

Deepen AI -- https://deepen.ai   |   Full audit: https://deepen-robograde.pages.dev
```

## Install

The base install is deliberately light (numpy, click, rich -- no torch, no
ROS). Add the extra(s) for the formats you actually use:

```bash
pip install deepen-grade                    # base: no format readers yet
pip install "deepen-grade[mcap]"            # .mcap (ROS 2 default)
pip install "deepen-grade[ros]"             # .bag (ROS 1) / .db3 (ROS 2 sqlite)
pip install "deepen-grade[lerobot]"         # LeRobot local path or HF repo-id
pip install "deepen-grade[cluster]"         # enables cross.behavioral_clustering (scikit-learn)
pip install "deepen-grade[all]"             # every format reader (not cluster -- see below)
```

For the latest development version: `pip install "deepen-grade[all] @ git+https://github.com/mmusa/deepen-grade"`

`[ros]` uses the pure-Python [`rosbags`](https://pypi.org/project/rosbags/)
library -- no ROS installation required. `[lerobot]` reads the LeRobot
parquet/JSON dataset format directly (`pandas` + `pyarrow` +
`huggingface_hub`) -- it deliberately does **not** depend on the `lerobot`
training package, which pulls in torch and is much heavier than a CLI needs.
Video files are never downloaded or decoded: every check operates on
recorded state/action arrays and timestamps, not pixels.

If you invoke `deepen grade` on a format whose extra isn't installed, you get
a clear one-line fix (`pip install "deepen-grade[...]"`), not a stack trace.
`[cluster]` is the one exception that isn't a format reader: without it,
every other check (including the rest of cross-episode analysis) runs
exactly as normal, and `cross.behavioral_clustering` alone reports
`NOT_ASSESSED` naming the missing extra -- never an import error.

## Usage

```bash
deepen grade my_episode.mcap
deepen grade my_ros1.bag
deepen grade path/to/ros2_bag/            # directory containing metadata.yaml + *.db3
deepen grade path/to/lerobot_dataset/     # local LeRobotDataset v2/v3 directory
deepen grade some-org/some-dataset        # HuggingFace repo-id (downloaded, no video)

deepen grade my_episode.mcap --json                 # machine-readable, for CI
deepen grade my_episode.mcap --json -o report.json
deepen grade my_episode.mcap --min-grade B          # exit 1 if grade < B (CI gate)

deepen grade some-org/big-dataset --sample 200 --seed 0    # grade a random 200-episode sample
deepen grade some-org/big-dataset --max-episodes 200       # grade the first 200 episodes
deepen grade some-org/container-repo --subdataset team_a/session1   # grade one nested dataset only
deepen grade some-org/gated-dataset --hf-token hf_xxx       # or just set HF_TOKEN
deepen grade my_episode.mcap --quiet                        # no HF progress bars, no CLI chatter
```

### Container repos: datasets nested inside a repo

Some Hub repos aren't a single LeRobot dataset at the root -- they nest one or
more complete LeRobot datasets one or two directories down (a
per-contributor directory, an `ImitationLearning/` wrapper, etc). `deepen
grade` detects this automatically -- for a HuggingFace repo-id, from a single
cheap file-listing API call, before anything is downloaded -- and grades
every nested dataset together, prefixing each episode ID with its
sub-dataset's relative path (`team_a/session1::episode_000042`). A warning in
the report lists every sub-dataset found. Use `--subdataset <relpath>` to
grade just one of them: for a repo-id, this scopes the *download* to that
sub-dataset's own `meta/`/`data/` paths, never fetching the rest of the
container -- a repo with tens of thousands of files across hundreds of
sub-datasets downloads only the one requested.

### Sampling large corpora: `--sample` / `--max-episodes`

Some Hub repos have 100k+ episodes; downloading and reading the whole thing
can OOM a laptop or blow a CI time budget long before grading starts.
`--sample N` grades a uniform random sample of N episodes (seeded via
`--seed`, default `0`, so the same command always picks the same episodes);
`--max-episodes N` grades the first N episodes in natural order instead.
For a repo-id, the *download itself* is scoped to the selection: episode
metadata is fetched first (LeRobot v3's `meta/episodes/*.parquet`, or v2's
per-episode filenames straight from the file listing -- no extra request),
N episode indices are selected from the true dataset-wide count, and only
the parquet file(s) that actually hold those episodes are downloaded. A v3
dataset that shards many episodes per file will still fetch every shard a
selected episode falls in (this is a whole-file download, not a row-level
one), so total bytes scale with the dataset's own chunking density, not with
the sample size alone -- but a repo whose full corpus is tens of gigabytes
routinely downloads a small fraction of that for a few hundred sampled
episodes. A local path or an unrecognized repo layout falls back to reading
(or downloading) every file and filtering locally -- never worse than before,
just not download-scoped. Either way, episodes are materialized one file at
a time, never concatenated into one corpus-sized DataFrame, so peak memory is
bounded by the largest single file, not the corpus. A `"sampling"` block
appears in the JSON report (`{"mode": "sample"|"head", "n", "seed",
"episodes_total"}`, `episodes_total` always the TRUE dataset-wide count, even
when the download itself only touched a subset of files) and the terminal
report prints a `GRADED ON A SAMPLE` banner; grade letters themselves are
unaffected.

### LeRobot format versions: v1.0, v2, v3

`deepen grade` reads all three. v2/v3 store every per-frame scalar column
(`episode_index`, `frame_index`, `timestamp`, `task_index`) as a bare value;
v1.0 stores the same columns wrapped in a length-1 array per row (its
`features.json` declares `"shape": [1]` even for scalars). This is
transparent -- detected and squeezed back to plain values before anything
else runs -- so v1.0 repos read exactly like v2/v3 ones, with one caveat:
v1.0's on-Hub layout is always per-episode files (`episode_{index:06d}.parquet`,
no `meta/episodes/*.parquet` shard index), so `--sample`/`--max-episodes`
download-scoping for a v1.0 repo-id uses the same filename-based path as v2
(see above), not v3's chunk-mapping.

### Partial reports: `--json -o` writes incrementally

With `--json -o report.json`, the report is written to disk incrementally --
at a progress-proportional interval (roughly every 10% of graded episodes,
never more often than every 200), not just at the end -- via a write-to-temp-
file-then-rename so the file on disk is always a complete, parseable JSON
document, never a half-written one. An in-progress write has `"partial":
true`; the final write sets it to `false`. This means a crash or CI timeout
mid-run still leaves a valid, gradeable-so-far report behind instead of
nothing.

### HuggingFace downloads: retries and rate limits

Repo-id downloads retry up to 3 times with exponential backoff on a dropped
connection or other transient Hub error. An HTTP 429 (rate limited) fails
immediately with an actionable message instead of retrying into the same
wall: anonymous downloads have a lower rate limit than authenticated ones, so
pass `--hf-token` (or set `HF_TOKEN`) to raise it.

## What it checks (and what each check cites)

Every check result carries a `claim_type` -- see **Grading** below for what
that means and why only some of these checks can move the letter.

### A. Hygiene -- deterministic, applies to any recording
| Check | What it measures | Claim type | Cited source |
|---|---|---|---|
| Topic frequency & drops | per-topic observed rate vs. that topic's own median interval, gaps > 3x it | DEFECT | ISO/WD 26264-1 companion (arXiv:2606.19769) |
| Cross-modal timestamp skew | ms skew between vision/proprioception/lidar streams | RISK | arXiv:2606.19769 |
| tf-tree integrity | single-parent tree, no static/dynamic conflicts, no cycles | DEFECT | REP 105 |
| Message-schema & state/action-dim consistency | consistent types/dims across a dataset's episodes | DEFECT | arXiv:2606.19769, ASAM OpenLABEL |

Cross-modal skew is RISK, not DEFECT, because its ms budget
(`SKEW_WARN_MS`/`SKEW_FAIL_MS` in `checks/hygiene.py`) is a Deepen convention
parameterized by sensor class, not a sourced standard -- see the module
docstring.

### B. Episode quality -- published metric implementations, all training-value (RISK)
| Check | What it measures | Claim type | Cited source |
|---|---|---|---|
| Idle / stall | near-zero normalized action/state velocity runs | RISK | DQAF (arXiv:2605.26349), SCIZOR (arXiv:2505.22626) |
| Smoothness | log-dimensionless-jerk of the speed profile | RISK | Balasubramanian et al. 2015 (IEEE TBME), DQAF |
| Joint-limit saturation | time spent near the DATASET-WIDE observed value range | RISK | DQAF |
| Gripper chatter | direction-reversal rate of gripper position | RISK | DQAF |
| Action-state consistency | normalized tracking error between commanded action and state | CHARACTERISTIC (dim mismatch) / NOT_ASSESSED (undeclared action space) / RISK (declared absolute) | DQAF, SCIZOR |

Every episode-quality check is a training-value proxy -- see **Grading**
below for why none of them can move the letter, and why that's the fix, not
a regression.

**On thresholds:** the papers above define *what to measure*; the specific
pass/warn/fail cutoffs (e.g. "stall_frac > 30% is a FAIL") are deepen-grade's
own conservative, documented defaults -- see
[`src/deepen_grade/checks/episode_quality.py`](src/deepen_grade/checks/episode_quality.py)
for every constant in one place. No universal numeric cutoff across every
robot embodiment and control rate exists in the literature; these are sane
starting points, not claims about what the cited papers themselves recommend.

### C. Calibration -- its own verdict, never part of the grade
deepen-grade checks that intrinsics/extrinsics are **present** and not
**obviously broken** (missing, NaN, an untouched identity/zero default).
That's it. It does not verify that a calibration is *geometrically correct*
-- doing that from a recording alone (reprojection, epipolar consistency,
drift over a session) is Deepen's patented targetless calibration
verification, which stays sealed and server-side.

Because a well-formed but geometrically *wrong* calibration passes every
local check, calibration is deliberately **excluded from the letter grade**
and reported as its own top-level verdict instead:

- `NOT ASSESSED` -- no calibration metadata found; quality is unknown, not good.
- `PRESENT -- ACCURACY NOT VERIFIED` -- metadata present and structurally sound.
- `STRUCTURALLY BROKEN` -- metadata present but obviously broken (NaN/unset default).

See
[`src/deepen_grade/checks/calibration_sanity.py`](src/deepen_grade/checks/calibration_sanity.py)
for the firewall this module enforces on itself.

Calibration metadata itself is read from whatever the source format actually
carries: for `.mcap`/`.bag`/`.db3`, from a `sensor_msgs/CameraInfo` message on
each camera topic (intrinsics) paired with a matching `/tf_static` transform
by frame_id (extrinsics, left `None` -- never invented -- if no matching
transform was published); for LeRobot, from the `meta/calibration.json`
sidecar convention or (narrowly) a DROID-style `meta/cam2base_extrinsics.json`
/ `meta/info.json["calibration"]` block.

### D. Plausibility -- does this even look like robot data?
A structurally valid LeRobot/mcap/bag file can still contain something that
isn't robot-learning data at all -- a non-robotics dataset reshaped to fit the
schema, for instance. `Robot-data plausibility` is a cheap, dataset-level
heuristic check that flags this: no state/action trajectory anywhere, a state
that never actually changes across any episode, or no usable timestamps.
It's always `INFO` (or `N/A` when nothing looks wrong), never affects the
score, and is deliberately **uncited** -- "does this look like robot data" is
not a metric any of the papers above define. See
[`src/deepen_grade/checks/plausibility.py`](src/deepen_grade/checks/plausibility.py).

### E. Cross-episode -- checks that need the whole dataset in view at once
Everything above grades one episode at a time. This layer groups episodes by
`episode.task` (untasked episodes form their own group) and compares them to
each other -- still only state/action arrays and metadata, no video, no
network, no new mandatory dependency.

| Check | What it measures | Claim type | Cited source |
|---|---|---|---|
| Trajectory consistency | 5 per-episode scalars (duration, peak normalized speed, motionless fraction, mean normalized jerk, final-pose distance to the group mean), Tukey-IQR-fenced within the task group | RISK | DQAF, LDJ (Balasubramanian et al.), Tukey 1977 |
| Endpoint agreement | start/end pose distance to the group mean, fenced the same way | RISK | DQAF, Tukey 1977 |
| Duplicate detection | exact: content hash of raw state/action/timestamps. near: cosine similarity + duration ratio on a temporal-order-preserving resampled signature, binary/low-cardinality dims excluded | exact: DEFECT, near: RISK | SCIZOR |
| Behavioral clustering | PCA + HDBSCAN over the resampled signatures (same binary/low-cardinality-dim exclusion as duplicate detection) | CHARACTERISTIC | Campello/Moulavi/Sander 2013, SCIZOR |
| Condition coverage | episodes/imbalance per task, success/failure balance, duration percentiles, state-space coverage per task | CHARACTERISTIC | ISO/WD 26264-1 companion, DQAF |
| Action/state variance profile | per-task-group variance over the resampled signature, top-5 highest-variance cells | CHARACTERISTIC | DQAF |
| Drift vs. `--baseline` | two-sample Kolmogorov-Smirnov test on this run's trajectory-consistency scalars (state-derived) plus action-magnitude scalars vs. a previous report's | RISK | Massey 1951 (KS test), DQAF |

Only `cross.duplicates`' exact-match branch is `DEFECT`-eligible -- a
byte-identical episode pair is an objective fact, not a proxy, so it's the
one cross-episode finding that can move the letter grade (see **Grading**
below). Everything else here is `RISK` (a training-quality proxy) or
`CHARACTERISTIC` (dataset inventory), same as the rest of the report.

**One-sided pose-distance fences with a floor.** Pose-*distance* metrics
(start/end pose, final pose) fence one-sided -- only unusually *far* from the
group mean flags; unusually close never does -- and carry a
practical-significance floor (`POSE_DIST_FLOOR`, 5% of per-dim normalized
range) below which a distance never flags. Without the floor, a dataset where
nearly every episode starts from the same scripted reset pose has a collapsed
distance IQR, and the fence lands close enough to zero to flag poses that
differ from the mean by a fraction of a percent of joint range.

**Task-group gate.** Every per-group statistic (trajectory consistency,
endpoint agreement, behavioral clustering, the variance profile) requires at
least 4 episodes with a usable trajectory in that task group; smaller groups
report `NOT_ASSESSED` naming the gate rather than fencing against a quartile
computed from a handful of points. `cross.duplicates` has its own,
dataset-wide version of this: below 2 episodes total, there is nothing to
compare against, so it abstains too instead of reporting a vacuous "no
duplicate found."

**Temporal-order-aware deduplication.** The resampled signature (32
normalized-time steps x the leading 32 state/action dims, per-dim normalized
by the dataset-wide range) preserves each episode's own chronological order.
A reversed copy of an episode therefore produces a different signature from
its forward original -- it is not silently folded into the same duplicate
group unless the underlying motion happens to be genuinely time-symmetric.

**Binary/low-cardinality dims are excluded from the near-dup/clustering
distance.** After per-dim range normalization every dim occupies the same
`[0, 1]` span, so a binary dim (a gripper open/close flag is the textbook
case) contributes exactly as much to the resampled signature's L2 norm as a
continuous joint sweeping its entire observed range -- a mere timing offset
in a dim like that can move cosine similarity across the near-dup threshold
in either direction, independent of whether the rest of the trajectory
actually matches. Dims with fewer than 5 distinct values dataset-wide (the
same threshold `joint_limit_saturation` uses) are zeroed out of the
signature used for `cross.duplicates`' near-match branch and
`cross.behavioral_clustering` -- never out of `action_variance_profile`,
which legitimately wants to show them as a dataset characteristic.

**O(n^2) guards.** Near-duplicate comparison is pairwise by construction.
Above 5,000 episodes with a usable signature, comparison scopes down to
within each task group; any group still larger than that after scoping (one
task covering most of a large dataset, where scoping alone doesn't help) or
a dataset above 20,000 episodes overall abstains with `NOT_ASSESSED` naming
the guard, rather than running an unbounded comparison. Exact-duplicate
detection is O(n) (a hash lookup) and is unaffected by either guard.

**`--baseline PATH`: drift vs. a previous run.**
```bash
deepen grade path/to/dataset --json --baseline old_report.json
```
Requires `old_report.json` to be a `--json` report with `schema_version >=
6` (this release's `episode_metrics` field). An older or otherwise
incompatible report is a valid file, not a CLI error -- `cross.drift` simply
reports `NOT_ASSESSED` naming the gap; a missing or unparseable file *is* a
CLI error (exit 2). A flagged metric names its Kolmogorov-Smirnov `D`
statistic, `p`-value, and before/after medians. Drift is a proxy signal
(`RISK`) and never moves the letter grade.

**Metrics compared.** Five trajectory-consistency scalars (duration, peak
normalized speed, motionless fraction, mean normalized jerk, final-pose
distance) plus two action-derived ones, `action_mean_abs` and
`action_magnitude_range` (mean and range of the per-dim-range-normalized
action array). The five state-derived metrics fall back to action only when
an episode has no state at all, so on any dataset with both channels they
never move if only the action channel changes -- the action-derived pair
closes that gap: an action-space-only corruption (e.g. action magnitudes
scaled down while state is untouched) is invisible to the first five and
caught by the last two.

**Fingerprint.** Every `--json` report carries a `"fingerprint"`: a sha256
over the sorted per-episode exact-duplicate content hashes.
Permutation-invariant (episode order never changes it) and content-sensitive
(any changed episode changes it) -- two reports quoting the same fingerprint
were graded on identical data. Also printed as a line in the terminal
report's header panel. A `--sample`/`--max-episodes` run's fingerprint
certifies only the episodes actually graded, not the whole dataset --
deterministic given `--seed`, so re-running the same sample reproduces the
same fingerprint, but two *different* samples of the same underlying dataset
legitimately have different ones. Check the report's `"sampling"` block
before comparing two fingerprints. A dataset with no decodable state/action
anywhere (no exact-duplicate hash on any episode -- e.g. a vision/language-
only modality) falls back to hashing each episode's own topic schema
(name/type/sample-count) instead of hashing nothing: two different
no-trajectory datasets no longer collide on the fixed `sha256("")` value.

**Constants.** Every threshold used above (the Tukey IQR multiplier, the
near-duplicate cosine/duration-ratio bar, the O(n^2) guard sizes, the
clustering noise-degenerate fraction, the drift D/p bar, and everything
else) is a named constant at the top of
[`src/deepen_grade/checks/cross_episode.py`](src/deepen_grade/checks/cross_episode.py)
-- same "one place to audit or override" discipline as
`episode_quality.py`'s thresholds (see that section's own note above): the
cited papers/methods define what to measure and how to fence it, not the
specific numeric cutoffs, which are deepen-grade's own conservative,
documented defaults.

## Grading: DEFECT-only letter, training_value profile, grade_schema

`grade_schema` (currently `"1.0.0"`, carried on every report) is a rewrite of
how the letter is computed, not just a version bump:

**The letter is DEFECT-only.** Every check result carries a `claim_type` --
`DEFECT`, `RISK`, `CHARACTERISTIC`, `BY_DESIGN`, or `NOT_ASSESSED` (see
[`checks/base.py`](src/deepen_grade/checks/base.py)'s `ClaimType`). Only
`DEFECT` findings can ever move the letter. `RISK` findings (idle/jerk/
chatter/joint-limit/action-state/cross-modal-skew -- every episode-quality
check, plus cross-modal skew) are proxy training-quality signals whose
thresholds are conventions, not sourced correctness claims; they're reported
in the report's `training_value` section instead (counts per check, zero
letter impact). `CHARACTERISTIC` (e.g. action/state dims differing by
architecture), `BY_DESIGN`, and `NOT_ASSESSED` (a required gate didn't hold --
e.g. an undeclared action-space semantics field) never touch either.

**Aggregation is defect DENSITY, worst-class-bounded -- not a penalty
average.** For every `DEFECT`-class check_id, deepen-grade computes that
check's defect density: `defective findings (WARN or FAIL) / gradable
findings (anything but N/A)`. The dataset's 0-100 integrity score is
`100 * (1 - worst_density)`, where `worst_density` is the MAXIMUM density
across DEFECT check classes -- never their average. This is deliberate: a
single systemic defect (e.g. every episode's tf-tree is broken) must not be
diluted by four other clean classes into a passing grade. The exact
functional form and worked edge cases live in
[`grading.py`](src/deepen_grade/grading.py)'s `integrity_score` docstring, in
the code, not just here. Grade bands are the same `GRADE_BANDS` table as
before (A >= 90, B >= 75, C >= 60, D >= 40, F otherwise), now applied to this
density-derived score instead of a WARN/FAIL penalty sum.

Calibration keeps its own separate top-level verdict, unaffected by any of
this -- see section C above.

**The NOT ASSESSED floor.** An empty gradable set must never auto-pass
(GRADING_TAXONOMY_V1.md section 3). If a dataset is small or sparse enough
that every DEFECT-eligible check individually abstains -- no topics with
enough messages for topic frequency, no tf concept, fewer than 2 episodes to
compare schemas across -- `defect_classes` comes back with no gradable
entries anywhere, and the dataset's letter is the string `"NOT ASSESSED"`
(not a letter band, not a 0-100 score's worth of confidence) instead of the
vacuous 100/A that a bare `max(density, default=0.0)` would otherwise hand
back. The report's `warnings` names every gate that didn't hold. A dataset
where DEFECT checks actually RAN and PASSED -- even trivially, even with
degenerate/garbage content -- is unaffected: real gradable evidence (density
0.0 included) always earns its letter; only genuine silence triggers this.
`--min-grade` treats `NOT ASSESSED` as failing any requested bar, including
`F`, since "couldn't be graded at all" must never read as "cleared the bar."

**Content-plausibility flag.** `plausibility.robot_data` (section D above) is
an index-curation gate, not a letter input -- but a confident-looking grade
must never silently coexist with "this doesn't look like robot-learning
data." When it fires, both `--json` (`content_plausibility: {"flagged",
"detail"}`, top-level, next to `overall`) and the terminal report (a
`CONTENT PLAUSIBILITY FLAG` line directly under `Overall grade`) surface it
prominently rather than leaving it to be found only in the dataset-level
checks table.

**Two checks were fixed as part of this rewrite** (both confirmed on a
50k+-episode BridgeData2 run before the fix):
- `action_state_consistency` used to hard-FAIL any dataset whose action
  space is a delta/velocity/torque/tokenized convention (the majority of
  modern manipulation datasets, e.g. BridgeData2's zero-centered deltas),
  because it compared action to state unconditionally. It now only runs the
  element-wise comparison when a trajectory *declares* `action_space ==
  "absolute"`; every other value (including the honest default of
  undeclared/`None` -- no ingest adapter in this codebase populates it yet)
  returns `NOT_ASSESSED` naming the missing declaration. A dim-count
  mismatch is now `CHARACTERISTIC` ("architectural, not a defect"), never a
  FAIL.
- `joint_limit_saturation` used to use each *episode's own* min/max as the
  limit proxy, so any low-variance dim (a binary 0/1 gripper flag; a short
  episode that only visits a narrow slice of a joint's true range) read as
  permanently "at the limit." It now computes the reference range across
  the WHOLE dataset (aggregated once before per-episode grading starts),
  and excludes binary/discrete dims (fewer than 5 distinct values
  dataset-wide) and near-constant dims outright. It remains RISK-only: there
  is no declared-joint-limits field in this codebase yet, so this is always
  a proxy, never a DEFECT.

**Terminal styling.** Per-check tables carry a `Claim` column (`DEFECT` /
`risk` / `characteristic` / `not assessed` / `by design`), styled distinctly
from the PASS/WARN/FAIL `Result` column, with a footnote reiterating that
only `DEFECT` rows can move the letter above -- a RISK-class FAIL sitting
right under an `A` grade must never read as if it caused that grade.

**`--json` report additions (schema_version 5):** `grade_schema`,
`defect_classes` (the density behind `overall`, per DEFECT check_id),
`training_value` (RISK-class severity counts, per check_id), and
`content_plausibility` (`{"flagged", "detail"}`). Nothing existing was
removed or renamed. `overall.grade` can now also be the string
`"NOT ASSESSED"` (see the floor, above).

**`--json` report additions (schema_version 6, cross-episode analysis):**
`fingerprint` (string), `episode_metrics` (per-episode trajectory-consistency
scalars), and `drift` (present iff `--baseline` was passed) -- see
**Cross-episode** above. Every `cross.*` check result shows up in
`episodes[].checks` / `dataset_level_checks` exactly like any other check.

**A note on comparability:** grades produced under `grade_schema` earlier
than `1.0.0` are not comparable to grades under `1.0.0` -- the method changed,
not the data. See [CHANGELOG.md](CHANGELOG.md).

## What this can't tell you locally

Every report ends with a funnel section, because it's true, not because it's
a marketing footer:

- Whether calibration is geometrically **correct**, not just present.
- A signed, reproducible acceptance certificate for a data purchase.
- Entry in the public Robot Dataset Quality Index leaderboard.
- Influence-function trainability scoring (which episodes help your policy).

Full sealed audit: **https://deepen-robograde.pages.dev**

## A self-assessment, not a certification

deepen-grade runs on your machine, on your data, with nothing checked by
Deepen -- so its report is a **self-assessment**, and it says so on its face
(`report_type: "self-assessment"` in `--json`). There is deliberately no
claimable "verified" badge here: a trust mark Deepen never saw would be an
honor-system claim, which is exactly what this tool exists to replace.
Signed, third-party attestation -- a report a counterparty can rely on --
is the sealed deep audit at [deepen-robograde.pages.dev](https://deepen-robograde.pages.dev).

## CI: fail the data PR on a bad episode

```bash
deepen grade path/to/dataset --json --min-grade B
```
exits non-zero if the overall grade is worse than `B`. A ready-to-use
composite GitHub Action is in [`action/`](action/action.yml); see
[`examples/ci-gate.yml`](examples/ci-gate.yml) (copy it into `.github/workflows/` in your repo)
for a full workflow that runs it on every PR touching a dataset directory.

```yaml
- uses: mmusa/deepen-grade@v0
  with:
    path: path/to/dataset
    min-grade: B
    extras: lerobot   # mcap | ros | lerobot | all
```

## Privacy

Nothing uploads by default -- full stop. `--share-report` is an explicit
opt-in flag for a future sealed/signed report from deepen-robograde.pages.dev; **it
is not implemented in this release** (see
[`src/deepen_grade/report/share.py`](src/deepen_grade/report/share.py)) and
performs no network call. Use `--share-report --share-report-dry-run
payload.json` to inspect exactly what the eventual payload would contain.

## Development

```bash
git clone https://github.com/mmusa/deepen-grade
cd deepen-grade
python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"
pytest
```

## License

Apache-2.0 -- see [LICENSE](LICENSE).

---

Built by [Deepen AI](https://deepen.ai), the data infrastructure layer for
physical AI. `deepen-grade` is the free, always-local doorway; the full
sealed audit (targetless calibration verification, acceptance certification)
lives at [deepen-robograde.pages.dev](https://deepen-robograde.pages.dev).
