Metadata-Version: 2.5
Name: gi-ingest
Version: 0.5.0
Summary: Offload robotics recordings from TF cards and deliver them to GILabs Data Platform
License: Proprietary
Requires-Python: >=3.11
Requires-Dist: boto3>=1.34
Requires-Dist: httpx>=0.27
Requires-Dist: mcap>=1.2
Requires-Dist: rich>=13.7
Requires-Dist: typer>=0.12
Provides-Extra: dev
Requires-Dist: moto[s3]>=5.0; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.24; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: ruff>=0.6; extra == 'dev'
Description-Content-Type: text/markdown

# gi-ingest CLI Guide

Offload robotics recordings from TF cards and deliver them to GILabs Data
Platform.

Reads ego rig cards, copies each session to a local disk and verifies the copy,
delivers the lot to GILabs as one sealed batch, and wipes the cards only once
the server confirms it holds the data. Built for the way an offload actually
goes: someone swapping cards at a reader for half an hour, a link pushing bytes
overnight with nobody watching, and a wipe the next morning.

## Installation

macOS and Linux. Windows is not supported: cards are looked for under
`/Volumes` and `/media`, so a Windows install finds nothing.

**Prerequisite — uv**, once per machine. It fetches its own Python, so nothing
else needs installing first.

```bash
curl -LsSf https://astral.sh/uv/install.sh | sh
```

Reopen your terminal afterwards so `uv` is on your PATH, then install the CLI
into an isolated environment of its own:

```bash
uv tool install gi-ingest
gi-ingest --version
```

## Authentication

Get a token from the vendor portal at
[vendor.gilabs.xyz](https://vendor.gilabs.xyz) → **Upload tokens**. It is shown
once and stored only as a hash, so copy it when it appears — the portal prints
the whole command below with your token already in it.

```bash
gi-ingest login --api-key gik_...
gi-ingest doctor
```

`doctor` checks the four things an overnight run depends on: your version, your
token, that a delivery destination has been granted, and that the staging disk
has room. Run it before a session rather than discovering a problem at 2am.

## Configuration

Everything is stored per machine in `~/.config/gi-ingest/config.json`, which
also holds your token and is written owner-readable only.

```bash
gi-ingest config show                                   # settings and where they live
gi-ingest config set staging-dir /Volumes/BigDisk/gilabs
```

| Setting | What it is | Default |
|---|---|---|
| `staging-dir` | Where cards are copied to. Rarely somewhere a multi-terabyte round fits by default | `~/gilabs-staging` |
| `project-id` | Which project deliveries are filed against. Set it with `gi-ingest destinations --select` | unset |
| `ledger-path` | The local record of what is staged, in flight and delivered | `~/.local/state/gi-ingest/ledger.db` |
| `api-url` | The GILabs backend | production |

Any of them can be overridden for a single run without changing what is stored:

```bash
gi-ingest stage --staging /Volumes/OtherDisk/gilabs
gi-ingest stage --project <id>
gi-ingest upload --ledger /Volumes/OtherDisk/other.db
```

`--ledger` is how you keep two independent offload runs on one machine from
sharing state — each needs its own, or they will see each other's queues.

The ledger is worth knowing about even though you rarely touch it. It is what
makes an interrupted upload resumable and lets `reclaim` know which cards are
safe to wipe, so it is the one file where losing it costs you a re-stage rather
than nothing. It survives upgrades, including the rename from `gilabs-ingest`.

## Upload a batch

One loop per round of cards: copy them, push them overnight, wipe them the next
morning. Three commands rather than one because they run on two different
timescales, and fusing them turns a 30-minute attended task into an all-night
one.

| Phase | When | What it does |
|---|---|---|
| `stage` | attended, minutes per card | Copies cards to a local disk and verifies each copy |
| `upload` | unattended, hours | Pushes everything staged as **one delivery**, overnight |
| `reclaim` | attended, next morning | Wipes only what the server confirmed |

```bash
# 1. Drain every card. Swap cards as each finishes; nothing is wiped yet.
gi-ingest stage
gi-ingest stage --watch                 # or let it auto-copy on insert

# 2. Leave this running. One delivery, resumable, safe to nohup.
gi-ingest upload --max-bandwidth 50M

# 3. Next morning, with the cards back in the reader.
gi-ingest reclaim
```

**1 — `stage`** copies each session to the staging disk, checksums it on the way
back out, and tells you which project the round is bound for before it starts.
The card is left completely intact, so it can be pulled the moment a copy
finishes.

**2 — `upload`** opens a **batch** and receives AWS credentials scoped to that
batch's folder alone; they cannot read or write anything else, including your
own other deliveries. Files go straight to S3, and the credentials are re-minted
as they expire, so a multi-hour run and a laptop that sleeps both just work. It
then **seals** the batch with a manifest listing every episode, part, size and
SHA-256, and GILabs verifies every declared file is present at its declared size
before ingesting anything — a half-finished upload can never be processed as if
it were complete.

**3 — `reclaim`** wipes only what the server has confirmed, checking each card's
own metadata against its records first. It clears the staging copy at the same
time, which is the one that actually fills a disk.

## Commands

```bash
gi-ingest --version       # what you are running
gi-ingest whoami          # vendor, scopes, and where you may deliver
gi-ingest destinations    # the org/project list, with ids for --project
gi-ingest doctor          # version, token, destination, disk
gi-ingest status            # what is still in play — staged, in flight, failed
gi-ingest status --card ID  # ...and that card's session names
gi-ingest history           # past deliveries: cards, episodes, size
gi-ingest history --card ID # what one card delivered
gi-ingest history --batch btch_...  # one delivery in detail, from the server
gi-ingest retry             # re-queue anything that failed
gi-ingest attribute         # fill in operator/environment for staged sessions
gi-ingest reset --card ... # send staged episodes back a step (see below)
```

## FAQ

### Who recorded this, and where?

At the end of each card, `stage` asks — once per recording date, since a day's
work is normally one place and one person:

```
2 session(s) on this card have no operator or environment.
Environment (name or id): warehouse
  env_00003  Manila warehouse · warehouse
Operator (name or id): maria
  operator_0001  Maria Santos · collector
Attributed 2 session(s) to env_00003, operator operator_0001.
```

Type any part of the name — the place, the person — and pick from what matches.
The ids come from the environments and operators your team registered in the
vendor portal; register them there first if the list is empty.

A card holding sessions from more than one date is asked about once per date,
and says so before the first prompt. Each date is a separate answer. Two
sittings on the same date share an answer — a morning and an evening in the
same place is the common case, and splitting it would ask twice for one answer.

**Both answers are required.** A blank line re-asks rather than skipping. This
used to be optional, and the result was 55 of 1,057 delivered episodes arriving
with the fields empty — every one of them reconstructed afterwards from a
vendor spreadsheet, and some of them not recoverable at all. Answering takes
seconds while the card is still in the reader. `^C` still aborts the whole
command if you genuinely cannot answer; the copy is left intact.

The answers are written into each staged session's `session.json`, which is
where the platform reads them from — the card itself is never modified.

### "No environments are registered for this vendor" — stage refused

The registry is checked before the first card is read, so this costs you a
message rather than an hour of copying. It means the server answered and the
answer was that this vendor has nothing to attribute a recording to. Register
the sites and people in the vendor portal, then stage again. Waiting does not
help — this is paperwork, not an outage.

### "The registry is unavailable" — stage carried on anyway

The opposite case, and deliberately not a refusal. An outage is a five-minute
problem and reading a card is an hour, so the copy goes ahead:

```
The operator and environment registry is unavailable (connection refused).
  Staging anyway — these sessions will be held back at upload until
  gi-ingest attribute has run.
```

Those sessions are held at upload. Run `gi-ingest attribute` once the server is
reachable and they go with the next delivery.

### I staged a card with no terminal attached, and now upload refuses it

A `stage` run under `nohup`, from a script, or with `--watch` in the background
has nobody to ask, so it stages the copy and says so:

```
! 12 session(s) staged with no attribution, and no terminal to ask.
  Run gi-ingest attribute before uploading; upload will refuse them.
```

`upload` then holds those sessions back rather than delivering them blank:

```
held session_00031-1787265461: no operator/environment
Refusing 12 session(s) with no attribution. Run gi-ingest attribute to fill them in.
```

Run `gi-ingest attribute` from a terminal. It asks the same questions, once per
recording date, and releases the sessions:

```bash
gi-ingest attribute
gi-ingest upload
```

Nothing is lost while they wait — the footage is on the staging disk and the
card still holds its second copy. Do it before the overnight run, while the
answers are still fresh.

### Why was a session marked unusable?

For one reason, and `stage` prints it:

```
unusable session_00006: no IMU stream — the device did not record one
```

The recording carries video but no IMU stream. This is not a fault in the copy
— some rigs have no working IMU and record video only — and the footage cannot
be reconstructed without it, so it is caught here rather than discovered days
later on a chart that will not load. It is checked before the copy, reading the
index at the tail of each `.mcap` part rather than the footage, so a session
that will not be delivered costs neither the staging disk nor a night of
upload. Pass `--allow-missing-imu` to stage these anyway; `stage` says so when
you do. `gi-ingest status --card <id>` shows the reason again later, under
**Why**.

**Short recordings are no longer refused here.** How short is too short is the
destination project's policy, and the project is GILabs' to read, not this
tool's to guess: a client that had not been upgraded could refuse footage the
project would have accepted, and refused footage cannot be re-taken. Short
clips now stage and upload like anything else, and GILabs marks the ones it
will not take.

Unusable sessions are not uploaded, and `reclaim` asks before wiping them:

```
1 session(s) on these cards were refused (94.0 MiB).
They were never uploaded, so wiping them deletes the only copy.
  1 × no IMU stream — the device did not record one
Wipe them too? [y/N]
```

Pressing Enter keeps them, and a `reclaim` with no terminal — scheduled, or
piped — always keeps them. `--wipe-unusable` / `--keep-unusable` answers ahead
of time. If you disagree with the refusal itself, `gi-ingest reset --card <id>
--to staged` puts the session back in the upload queue instead.

A session that cannot be **read** is never refused by either check. A `.umi`
session has no index at all, and a recording interrupted by a power cut has
lost its — withholding real footage over a missing few kilobytes is the more
expensive mistake. For the same reason, a session is only refused for a missing
IMU when every one of its parts was read and none had one: "no IMU in the parts
we could open" is not the same statement.

### Can I redo a card I have already staged?

Yes — `gi-ingest reset`:

```bash
gi-ingest status                                   # lists the card ids
gi-ingest status --card sd-9fa13                   # ...and that card's session names
gi-ingest reset --card sd-9fa13                  # forget the copy; stage it again from the card
gi-ingest reset --card sd-9fa13 --to staged      # keep the copy; upload it again
gi-ingest reset --session session_00042 --dry-run # one session, no changes made
```

The card ID is assigned at the first stage pass and saved in
`.gi-ingest-card-id.json` at the card root. It starts with `sd-` and remains the
same when the card moves between readers or mount names. Filesystem UUIDs are
kept unchanged: several FAT/exFAT cards can share one, and all of them must be
staged separately. Keep the marker on its card; do not copy it to another card.
`gi-ingest status` lists IDs beside the volume labels, and
`gi-ingest status --card <id>` names that card's sessions.

On upgrade, the first stage pass verifies recording hashes before migrating
legacy ledger entries. Existing staging paths, episode IDs, collection dates,
and upload/batch states are retained; a verified staged recording is not copied
again. This pass can take longer because it reads the card and retained staging
copy. Missing or conflicting evidence is reported for investigation. An
unmarked legacy card needs this stage pass before `reclaim` can identify it.
Invalid, unwritable or duplicated card markers stop staging before the first
copy. Formatting a card removes its marker and the next stage assigns a new ID.

Both `--card` and `--session` take a **unique prefix**, so `--card sd-9fa13` and
`--session session_00008` are usually enough. If a prefix matches two, the
candidates are listed and nothing is changed.

`status` includes delivered work; `gi-ingest status --card <id>` names it. That
is also the answer when `reset` says "already delivered": the episode is the
server's now, and there is nothing local left to redo.

`--to discovered` (the default) **deletes the staged copy** so `stage` takes it
off the card again — use it when you have reason to distrust the copy, and only
while the card still holds the footage. It tells you how much it is about to
delete and asks before doing it; `--dry-run` shows the same without changing
anything. `--to staged` keeps the copy and lets `upload` send it once more.

It will not touch an episode the server has already confirmed, and it will not
send one back to `discovered` after its card has been reclaimed: in both cases
the thing it would send you back to no longer exists. Failures do not need it —
`gi-ingest retry` re-queues those.

### The upload was interrupted — do I start again?

No. Re-run `gi-ingest upload`. It re-uses the same open delivery and skips files
already sent, so a restart costs a listing rather than the bytes. That covers a
kill, a crash, a dropped connection or a closed lid.

### Why is wiping the cards a separate step?

`upload` finishes hours after the cards were pulled, so it cannot wipe them —
they are back in the rig by then.

It also means **the card keeps a second copy until `reclaim` runs**, which is
deliberate: between `stage` and confirmation the staging disk would otherwise be
the sole home of a whole collection round.

### Where do the cards get copied to?

`~/gilabs-staging` by default, which is rarely where a multi-terabyte round
fits. Point it at the right disk once — see [Configuration](#configuration).

### Which project does a delivery go to?

Every delivery is filed against one project, and destinations are granted by
GILabs — you cannot add one yourself. If the portal's **Destinations** tab is
empty, ask your GILabs contact.

With one granted destination there is nothing to choose and no flag to pass.
**With several, you have to say which**, before staging rather than at upload:

```bash
gi-ingest destinations --select     # pick from a list; saved for this machine
gi-ingest destinations              # just show them
gi-ingest stage --project <id>      # or choose for a single run
```

You will also be asked directly the first time `stage` needs to know and the
answer is not obvious. A project id is a uuid, and retyping one off a portal
page is how a delivery ends up under the wrong project.

`stage` refuses to copy anything until this is settled, and both `stage` and
`upload` print the destination before they start. That is deliberate rather than
fussy: a delivery goes to one project, so cards staged for two of them in the
same session would be filed together and only one of them correctly — and
nothing about the upload would look wrong while it happened.

For the same reason `upload` refuses a queue holding episodes staged for
different projects. Upload one destination's worth at a time.

### What happens if a card was already delivered by someone else?

Stage and upload it. GILabs recognises the delivery from the contents of the
episode — the digests of its parts, computed during the staging copy — and
files it against the one it already holds rather than ingesting it twice.

This costs the upload. It buys one path through the tool: every session on
every card is staged, uploaded, confirmed and reclaimed the same way, so what
you have to know about a card is whether it is delivered, rather than which
sessions were held back and why. It also means a card is never held back on a
guess made offline, or by a copy of `gi-ingest` that had not been upgraded.

## Upgrading

```bash
uv tool install --upgrade gi-ingest
```

Worth doing when `doctor` says so. Older versions are refused for `upload` and
`reclaim`, because some of them lost data without reporting it — a card wiped
against another card's confirmation, and an interrupted upload that could never
be sealed.

## Getting help

`gi-ingest doctor` first: it answers most questions, and its output is the
useful thing to send on. Then your GILabs contact, with the version it printed.
