Metadata-Version: 2.3
Name: rideralerts-scraper
Version: 0.1.0a2
Summary: Poll vehicle positions from rideralerts.com InfoPoint (Avail AVL) transit agencies into SQLite
Author: Alex Garcia
Requires-Dist: click>=8.1
Requires-Dist: httpxyz>=0.31.2
Requires-Dist: sqlite-tg>=0.0.1a21
Requires-Python: >=3.13
Description-Content-Type: text/markdown

# rideralerts-scraper

CLI for continuously polling vehicle positions from transit agencies
hosted on `rideralerts.com` InfoPoint (Avail AVL) REST APIs into a
SQLite database. Known agencies: Montebello Bus Lines
(`mbl.rideralerts.com`) and Norwalk Transit (`nts.rideralerts.com`);
any other InfoPoint tenant works by subdomain or base URL.

```sh
uv run rideralerts-scraper run -a montebello -o vehicle_positions.sqlite   # runs until ^C
uv run rideralerts-scraper run -a mbl -a norwalk \
    -o vehicle_positions.sqlite     # poll both agencies into one db
uv run rideralerts-scraper run -a norwalk -o vehicle_positions.sqlite \
    --within query.geojson          # only store positions inside the boundary
uv run rideralerts-scraper run -a mbl -o vehicle_positions.sqlite \
    --routes 10,40                  # only poll lines 10 + 40
uv run rideralerts-scraper stats -o vehicle_positions.sqlite
uv run rideralerts-scraper vehicles -o vehicle_positions.sqlite \
    -a norwalk                      # only list Norwalk's vehicles
```

`--agency` accepts a known name or alias (`montebello`/`mbl`,
`norwalk`/`nts`), a bare `rideralerts.com` subdomain (`sta`), a
hostname (`sta.rideralerts.com`), or a full InfoPoint base URL
(`https://sta.rideralerts.com/InfoPoint`). The stored `agency` slug is
the subdomain (first hostname label), so `montebello` and `mbl` are
the same agency.

Each poll hits `GetAllVehiclesForRoutes` and returns the full current
status of every bus on the selected routes; responses are decomposed and
flushed to SQLite once per poll. `schema.sql` splits the payload into a
history table (`vehicle_status`, deduped on
`(agency, vehicle_id, observed_at)`) with a trigger maintaining the
latest-state table `vehicle`; each agency's route directory from
`GetVisibleRoutes` lands in `route` at the start of each run. One
database can hold any number of agencies — every table is keyed by the
`agency` slug, since vehicle and route ids are only meaningful within
one agency. The raw payload JSON is not stored — every field the API
sends has its own column, with the .NET-style `/Date(<epoch_ms><±hhmm>)/`
timestamps normalized to unix seconds. The insert statements live in
`procedures.sql`. The typed wrapper `_queries.py` is codegen'd from
both SQL files — after editing either, regenerate with `make` (runs
`solite codegen` piped through `tools/codegen.py`; solite is pinned to
0.0.1a36 because later prereleases stopped embedding the schema in the
generated setup).

## CLI Reference

### `run` — poll vehicle statuses into SQLite until interrupted

Fetches each agency's route directory once at startup, then polls the
vehicles endpoint on an interval and inserts status snapshots.
Databases are created in WAL mode. Filters (`--routes`, `--within`)
compose with AND; filtered statuses are dropped before they reach the
database.

| Flag | Default | Description |
|------|---------|-------------|
| `-a`, `--agency AGENCY` | required | Agency to poll: known name/alias (`montebello`/`mbl`, `norwalk`/`nts`), bare subdomain, hostname, or InfoPoint base URL. Repeat to poll several agencies into the same database. |
| `-o`, `--out FILE.sqlite` | `vehicle_positions.sqlite` | Database to poll into; created (in WAL mode) if missing. |
| `--routes IDS` | all published routes | Comma-separated route ids to poll, e.g. `10,40,90`. MBL and NTS route ids are the rider-facing line numbers themselves. Unselected routes are never requested. Route ids are per-agency, so this needs a single `--agency`. |
| `--within FILE.geojson` | no filter | Only store positions inside this geometry (GeoJSON Feature or bare geometry; WKT/WKB also work), tested per-status with [sqlite-tg](https://github.com/asg017/sqlite-tg). Statuses with no position are dropped. |
| `--every SECONDS` | `10` | Poll interval (each agency is polled once per interval). |
| `--bucket SECONDS` | off | Log one aggregated line per interval (with running db totals) instead of one line per poll. Warns if an interval passes with no statuses. |
| `--verbose`, `-v` | off | DEBUG logging; with `--bucket`, also shows the per-poll lines. |

Log lines are prefixed with the agency slug and count every drop
reason: `duplicate` (the API re-serves each vehicle's unchanged status
every poll; deduped by primary key), `outside boundary` (`--within`),
`invalid` (unparseable or failed poll responses).

### `vehicles` — list the latest known position of each vehicle

One line per vehicle from the `vehicle` table (agency, route, fleet
name, lat/lon, speed, schedule deviation, status, age of last
observation), ordered by agency then route. Opens the database
read-only, so it's safe against a live `run`.

| Flag | Default | Description |
|------|---------|-------------|
| `-o`, `--out FILE.sqlite` | `vehicle_positions.sqlite` | Database to inspect. |
| `-a`, `--agency AGENCY` | all agencies | Only list this agency's vehicles (same formats as `run --agency`). |
| `--within FILE.geojson` | no filter | Only list vehicles currently inside this geometry (same formats as `run --within`). |

### `stats` — print row counts and time range

One-shot summary: per-agency counts of statuses, vehicles, and routes,
plus the `received_at` time span of the stored history. Read-only.

| Flag | Default | Description |
|------|---------|-------------|
| `-o`, `--out FILE.sqlite` | `vehicle_positions.sqlite` | Database to inspect. |
