Metadata-Version: 2.5
Name: manta-trading-data
Version: 0.9.0
Summary: manta.digital news and market data management utilities
Author-email: Erik Corkran <erik.corkran@manta.digital>
Requires-Python: <3.13,>=3.12
Requires-Dist: aiofiles>=24.1.0
Requires-Dist: backoff>=2.2.1
Requires-Dist: chromadb>=0.5.0
Requires-Dist: cryptography>=44.0.0
Requires-Dist: fastapi>=0.115.0
Requires-Dist: httpx>=0.28.0
Requires-Dist: msgpack>=1.1.0
Requires-Dist: orjson>=3.10.0
Requires-Dist: packaging>=24.0
Requires-Dist: pandas>=2.3.2
Requires-Dist: psycopg-pool>=3.2.0
Requires-Dist: psycopg[binary]>=3.2.0
Requires-Dist: pydantic-settings>=2.9.1
Requires-Dist: python-dotenv>=1.0.1
Requires-Dist: pytz>=2024.1
Requires-Dist: pyyaml>=6.0
Requires-Dist: tomli-w>=1.2.0
Requires-Dist: typer>=0.15.0
Requires-Dist: uvicorn[standard]>=0.34.0
Provides-Extra: dev
Requires-Dist: mypy>=1.13.0; extra == 'dev'
Requires-Dist: psutil>=5.9.0; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.23.5; extra == 'dev'
Requires-Dist: pytest-cov>=4.1.0; extra == 'dev'
Requires-Dist: pytest-mock>=3.12.0; extra == 'dev'
Requires-Dist: pytest-timeout>=2.3.1; extra == 'dev'
Requires-Dist: pytest>=8.1.1; extra == 'dev'
Requires-Dist: ruff>=0.9.0; extra == 'dev'
Description-Content-Type: text/markdown

# manta-trading

Data acquisition, storage, and serving for equities. CLI-first, EODHD-backed,
TimescaleDB storage. PyPI distribution `manta-trading-data`, import package
`manta_trading`, CLI entry point `mt`.

---

## Installation

Requires Python 3.12+ and [uv](https://docs.astral.sh/uv/).

```sh
uv tool install manta-trading-data
```

`mt --help` should work immediately — no clone, no virtualenv activation.

The package is published on PyPI as **`manta-trading-data`**, but the Python
import package is still `manta_trading` (`import manta_trading`) and the CLI
command is still `mt` — only the install/upgrade name changed.

### Updating

```sh
mt update          # check PyPI and install a newer release (prompts first)
mt update --yes    # non-interactive: install without prompting
mt update --json   # pure query: report versions, change nothing
```

`mt update` upgrades `uv tool` installs itself; on pipx or pip installs it
prints the right command for your environment instead of running it. The
equivalent manual command is always:

```sh
uv tool install --upgrade --refresh-package manta-trading-data manta-trading-data@latest
```

(`--refresh-package` matters right after a release: uv resolves against
cached index metadata, so without it the upgrade can succeed while installing
nothing. `mt update` runs this exact command and verifies the version moved.)

In a development (editable/source) checkout `mt update` refuses and points you
at `git pull && uv sync` — it makes no network call there.

### Development setup

To work on the code itself, use a source checkout instead:

```sh
git clone https://github.com/manta-digital/trading-data
cd trading-data
uv sync
source .venv/bin/activate   # macOS / Linux
# .venv\Scripts\activate    # Windows
```

`mt --version` reports `dev` in a source checkout (no installed distribution
metadata to read); a `uv tool install` reports the real published version.

---

## Environment

Copy `.env_sample` to `.env` and fill in the values.

| Variable | Required | Description |
|---|---|---|
| `MT_TIMESCALE_DB_URL` | Yes | PostgreSQL connection URL for the TimescaleDB instance |
| `MT_EODHD_API_KEY` | Yes | EODHD API token (data acquisition + universe rebuild) |
| `MT_FINNHUB_API_KEY` | Recommended | Finnhub token (IPO-date enrichment for instruments) |
| `MT_LOG_LEVEL` | No | Log level: `DEBUG`, `INFO` (default), `WARNING`, `ERROR` |
| `MT_MINUTE_PROVIDER` | No | Minute data provider (default: `eodhd`) |
| `MT_DAILY_PROVIDER` | No | Daily data provider (default: `eodhd`) |
| `MT_EODHD_DAILY_LIMIT` | No | Daily API credit cap (default: `100000`) |
| `MT_API_MAX_BARS_PER_REQUEST` | No | Serving-API bars-per-request ceiling (default: `75000`) |
| `MT_API_STATEMENT_TIMEOUT` | No | Serving-API `statement_timeout` (default: `20s`) |

---

## Setting up a new database

The migration chain is the single source of schema truth. Bringing a fresh,
empty Postgres database to the current schema is one command:

```sh
# 1. Create the database (TimescaleDB extension must be available on the instance).
PGPASSWORD=… createdb -h <host> -U postgres trading

# 2. Point at it and initialize.
export MT_TIMESCALE_DB_URL=postgresql://postgres:…@<host>:5432/trading
mt data init
```

`mt data init` is idempotent — re-running it on a healthy database applies zero
migrations. Use `--validate-only` to inspect without changing anything.

Verify after init:

```sh
mt data migrate status   # all rows should report "applied"
mt data caggs status     # 7 caggs, all with a refresh policy
```

---

## Typical workflows

### First-time universe build

```sh
# Rebuild the instrument registry from EODHD (~33k symbols after OTC filter).
# Finnhub enrichment populates first_listing_date and promotes venue from
# transient 'US' to authoritative exchange. Takes ~9 hours at 60 req/min.
mt data instruments rebuild

# Skip Finnhub if you want registry populated quickly without IPO dates.
mt data instruments rebuild --skip-finnhub

# Populate delisted_date for delisted symbols (slice 159).
mt data instruments populate-delisted-dates
```

### Ongoing data acquisition

```sh
# Run daemon indefinitely: daily + minute cycles + once-per-day CA update.
# Defaults to full active universe. Ctrl-C or SIGTERM exits cleanly.
mt data daemon run

# Limit to minute data only; exit when universe is fully caught up.
mt data daemon run --minute --stop-when-done

# Limit to a named list; stop when done.
mt data daemon run --list priority1 --stop-when-done

# Limit to specific symbols; stop when done (--stop-when-done implied).
mt data daemon run --symbols AAPL,MSFT,SPY

# Cap credit spend.
mt data daemon run --max-credits 5000
```

### Targeted gap fill

```sh
# Fetch all UNKNOWN daily gaps for the full universe.
mt data pull 1d --universe

# Fetch minute gaps for a specific symbol.
mt data pull 1m --symbol AAPL

# Fetch minute gaps for a named list, verbose progress.
mt data pull 1m --list priority1 -v

# Preview what would be fetched without making changes.
mt data pull 1m --universe --dry-run

# Reset terminal gaps (PROVIDER_HOLE / RETRY_EXHAUSTED) then refetch.
mt data pull 1m --symbol AAPL --reset

# Include delisted symbols (requires --universe).
mt data pull 1d --universe --include-delisted
```

### Reading data

```sh
# Read adjusted daily bars for AAPL (default: adjusted=True).
mt data get AAPL 1d

# Read raw minute bars for a date range.
mt data get AAPL 1m --start 2024-01-01 --end 2024-03-31 --raw

# Output as JSON or CSV.
mt data get AAPL 1d --json
mt data get AAPL 1d --csv
```

### System health

```sh
# Show non-OK symbols (GAPS, STALE, FAILED) — default view.
mt data status

# Show all symbols including OK.
mt data status --all

# Drill into one symbol: detail panel + full gap listing.
mt data status --symbol AAPL

# Filter to daily or minute only.
mt data status --daily
mt data status --minute

# Machine-readable output.
mt data status --json
```

### Corporate actions

```sh
# Bulk-fetch yesterday's splits + dividends for the full exchange (200 credits).
mt data ca update

# Full history for a single symbol.
mt data ca update --symbol AAPL

# Full history for a named list.
mt data ca update --list priority1

# Inspect stored CA data.
mt data ca show --symbol AAPL
mt data ca list --from 2024-01-01 --to 2024-12-31
```

### Symbol lists and index universes

```sh
# List defined named lists with member counts.
mt data lists ls

# Print members of a list.
mt data lists show priority1

# Refresh the S&P 500 snapshot.
mt data lists refresh-sp500

# Show tracked index universes.
mt data universes ls

# Members of SP500 as of a date (point-in-time, survivorship-bias-free).
mt data universes as-of --name sp500 --date 2020-01-01

# Refresh index constituent tracking from source.
mt data universes refresh
```

### Continuous aggregates

```sh
# Status of all 7 caggs (last refresh, policy, row counts).
mt data caggs status

# Manually refresh all caggs (useful after a large backfill).
mt data caggs refresh

# Refresh a specific granularity.
mt data caggs refresh --granularity 1h
```

### Trading session horizon

```sh
# Extend trading_sessions for all calendars (usually automatic via daemon/status).
mt data extend

# Extend a specific calendar.
mt data extend --calendar NYSE

# Alert if horizon is < 90 days out (useful in CI).
mt data extend --strict
```

### Schema migrations

```sh
# Check migration state.
mt data migrate status

# Apply pending migrations.
mt data migrate apply
```

### Data Serving API

```sh
# Start the API server (default: 0.0.0.0:8100).
mt serve

# Custom host/port, multiple workers.
mt serve --host 127.0.0.1 --port 8200 --workers 4

# Dev mode with auto-reload.
mt serve --reload
```

API endpoints:
- `GET /api/v1/health` — liveness check, plus a coarse `coverage` freshness signal
- `GET /api/v1/bars/{symbol}?granularity=1d&start=…&end=…&adjusted=true` — OHLCV bars.
  Responses carry `is_stale`: `true` means the continuous aggregate serving this
  granularity is behind its source, so the bars may be incomplete. Raw grains
  (`1m`, `1d`) are never stale by construction.
- `GET /api/v1/symbols?search=<prefix>` — list instruments
- `GET /api/v1/symbols/{symbol}` — instrument detail + available data ranges.
  See [`available` semantics](#available-semantics) below for what the reported
  range does and does not guarantee.
- `GET /api/v1/status?symbol=…&health=…&granularity=…&all=true` — per-symbol
  data-health rows, a whole-registry health summary, and coverage freshness.
  **`rows` defaults to unhealthy entries only** (`GAPS`, `STALE`, `FAILED`),
  matching `mt data status`; pass `all=true` for everything or `health=OK` for
  healthy rows. A healthy symbol therefore returns `count: 0` by default — that
  means "nothing wrong", not "no such symbol". `summary` is always the full
  unfiltered whole-registry breakdown, whatever `rows` was filtered to.
- `GET /api/v1/gaps/{symbol}?granularity=1m` — data gap listing
- `GET /docs` — Swagger UI

The full schema is committed at [`docs/api/openapi.json`](docs/api/openapi.json)
and regenerated with `uv run python scripts/dump_openapi.py` (no database
required).

#### `available` semantics

`GET /api/v1/symbols/{symbol}` reports one `{start, end}` per granularity. The
two ends are computed differently and carry different guarantees, which matters
if you use them to decide what to request:

- **`end` is exact.** It comes from a direct probe of the bar tables, bounded so
  it stays fast, and it reflects data written right up to the moment of the
  request. If a bar exists, `end` includes it.
- **`start` is as of the last coverage materialization.** It comes from the
  coverage continuous aggregates, which a background policy refreshes. Deep
  history *backfilled after* the relevant coverage bucket was last materialized
  will not move `start` until that bucket is rebuilt — so `start` can be later
  than the true first bar, never earlier. There is no cheap exact answer here:
  probing below the coverage floor costs 0.4–1.4 s per symbol on production
  (measured), because the bound excludes chunks *after* the start, which for a
  symbol with deep history is almost none of them.

Both ends are UTC dates. A granularity with no data is omitted entirely — an
empty `available` means "no bars for this symbol", not "unknown symbol" (an
unknown symbol is a `404`).

**One documented gap.** The leading-edge probe is bounded by a universe-wide
coverage edge rather than each symbol's own. A bar could in principle be missed
if it falls between an individual symbol's coverage end and that universe edge
*and* was written after coverage last materialized. Measured across a 28-symbol
sample on production 2026-08-04 — dense, delisted, daily-only, and no-data
instruments — the merged answer was **identical to a direct `MIN/MAX` scan for
every symbol**, and no symbol had a single raw bar inside that window. The gap
closes on its own when the coverage refresh repair lands.

#### Error shapes

Every error this server raises has the same body:

```json
{ "error": "<message>" }
```

The one deliberate exception is FastAPI's own request-validation failure — an
unparseable date, an unknown `granularity` — which keeps its native body so
clients retain the per-field detail:

```json
{ "detail": [ { "loc": ["query", "granularity"], "msg": "…", "type": "…" } ] }
```

| Status | Meaning |
|---|---|
| `404` | The symbol is not in `instruments`. **Only** that. |
| `422` | The request is malformed, the range is reversed, or the window exceeds the bar ceiling. |
| `500` | An unexpected server fault. The body is sanitized. |
| `504` | The database cancelled the query at the statement timeout. Narrow the range or use a coarser granularity. |

#### Date windows are inclusive at both ends

`start` and `end` are both inclusive, at every granularity: `start=2024-06-10&end=2024-06-14`
returns Monday through Friday, and `start=2024-06-10&end=2024-06-10` returns that
whole day. Timestamps are UTC, and the store covers 08:00–23:59 UTC.

#### Empty windows are `200`, not `404`

A known symbol with no bars in the requested window returns `200` with
`count: 0` and `bars: []` — a weekend, a holiday, or a pre-listing date is not
an error. `is_stale` is still populated, so "no bars *and* the aggregate is
stale" is distinguishable from "no bars because the market was closed". A `404`
now means exactly one thing: the symbol is unknown.

#### Range cap

A bars request is admitted or rejected **before any database work**, from an
estimate computed from the window alone: `span_days × bars_per_trading_day ×
(252/365)`. Exceeding `MT_API_MAX_BARS_PER_REQUEST` (default 75,000) is a `422`
whose message names the estimate, the ceiling, and the maximum span for that
granularity. There is no pagination and no silent truncation.

Because the store covers extended hours (08:00–23:59 UTC, ~960 one-minute bars
on a dense day), the cap binds only at intraday grains:

| Granularity | Max span per request (at 75,000) |
|---|---|
| `1m` | ~113 days |
| `5m` | ~565 days |
| `15m` | ~1,697 days |
| `1h` and coarser | effectively unbounded |

For bulk history beyond these spans, query TimescaleDB directly rather than
paging over HTTP.

#### Server settings

| Variable | Default | Effect |
|---|---|---|
| `MT_API_MAX_BARS_PER_REQUEST` | `75000` | Bars-per-request ceiling used by the range cap. |
| `MT_API_STATEMENT_TIMEOUT` | `20s` | Per-connection `statement_timeout` on all three pools the API opens. A query that exceeds it becomes a `504`. |

Both are read once at startup; changing either requires a server restart. Note
they interact — raising the bar ceiling without also raising the timeout trades
a fast `422` for a slow `504`.

The API is unauthenticated and CORS-open by design: it is read-only and bound to
a LAN host. Exposing it beyond the LAN, or adding any route that writes, makes
authentication a prerequisite.

---

## Integration tests

Most integration tests under `test/integration/` require `MT_TIMESCALE_DB_URL`
set to a database that already has the schema applied. They run against that DB
and use a per-test fixture to reset state.

`test/integration/test_cold_start.py` is the exception: it creates and drops
throwaway UUID-named databases for each test, so it requires an admin connection:

```sh
export MT_TIMESCALE_TEST_URL=postgresql://postgres:…@<host>:5432/postgres
uv run --extra dev pytest test/integration/test_cold_start.py
```

CI wiring for integration tests is tracked in issue #17.

---

## Project structure

```
src/manta_trading/
  api/                   # Outbound provider HTTP clients (EODHD, Finnhub)
  api_server/            # FastAPI app (mt serve)
  cli/                   # Typer CLI commands
  config/                # Settings (pydantic-settings, MT_* env vars)
  data/
    acquisition/         # Daemon, orchestrators, gap tracking
    adjustment/          # Adjusted-on-read: compute_k_factor, adjusted()
    base/                # InstrumentRegistry, TradingCalendar
    maintenance/         # auto_extend, status_queries
    universe/            # EODHD symbol-list client, Finnhub IPO client
  market/
    schema/              # Migration definitions and runner
config/
  symbol-lists.yaml      # Named symbol lists (priority1, priority2 / sp500)
test/
  unit/                  # Unit tests (no DB required)
  integration/           # Integration tests (require MT_TIMESCALE_DB_URL)
```
