Metadata-Version: 2.5
Name: jgtpricedb-util
Version: 0.2.2
Summary: Run-to-completion jobs for the jgtpricedb price store and its chart projections
License: MIT
Keywords: forex,jgt,oanda,price-data,trading
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Financial and Insurance Industry
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Office/Business :: Financial :: Investment
Requires-Python: >=3.10
Requires-Dist: jgtpricedb>=0.1.2
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == 'dev'
Provides-Extra: indicators
Requires-Dist: jgtpy>=0.6.29; extra == 'indicators'
Provides-Extra: postgres
Requires-Dist: jgtpricedb[postgres]>=0.1.2; extra == 'postgres'
Description-Content-Type: text/markdown

# jgtpricedb-util

Run-to-completion jobs for a [`jgtpricedb`](https://pypi.org/project/jgtpricedb/)
price store.

`jgtpricedb` is **the store** — schema, bar identity, the trading-session period
grid, and the anchor-based refresh engine that writes only what moved.
`jgtpricedb-util` is **the steward** — the handful of things an operator
actually runs against that store, each one a job that starts, does a stated
amount of work, prints one greppable verdict, and stops.

Nothing here is a daemon. `light` counts its ticks and exits. `freshness` and
`probe-relabel` measure and exit. `bootstrap` and `refresh` run one pass. That is
what makes the same image safe from cron, from a compose one-shot, and from a
person's hands at six in the morning: **we run it, it stops when it is done, we
don't delete it.**

## The verdict line

Every job's last line of stdout is:

```
JGTPDB <job> OK|FAIL <summary>
```

and the exit code agrees with it:

| code | meaning |
|---|---|
| `0` | OK — the job ran and measured no failure |
| `1` | FAIL — the job ran and measured a failure |
| `2` | refused — the job could not be performed (bad arguments, absent directory, unreadable file, broker unreachable, or an abort part-way through) |

`1` and `2` are separate because "this data is wrong" and "I could not look"
call for different responses, and a caller that retries the first is wasting its
time. Grep the line, or read `$?`; they never disagree, because both come from
the same object.

The line is printed on **every** exit, including the ones nobody plans: a usage
error, an unreadable file, an unexpected exception. The traceback goes to
stderr where a person can read it; stdout still ends with the line, because a
caller that greps for it and finds nothing cannot tell a crashed job from one
that has not finished yet.

## The jobs

| job | what it does |
|---|---|
| `bootstrap` | Registers every named series from a holdings directory and runs one refresh pass. How a store first learns what exists. |
| `refresh` | One delta pass over series the store already knows. Registers nothing new. `--oanda` fetches candles into the source directory first. |
| `light` | Bounded forming-bar loop: `--loop N --interval S`, then exit. Rewrites the anchor when its values move; never appends. |
| `oanda-fetch` | OANDA v20 candles into the exact 15-column jgt CSV, at the filename `CsvSource` reads. |
| `export` | The store writes the 15-column price CSV back out, at the path every consumer already reads. Byte-for-byte the file it replaces. |
| `fold` | Folds the rolling PDS into the deep PDS by Date (rolling wins on overlap, rows only grow); no broker. The nightly step that keeps the deep store current from what the day already retrieved. |
| `derive-cds` | Materializes retained PriceDB exports into indicator-complete chart history and atomically publishes each CDS file. |
| `freshness` | Does the directory a server *serves* hold the bars the writer *wrote*? Inode identity, newest-bar drift, per-timeframe staleness. |
| `probe-relabel` | Does the period grid rename the broker's bars? Any timeframe above 0% exits 1. |
| `snapshot` | Tar the paths a run is about to touch, before it touches them. Prints the archive's sha256. |
| `verify` | Byte-compare two files; names the first offset that differs. |

### `bootstrap` and `refresh` are one engine, two authorities

They share a body because `jgtpricedb` shares one: `refresh_series` reads what
the store already holds for the periods a source offers and picks its own
algorithm — bootstrap when the range is empty, incremental update when it is not.
Forcing that choice from outside would be guessing at state the store can simply
read.

What differs is **where the list of series comes from**, and that is the whole
difference the two names carry:

- `bootstrap --all-series` means *every `<INSTRUMENT>_<TF>.csv` in the holdings
  directory*, and registers them.
- `refresh --all-series` means *every series already registered in the store*,
  and registers nothing. A CSV that appears in the source directory unannounced
  is not silently adopted mid-week.

`--now ISO` (0.1.1) is on `bootstrap`, `refresh` and `light`, the way it has
always been on `freshness`: it decides bar completeness as of a named instant
instead of the wall clock, **and moves the calendar with it**. "Is this bar still
forming?" is a question about an instant, and a run that answered it from the
clock while being handed a different one was answering a different question — a
captured holdings file then reads as complete in January and forming in August.

### `light` — bounded, and quiet when there is nothing to move

`--loop N` is a count of ticks, not a duration, and there is no `--forever`.
The job spends its count and exits, which is what makes the same image safe from
a timer, where the restart is somebody's stated policy rather than a while-loop's
accident.

Each tick finds the series' anchor and rewrites it if the values moved. It never
appends. Two things are therefore normal rather than wrong, and the verdict line
counts them apart from errors:

- **`deferred=`** — the series has no forming bar, so a full refresh owns it.
  Over a weekend every series is here. A run of nothing but deferrals still
  exits `0`, because a job that failed every Saturday would be a job nobody
  reads on Monday.
- **`rollovers=`** — the period advanced while the loop was running. The append
  belongs to `refresh`, so the tick says so instead of writing a second forming
  bar.

`errors=` is what fails the run, and `divergences=` fails it only under
`--strict`.

### `export` — the store becomes authoritative and nothing downstream is told

The Phase-3 bridge, from the operator's side. `jgtpy`, `jgtml` and
`jgt-data-server` keep opening `$JGTPY_DATA/pds/<INSTRUMENT>_<TF>.csv`; what
changes is which side of that file is the truth.

```bash
# the served window, every active series
jgtpdb export --db /work/prices.db --out /work/pds --all-series --bars 330

# one series, whole history, line endings pinned
jgtpdb export --db /work/prices.db --out /work/pds \
              --instrument EUR-USD --timeframe H1 --full --terminator lf
```

The claim is checkable and was checked: a holdings file read into a store and
written back out is **byte-for-byte itself** — 89 of 89 live series across 13
instruments and all seven timeframes, 95 208 bars, with the live forming bar as
the last row in 42 of them. The format is measured rather than assumed, and
`rispecs/03-csv-export.spec.md` in the store repository holds the measurement:

- quote columns keep the double the feed published — the holdings themselves
  carry `151.04600000000002`, and rounding it would write a different number
  than the store holds;
- the derived columns are quantized at the instrument's display precision **+1**
  (`Open`/`High`/`Low`/`Close`) and **+2** (`Median`), which is the exact decimal
  width of a mid and of a median, so the quantum removes the binary noise the
  division introduces and nothing else — and because the quantum is the exact
  width, no rounding tie ever arises to be broken one way or the other;
- the line terminator belongs to the **file being replaced**. Both CRLF and LF
  live in the holdings directory today, written by two writers, and a file's
  ending changes when the other one touches it — three files flipped LF to CRLF
  in a single day of the last measurement. `--terminator match` is the default;
  `crlf` and `lf` pin it, and a pinned choice outranks what the file carries.

Two refusals are worth knowing before an operator meets them:

- a series with **no bars** is refused rather than written. A header-only CSV is
  what a year of bars looks like after an unpopulated store wrote over it, and it
  overwrites without raising anything;
- a `--db` that does not exist is a refusal, not an empty database created on the
  spot and exported from under an OK line.

Each file is written beside itself and moved into place, so a consumer reading
mid-run gets the previous file rather than half of the next one. Cost is around
70 µs a bar — 22 ms for a 300-bar window, 169 ms for a seven-timeframe set — next
to nothing beside the recomputation the store exists to make incremental.

**One writer.** The export becomes the writer of a served directory when the
current writer stops, never beside it. Two processes writing one price file is
the failure `freshness` was built to catch.

### `freshness` — the job today's incident is made of

A writer container and a reader container each mounted a **different host
directory at the same container path**, because a stray shell variable was set
when one of them was launched. Both processes were healthy. Both logs were clean.
The writer wrote current bars all day and the reader served bars from days
earlier, and nothing inside either container could tell, because from inside,
`/data/current/pds` is `/data/current/pds`.

Three questions catch it, and no one of them is enough alone:

1. **Identity** — `st_dev`/`st_ino`. Same inode: one file, no split possible.
   Different inode: not yet a fault, a copy is legitimate — but now the other two
   questions carry the weight.
2. **Drift** — does the served file's newest bar match the expected one's? This is
   the direct signature: two files, each internally consistent, describing
   different moments. One period of slack, because the writer may legitimately
   be a bar ahead of a file the server copied a moment ago — which is also this
   check's honest limit: a split whose two sides have not yet parted by a whole
   period is invisible to it, and the alternative is failing every server that
   serves from a copy.
3. **Staleness** — is the newest bar older than this timeframe's budget? This
   catches what the first two cannot: both paths being the *same* stale
   directory, where nothing diverges because nothing is being written.

The weekend is not an outage. Forex closes Friday 17:00 New York and opens Sunday
17:00, so a Saturday probe correctly finds every file hours old. The store's own
`ForexCalendar` is consulted and an age is **minutes of open market** — the
closure is not counted, because no bar could have been written in it.

Not counting it only over the weekend is not enough, and the difference is a
whole class of false alarm: holding the clock still until Sunday 17:00 and then
releasing it puts a 48-hour cliff at the reopen, where a file holding Friday's
last bar reads one minute old all weekend and 2 881 minutes old the instant the
market opens — every series on the feed failing at once, every week, with
nothing wrong and no bar missed. Elapsed open time has no such edge, and it
equals the wall clock whenever the market never shut in between.

Defaults are roughly three periods plus slack per timeframe, overridable:

```bash
jgtpdb freshness --served /data/current/pds --expected /writer/pds \
                 --max-age-minutes H1=120,m15=40
```

### `probe-relabel` — the measurement that produced `jgtpricedb` 0.1.1

A store cut on naive UTC renamed **100%** of the H4, D1, W1 and M1 bars it was
given, because the feed cuts those periods on the 17:00 America/New_York session
boundary. It renamed them silently: the bar kept its prices, `period_start` moved
its timestamp, the key came from the moved timestamp, and no exception was raised
anywhere.

The probe is one line of arithmetic run over real files, using the library's own
grid:

```
period_start(timeframe, ts, session_for(instrument)) == ts
```

Run it against holdings rather than a fixture. A fixture encodes what the grid
believed on the day it was written; the CSVs encode what the broker actually
published, which is the only authority there is.

### `oanda-fetch` — session alignment is not optional

OANDA aligns `D`, `W`, `M` and `H4` candles to `dailyAlignment` in
`alignmentTimezone`, and the default is 00:00 UTC — *not* the feed's boundary.
Left at the default, every session-anchored bar would land an offset away from
the holdings and the store would key it as a different bar. This job sends
`dailyAlignment=17`, `alignmentTimezone=America/New_York` and
`weeklyAlignment=Saturday` for exactly those timeframes, all three read off
`jgtpricedb`'s own `NY_TRADING_SESSION` rather than restated here.

**`weeklyAlignment` is `Saturday`, and the reason is worth a paragraph.** The
trading week is *labelled* Sunday — that is `session.week_opens_on` — but the
boundary that opens it is the evening before, Saturday 17:00 New York. OANDA's
parameter names the boundary's weekday, not the label's. Asking for `Sunday`
returns a weekly candle stamped one day late: measured against the live
practice API, every W1 bar of both a summer and a winter window landed off
`period_start`'s grid. The store recovers, because identity floors a stamp
before it keys anything — which is exactly what made it silent. The CSV on disk
disagreed with the holdings for the same week, and `probe-relabel` over that
directory reported 100% on W1, a defect belonging to the fetch rather than to
the grid. `tests/test_session_grid.py` keeps both responses, correct and wrong,
as captured evidence.

`Median` has no OANDA field, so it is **derived** — from bid and ask, the
authoritative sides, by the store's own formula. Not from OANDA's mid candle:
that candle is built from mid ticks rather than averaged from its own bid and
ask, so its `High` and `Low` genuinely disagree with `(bid + ask) / 2`. Those
cells are written through as what the broker published and `CsvSource` reports
them as validate divergences, which is the design — the store keeps its own
derivation and says where the source differed. A Median computed off that mid
would inherit the same disagreement and report it against a column that was
never the problem; over 149 captured live candles that was six false reports.

An `--oanda` refresh therefore prints a handful of `divergences=` on the mid
columns as a matter of course. That is a finding about the broker's mid stream,
not a failed run — and `--strict`, which turns findings fatal, will fail such a
run on purpose.

## Quickstart — pip

```bash
pip install jgtpricedb-util

# fill a store from the existing holdings
jgtpdb bootstrap --db /tmp/prices.db --holdings $JGTPY_DATA/pds \
                 --instrument EUR-USD --timeframe H1

# ask the broker for today, then take the delta
export OANDA_TOKEN=...            # practice by default
jgtpdb oanda-fetch --instrument EUR-USD --timeframe H1 \
                   --since 2026-07-29 --out /tmp/oanda
jgtpdb refresh --db /tmp/prices.db --holdings /tmp/oanda \
               --instrument EUR-USD --timeframe H1

# watch the forming bar move, ten times, then stop
jgtpdb light --db /tmp/prices.db --holdings /tmp/oanda \
             --instrument EUR-USD --timeframe H1 --loop 10 --interval 60 --oanda

# write the store back out as the CSV every consumer already reads
jgtpdb export --db /tmp/prices.db --out /tmp/pds --all-series --bars 330
jgtpdb verify --a /tmp/pds/EUR-USD_H1.csv --b $JGTPY_DATA/pds/EUR-USD_H1.csv
```

## Quickstart — container

The image is a job container: `restart: "no"`, one job per invocation.

```bash
cd jgt-pricedb-util

# build from PyPI (the normal path)
docker compose build

# run a job — the entrypoint is jgtpdb, so the command is just the job
docker compose run --rm jgt-pricedb-util probe-relabel --holdings /data/current/pds
docker compose run --rm jgt-pricedb-util bootstrap --db /work/prices.db \
    --holdings /data/current/pds --instrument EUR-USD --timeframe H1
docker compose run --rm jgt-pricedb-util freshness \
    --served /data/current/pds --expected /data/current/pds
```

To run **this checkout** instead of the published release — before a release
exists, or to prove a change — export both variables before building, and leave
them exported for the run:

```bash
export JGT_PRICEDB_UTIL_LOCAL=1 JGT_PRICEDB_UTIL_TAG=local
docker compose build
docker compose run --rm jgt-pricedb-util probe-relabel --holdings /data/current/pds
```

Both, not one. `LOCAL` decides what gets installed and `TAG` decides what the
image is called; setting only the first builds this checkout under the released
version's name, and setting only the second is a rename of the PyPI build.
Exporting them once covers the build and every run after it — forget them on
the `run` and compose quietly reaches for the released image instead.

The plain-docker equivalent, if you would rather not export anything:

```bash
docker build --build-arg LOCAL=1 -t jgtpricedb-util:local .
docker run --rm -v "$JGTPY_DATA:/data/current:ro" -v "$PWD/work:/work" \
    jgtpricedb-util:local probe-relabel --holdings /data/current/pds
```

Mounts, as composed:

| container path | host | mode |
|---|---|---|
| `/data/current` | `${JGTPY_DATA:-/b/trading/jgtml/data/current}` | read-only |
| `/work` | `./work` | read-write — the store, fetched candles, snapshots |

`/data/current` is mounted **read-only** deliberately. This package reads the
holdings and writes to `/work`; a job that could rewrite the pipeline's own
output is a job that can cause the incident `freshness` exists to detect.

`OANDA_TOKEN`, `OANDA_ACCOUNT_ID` and `OANDA_ENVIRONMENT` pass through from the
host environment. The environment defaults to `practice` when unset, in this tool
as in every other jgt tool.

### `derive-cds` — one provider boundary for historical chart data

The PriceDB export is the retained source. `derive-cds` applies the canonical
Williams engine to the whole retained window, verifies that dense Alligator,
AO and AC columns exist on every output row, then atomically replaces CDS.
An invalid PDS, failed calculation, empty result or indicator gap leaves the
previous live CDS untouched. The chart server only reads the resulting file.

```bash
jgtpdb derive-cds --pds /data/current/pds --out /data/current/cds \
  --instrument EUR-USD,GBP-USD --timeframe m15,H1 --bars 10000
```

`--bars N` publishes the last N complete rows and reads only N plus the 610-bar
indicator warmup, so a 1.6M-row m5 archive is not loaded to refresh its tail.
Without it the whole export is projected. The jgt-data-server updater builds its
rolling CDS through this same `materialize_cds` seam (10 000 bars by default —
the price service holds thousands of bars per timeframe, not the broker's 500),
and `oanda-fetch --count` above 5000 pages backward through the v20 API to reach
that depth.

## Dependencies

Ordinary jobs remain `jgtpricedb` plus the standard library. No `requests` —
`urllib`. The optional `[indicators]` extra adds the canonical Williams engine
for `derive-cds`; that dependency is confined to the provider job and is never
loaded by bootstrap, refresh, export, freshness or chart-serving code.

`jgtpricedb` itself brings `sqlalchemy` and `pandas`, so the image already
contains the tabular runtime used to validate and atomically publish CDS.

## Related

- [`jgtpricedb`](https://pypi.org/project/jgtpricedb/) — the store this stewards.
- The session-grid decision, its evidence and its measurement live in
  `jgtpricedb`'s `rispecs/01-price-store.spec.md`, *The Session Grid*.
