Metadata-Version: 2.1
Name: pygar-client
Version: 4.19.0
Summary: A Python client for the GAR protocol
Home-page: https://www.trinityriversystems.com/docs/gar_protocol/
Author: Jon Hill
Description-Content-Type: text/markdown
Requires-Dist: msgspec >=0.18
Requires-Dist: websockets >=13

# Generic Active Records Protocol Documentation

## Overview

The GAR protocol is a WebSocket-based messaging system built on top of TCP. It is used for streaming and snapshot-style data delivery and operates over a single bi-directional socket connection. When establishing a connection, clients must specify the subprotocol `"gar-protocol"`:

```
Sec-WebSocket-Protocol: gar-protocol
```

## Encoding

The server supports two formats for message payloads: **binary** and **JSON**.

- **JSON (opcode 0x01)** is recommended for human-readable debugging and platform-agnostic integration.
- **Binary (opcode 0x02)** is compact and efficient but platform-dependent due to endianness, sizing, and alignment.

If the first byte of the initial Introduction message is `{`, then JSON mode is assumed regardless of the WebSocket opcode.

## Binary Format

- Each message begins with a `message_type` enumeration value.
- Immediately followed by a binary-encoded struct corresponding to that type.
- Must conform to platform-specific alignment and byte order.
- Should only be used when both ends are aware of the exact schema and platform.

## JSON Format

JSON messages are structured with a `"message_type"` string and a corresponding `"value"` object.

Example:
```json
{
  "message_type": "Subscribe",
  "value": {
    "subscription_mode": "Snapshot",
    "nagle_interval": 0,
    "name": "S1",
    "key_id": 0,
    "topic_id": 0,
    "class_list": ["Underlier"],
    "key_filter": null,
    "topic_filter": null
  }
}
```

This format is more portable and ideal for most clients.

## Framing

Each websocket frame contains a single message - a json {} or binary object.

## Session Lifecycle

1. **Introduction** must always be the first message. It specifies protocol version, heartbeat expectations, and optional settings such as `unique_key_and_record_updates`.
2. Clients may then **Subscribe** to topics and keys or **Publish** messages.
3. Clients must send **Heartbeat** messages periodically to keep the connection alive. If none is received within the specified timeout, the server will disconnect the client.
4. The **first heartbeat** has a 10x grace period to allow for slow client setup.
5. All topics and keys must be explicitly introduced before publishing record messages.
6. IDs `0` are considered **invalid** for both `topic_id` and `key_id`.
7. Enumeration of topics and keys is not globally synchronized between client and server. Each side must track their own ID mappings.

### Introduction Options

The introduction message supports the following optional fields:

- **`unique_key_and_record_updates`** (bool, default false): If true, only one record update will be sent per key/record even if multiple subscriptions apply. This only affects streaming record updates — snapshot data is still sent for each subscription. Note that `active_subscription_group` will not be reliable when using this option.
- **`disconnect_on_write_failure`** (bool, default false): bail-out-first writers. The server closes the connection on the first client message with a failed write — after reporting it, as a **non-recoverable** `Error` carrying `failed_writes` (delivered even when the `active_ownership` `msg_ref` is 0, which is otherwise log-only). Every write that succeeded stays landed; this is not transactional — a client that needs atomicity batches its writes so each batch is safe under partial failure. Meant for critical processes that would rather restart than continue past a refused write, and for integration tests; subscriptions on the same connection go down with it, so a process that both watches and writes should keep them on separate connections. Through a proxy, a failure detected upstream reaches the client as a relayed `failed_writes` Error and closes the client's connection the same way, but only when the proxy learned of it — i.e. with a nonzero `msg_ref`, since the proxy's own upstream connection does not carry the flag. pygar: `GARClient(..., disconnect_on_write_failure=True)`; jsgar: the `disconnectOnWriteFailure` constructor argument. Both libraries already treat a non-recoverable `Error` as fatal (`exit_code` 1, stop). The REST analogue is `?fail=true` (see [Write-failure replies](gar_rest_api.md#write-failure-replies)).

## Record Updates

Record updates flow both ways through the same messages: a client writes with them, and a server streams
**post-snapshot updates to a subscriber** with them too — only the initial snapshot (and nagle-coalesced
ticks) travel as [batch updates](#batch-updates). Anything that must ride with every streamed record
therefore has to live on these messages as well as on `batch_update`.

The GAR protocol supports record updates via:

### Binary Format
- **FixedLengthRecordUpdate**: `record_id` directly followed by the fixed-size record data.
- **VariableLengthRecordUpdate**: serialized `binary_record_update` struct with embedded length field.

### JSON Format
- Encoded using `json_record_update` with clear field names and value.

Example:
```json
{
  "message_type": "JSONRecordUpdate",
  "value": {
    "record_id": {"key_id": 1, "topic_id": 22},
    "value": 0.3
  }
}
```

### Augment bypass (assume_cached_subscriptions_cover)

A subscription with `assume_cached_subscriptions_cover: true` asserts that the **proxy** serving it has
cached subscriptions covering every record it asks for. That proxy then skips this subscription's augment
differencing entirely and serves it — derived topics included — from its cache, removing the poller-thread
set-algebra the snapshot blocks on. The assertion is per subscription: an operator turns it on for the few
heavy subscriptions the cache was provisioned around, and every other subscription on the same proxy keeps
augmenting normally. No proxy config gates it — the assertion is the caller's own, and a wrong one
under-serves only the subscription that made it; a plain (non-proxy) server has no cache to bypass and
ignores the flag. **Fail-open:** anything the cache does not cover is silently absent or null on that
subscription. REST equivalent `?assume-cached=true`, `trsgar --assume-cached`. See
[augment_bypass_spec](../../features/proxy/augment_bypass_spec.md).

### Record tick_time

A subscription with `include_tick_time: true` receives each record's **tick_time**: the owning server's
wall-clock nanoseconds at the record's last write. It advances on **every** write, including one whose
value did not change, so it answers "is the writer alive", not "did the value change". A value-unchanged
write is throttled (no update); a flagged subscription instead receives a **`RecordTick`**
(`record_tick {record_id, tick_time}`) — stamp only, no value — which clients that don't track stamps can
ignore. Proxies forward the flag upstream and relay the origin's stamp unchanged, never their own receipt
time; a relayed tick never triggers recomputation.

- `json_record_update` / `binary_record_update` carry it as `tick_time` (JSON omits the member, binary
  sends 0, when the subscription didn't opt in).
- `batch_update.key_major_update.tick_times` is an array parallel to `topics`; JSON carries a
  `"tick_times"` object beside `"topics"` with the same `<topic_id>` member names. Empty/absent otherwise.

Off by default — payloads keep their size and shape. See [features/server_tick_time](../../features/server_tick_time/server_tick_time.md).

### Record diagnostics

A derived-topic record that has no value carries a structured **diagnostic** saying why — a
`derived_diagnostic {reason, missing_inputs[], text}`. `reason` is one of `InputMissing`,
`KeyCastMissed`, `RequiredArgSkip`, `Suppressed` (the *null class*: plain absence with a cause) or
`ComputeError` / `InputError` (the *error class*: the function raised or failed — `text` is the
stripped exception message — or one of its inputs is in error). Errors are poison: an `InputError`
consumer stays in error whatever its optional/default argument policy, and `??` coalesces nulls
only. `missing_inputs` names each absent or erroring input — its argument index and declared
parameter name, the input topic's path and the key holding it — so a client can chase a null or an
error upstream, record by record, to its root cause.

- Every subscription receives error-class diagnostics with no opt-in; the null-class reasons only
  reach subscriptions that set `include_diagnostics: true`.
- The wire carries them as **`RecordDiagnostic`** (`record_diagnostic {record_id, diagnostic}`),
  sent after the batches that introduce the record's key during a snapshot, and streamed on change.
  A diagnostic replaces the record's value: a value→diagnostic transition arrives as the ordinary
  `DeleteRecord` followed by the `RecordDiagnostic`; a value arriving supersedes the diagnostic.
- A proxy computes derived topics itself, so its diagnostics are its own computation's; a client
  that takes derived records from a server (trsgar, pygar, jsgar) applies the relayed diagnostic
  verbatim.

See [null_reasons_spec](../../features/derived_topics/null_reasons_spec.md).

## Density

The `density` field controls how records are grouped in JSON output. The default density is **KeyMajor**.

- **KeyMajor** (default): `{ "key": "key1", "topics": { "topic1": <record>, "topic2": ... } }`
- **TopicMajor**: `{ "topic": "topic1", "keys": { "key1": <record>, "key2": ... } }`
- **RecordMajor**: `{ "key": "key1", "topic": "topic1", "value": <record> }` (one record per object)

## Batch updates

Multiple updates can be sent together in a batch. Initial snapshots after subscribing are always sent in one or more batch.

Records are sent in key-major format. Keys are introduced along with their class or class_list. Key names and classes may be omitted when keys have already been introduced.

Binary format uses the `batch_update` struct.

JSON format is similar, except the records within each "topics" object are JSON objects with topic_id numbers as member names: { "<topic_id>": record_value, ... }

Example:
```
{
  "message_type": "BatchUpdate",
  "value": {
    "default_class": "A",
    "keys": [
      {
        "key_id": 1,
        "name": "key1",
        "topics": {
          "20": 10
        }
      },
      {
        "key_id": 2,
        "name": "key2",
        "class": "B",
        "topics": {
          "21": 20
        }
      },
      {
        "key_id": 3,
        "name": "key3",
        "classes": ["A", "B", "C"],
        "topics": {
          "20": 30,
          "21": 31,
          "22": 32
        }
      }
    ]
  }
}
```

## Subscriptions

Clients may initiate:

- **Snapshot** subscriptions (one-time data snapshot),

- **Streaming** subscriptions (continuous updates),

- **UpdatesOnly** subscriptions (continuous updates without an initial snapshot),

- **DeleteKeys** (delete all matching keys and their records)

- **DeleteRecords** (delete all matching records)

**DeleteKeys** and **DeleteRecords** are *one-shot*: the server deletes the set matching the
subscription's filters at subscribe time (resolving key cross-references — e.g. a referenced parent key
is removed even while a child still points at it) and does **not** act on keys/records created later, so
a delete subscription left active will not re-delete a future write under the same filter. The deletion
is processed **incrementally as a snapshot** — scheduled async on the server's poller, *not* applied
synchronously when the subscribe is received — and is complete only once the server sends
`"status": "Finished"` (see [Subscription Status](#subscription-status)). **Do not send `Unsubscribe`
(or close the connection) before that Finished status arrives:** doing so cancels the in-flight snapshot,
leaving the delete partial or unstarted. The REST API runs these deletes internally and blocks the HTTP
response until the snapshot finishes, so REST `DELETE` callers get the completed guarantee for free — a
direct WebSocket client must wait for `Finished` itself.

Subscription filters can be applied using:

- Direct `key_id` or `topic_id`,

- `_class` filters to only keys of this class,

- Regular expressions via `key_filter` or `topic_filter`,

- `referencing_class_list` to include back-referenced keys and records (see below),

- `join_class_list` + `join_mode` to join a second class list against `class_list`'s key set:
  `Intersect` accepts only keys carrying a class from **both** lists ("keys of X that are *also*
  Y"); `LeftJoin` leaves the key set unchanged while the join classes contribute their topics on
  accepted keys that carry them. In both modes the join classes' topic closures become servable. A
  non-empty `join_class_list` without a `join_mode` is a subscribe error. Membership follows class
  changes on streaming subscriptions: a key gaining its join class is introduced when it enters the
  scope, and a per-class delete of the join class is relayed as usual — inferring exclusion from it
  is the client's job.

Heavy topics (inferred for comprehensions, array-typed values and class/comprehension-fed derivations, or
stated with `{heavy_override: true}`) and explicit-only topics (`{explicit: true}`, or `{once: true}`-gated) are left out of every
subscription unless `include_heavy` / `include_explicit` is set. Sweeps (class, regex, subscribe-all) simply
omit them; a `topic_id_list` that **names** one without the flag is a subscribe error — the server replies
with a non-recoverable `Error` stating the topic, why it is excluded, and the flag to set, exactly as for an
unknown topic id — rather than coming up `Streaming` minus a topic the client would wait on forever.

Subscriptions may be changed by sending another subscription messages using the same name. After the ProcessingSnapshot subscription status message is returned, all subsequent updates will be for the new subscription settings.

To cancel an active stream, send an `Unsubscribe` message with the subscription name. Alternatively, unsubscribe by sending another subscription with "subscription_mode": "Unsubscribed".

If a topic filter regex is provided, it is checked against all relative topic paths. So for example, if the working namespace is "root_ns::sub_ns::w_ns", the topic "root_ns::sub_ns::tn" will
be matched by any of the topic filters "tn", "sub_ns::tn", or "root_ns::sub_ns::tn". If the working namespace was "root_ns::other_ns", only "sub_ns::tn" and "root_ns::sub_ns::tn" would match.

The working namespace defaults to the working namespace sent in the introduction message. It may be overridden in the subscription request using the working_namespace field.

Clients may assign an arbitrary numeric subscription_group id for the purpose of isolating callbacks.
An active_subscription_group message is sent before record or key updates to denote the subscription_group.
Many updates may follow a single active_subscription_group message, and the initial active subscription_group is assumed to be 0.
subscription_group ids do not need to be unique across subscriptions.

Large subscriptions may overwhelm the client, server, or network. snapshot_size_limit may be set to throttle the initial snapshot response.
Each time the limit is hit, the server sends back a SubscriptionStatus message with status = "NeedsContinue" and stops sending additional snapshot data.
The snapshot processing will continue once the client sends a SubscribeContinue message, until another limit is hit or the snapshot finishes with status = Streaming or status = Finished.

Clients do not need to continue with the snapshot if snapshot_size_limit is hit. They may choose to continue, pause for some time or indefinitely, or unsubscribe.

The same protocol paces bulk *publications*: a publisher using snapshot_size_limit pauses at each NeedsContinue until the receiver replies SubscribeContinue. A plain server replies once it has applied the batch; a proxy additionally defers the reply while any publishing server's write backlog is above a pacing threshold, so a bulk publication cannot overflow a slow server's write queue. The pacing knobs are `g::deployment::ProxyPerformanceTuning` records (a per-proxy override on the proxy's own key, else the `global` key's defaults); if a backlog never drains, the threshold doubles on a timeout until pacing yields, so the publisher is throttled but never wedged.

### Referencing Class List

The `referencing_class_list` subscription field enables back-reference expansion. When populated:

- Keys and records are scanned and included as usual based on other filter criteria
- Records of included topics that directly reference any included keys through classes in the `referencing_class_list` are also included
- Keys of referencing records are included, and recursively along with any of their referencing keys and records
- Once included, referencing keys and records are NOT removed from the subscription if references are later dropped
- Referencing topics must be included in the subscription for this to take effect

This is useful for following key references across classes, such as including all subscriptions that reference a given server.

## Echo

Clients may send an **EchoRequest** message containing a `msg_ref` (message reference number). The server will respond with an **EchoResponse** containing the same `msg_ref`. This can be used to confirm that all prior writes have been processed before continuing.

In proxy chains, the echo propagates end-to-end: the proxy forwards the echo to its upstream server and only responds to the client once the upstream echo response arrives. This guarantees that all prior writes — including record updates and deletes — have been fully processed through the entire chain by the time the client receives the echo response. The REST API uses this mechanism internally, so HTTP POST and DELETE responses also carry this guarantee.

## Compare-exchange

A **CompareExchange** message applies a write only if the record's current value is what the client
expects, so two writers racing on one record cannot silently overwrite each other. It carries a
`record_id`, an inline `msg_ref` correlation id, a `test` (the expected current value) and a `value`
(what to write on a match). The server replies with a **CompareExchangeResult** echoing that
`msg_ref` and carrying `success`; on a mismatch it also returns the record's **current** value, so a
client can retry against fresh state without a separate read.

**Null means absent, on both sides.** A missing record reads as null, so:

- `test: null` matches a record that is **missing** — this is create-if-absent. (For a topic whose
  type declares a default, a missing record instead reads as that default, so the matching test is
  the default value.)
- `value: null` is an **empty** value, so a successful compare **deletes** the record. This holds for
  every topic type, including fixed-length numeric ones — a null value is never stored as a zero.

Binary clients express the same two things with a zero-length `test` / `value` byte array.

Together those give compare-and-delete: `test` = the value you saw, `value` = null. That makes a
record a **lock-free "consume exactly once" flag** — several readers that all observed *X* each
attempt the delete against *X*, exactly one succeeds, and the losers are told so. Nothing is claimed,
so a reader that dies mid-operation strands nothing. The same operation is available over
[REST](gar_rest_api.md#url-format) via `atomic-compare-exchange=true`.

Compare-exchange propagates through proxy chains: the proxy forwards it upstream and relays the
result back to the originating client. The REST equivalent forwards the same way, so both surfaces
are atomic through a proxy.

## Subscription Status

A SubscriptionStatus message will be sent to delineate the lifetime of the subscription.

- "status": "ProcessingSnapshot" is sent at the start of processing. Topic, Key, and record updates will follow.

- "status": "NeedsContinue" is sent if snapshot_size_limit is hit; see Subscriptions above.

- "status": "Streaming" is sent for streaming subscriptions once the snapshot has been fully processed. Additional messages are live updates.

- "status": "Finished" is sent once the snapshot has been fully processed and the subscription mode is non-streaming. It is also sent after an unsubscribe has been processed.

## Heartbeats

Clients specify a heartbeat timeout interval within their introduction. If the server does not receive a Heartbeat message within this interval, it sends the client an Error message ("Heartbeat timeout") and closes the connection gracefully — a client that still reads (e.g. one whose sending stalled) receives the error before the close; one that doesn't is dropped after one further timeout interval.

The server also specifies its timeout interval, and sends regular heartbeats. Clients may choose to respect missed heartbeats and disconnect.

Heartbeats are expected to be sent at twice the frequency of the timeout interval. e.g. with timeout of 4 seconds, heartbeats are sent every 2 seconds.

The send cadence is fixed-rate (anchored to the previous scheduled instant, not to when the handler last ran), but a missed-tick backlog never replays. The send loop is single-threaded, so a slow client that backs up the send buffer can stall it for many intervals; rather than firing a burst of catch-up heartbeats onto the already-slow client when it resumes, the sender snaps the next beat forward past the current time, collapsing the backlog to a single heartbeat one interval out.

## Synchronizing without sleep

Client code (and test harnesses) frequently needs to wait until some condition holds — the server is
accepting connections, prior writes have been applied, a subscription is live, a derived value has
settled — before taking the next step. **Do not approximate these waits with a fixed `sleep`.** A
blind delay is simultaneously too short (it races under load) and too long (it wastes time on a fast
path), and it converts a missed event into a silent, intermittent failure. The protocol exposes an
explicit signal for each case; synchronize on the signal and the timing becomes deterministic.

- **Server is ready to accept connections.** Launch the server with `--server-ready-out FILE`; it
  writes the bound `{protocol, url}` endpoints to `FILE` once it is listening. Poll until the file is
  non-empty, then read the URL — never sleep waiting for a port to bind.

- **All prior writes have been processed.** Send an **EchoRequest** and wait for the matching
  **EchoResponse** (see [Echo](#echo)). In a proxy chain the echo is end-to-end, so its response
  guarantees every prior record update/delete has propagated through the whole chain. The REST API
  uses this internally, so an HTTP `POST`/`DELETE` response already carries the guarantee — no
  follow-up wait is needed.

- **A streaming subscription is live.** After subscribing, wait for the
  `"status": "Streaming"` **SubscriptionStatus** message (see [Subscription Status](#subscription-status));
  it is sent once the initial snapshot has been fully processed and means subsequent messages are
  live updates. This matters most for **live-only topics** (`history: none`): such a record is pushed
  only to subscribers that are *already* streaming when it is produced and is never retained, so a
  subscriber still in `ProcessingSnapshot` when the event fires misses it permanently. Trigger the
  event only after the `Streaming` status arrives.

- **A derived or once-gated value has settled before you act on it.** Poll the value (over REST, or
  via your subscription stream) until it holds the expected value, rather than sleeping a guessed
  interval. This is essential before firing an event that *samples* a value: a derived topic with a
  `{once: true}` input that has never been valued is **suppressed entirely** (the whole record is
  withheld), so an event that triggers it before the once-input is populated produces nothing.

- **A non-streaming snapshot subscription has completed** — most importantly a **DeleteKeys** /
  **DeleteRecords** delete. Wait for the `"status": "Finished"` SubscriptionStatus before treating the
  result as done or sending `Unsubscribe`: the snapshot (and any delete it performs) is processed
  incrementally, so an `Unsubscribe` issued synchronously after the subscribe cancels the in-flight work.
  The REST API waits for `Finished` for you — an HTTP `DELETE` returns only after the delete has fully
  processed — so this caveat applies to direct WebSocket clients (e.g. browser/Node `subscribe` callers),
  not REST callers.

For ad-hoc polling, use a bounded retry with a small interval and a hard timeout (e.g. check every
100 ms up to a few seconds) so a stuck condition surfaces as a clear timeout rather than a hang. The
only place a fixed delay is defensible is a *negative* check — confirming that something does **not**
happen within a window.

## Termination

To gracefully close a session, clients should send a `Logoff` message. However, clean TCP connection closure is also acceptable.

If the server encounters an issue, it may respond with an `Error` message prior to disconnection.

## Failed writes

Recoverable write failures do not disconnect. Every failed write from processing one client message
batches into ONE recoverable `Error` message whose `failed_writes` array identifies each failure —
a shape-scoped reason enum (`record_write`/`key_write` union), the addressed ids and names,
`malformed` (server-classified: a request defect vs a server-state condition), per-failure `detail`,
and `origin` (the server or proxy that detected the failure; proxies relay upstream failures
verbatim). Delivery requires a nonzero `msg_ref` on the `active_ownership` marker — with `msg_ref`
0 the failure is logged server-side only. pygar (`register_write_errors_handler`) and jsgar
(`registerWriteErrorsHandler`) deliver the array to a callback as
`(failed_writes, message, msg_ref)`. A connection introduced with `disconnect_on_write_failure`
instead receives the batch as a non-recoverable `Error` (msg_ref 0 included) and is then closed —
see [Introduction Options](#introduction-options). Pinned end-to-end by `ipc/write_errors_test.bash`.

## Key teardown is per class

All key teardown is delivered as `KeyUpdate` messages carrying `deleted_class` — one message per
class. When a server deletes a key entirely, it sends one `deleted_class` KeyUpdate per class the
receiver was shown (each with an empty remaining `class_list`), followed by
`KeyIdRetired` — a connection-scoped bookkeeping message meaning only "the sender's
introduction of this key id is gone; drop your mapping for it", never a data teardown. There is
no whole-key teardown broadcast: a subscriber's view of a key ends when its last subscribed class
is deleted. (Distributed servers can each hold a *subset* of a key's classes, so "all classes
gone" on one server is not global truth — per-class messages are the only view-independent form.)

`DeleteKeyCommand` is the client→server *command* requesting whole-key deletion at the receiving
server; servers never broadcast it.

## Example Session (Client Perspective)

```json
Sent: {"message_type": "Introduction", "value": { "version": 650269, "heartbeat_timeout_interval": 3000, "user": "jonh" }}
Received: {"message_type": "Introduction", "value": { "version": 650269, "heartbeat_timeout_interval": 3000, "user": "jserver" }}
Sent: {"message_type": "Subscribe", "value": { "subscription_mode": "Streaming", ... }}
Received: {"message_type": "ProcessingSnapshot", "value": { "name": "S1" }}
Received: {"message_type": "TopicIntroduction", "value": { "topic_id": 18, "name": "top_bid_price" }}
...
Received: {"message_type": "SnapshotComplete", "value": { "name": "S1" }}
Sent: {"message_type": "Heartbeat", "value": { "u_milliseconds":, 1745425692890 }
Received: {"message_type": "Heartbeat", "value": { "u_milliseconds":, 1745425693895 }
...
Sent: {"message_type": "Logoff"}
```

## Auditing Topics (Control Port)

When running in audit mode (`--ws-control-port <port>`), GAR will emit internal topic traffic for monitoring

## g Schema

 - [g.gar](g.gar)

## Protocol Schema

- [proto.gar](g/proto.gar)

## Deployment Schema

- [deployment.gar](g/deployment.gar)
