Metadata-Version: 2.5
Name: contextdb-memory-ci
Version: 0.1.0a1
Summary: Block AI agent memory regressions in CI with ContextDB
Project-URL: Homepage, https://contextdb.ai/evals
Project-URL: Documentation, https://contextdb.ai/docs
Project-URL: Repository, https://github.com/atomsai/contextdb-clients
Project-URL: Issues, https://github.com/atomsai/contextdb-clients/issues
Author-email: Atoms AI <gaurav@saaslabs.co>
License-Expression: Apache-2.0
License-File: LICENSE
License-File: NOTICE
Keywords: agent-memory,ci,contextdb,evaluation,memory
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Requires-Python: >=3.10
Requires-Dist: httpx<1,>=0.27
Provides-Extra: dev
Requires-Dist: build>=1.2; extra == 'dev'
Requires-Dist: mypy>=1.11; extra == 'dev'
Requires-Dist: pytest>=8; extra == 'dev'
Requires-Dist: pyyaml>=6; extra == 'dev'
Description-Content-Type: text/markdown

# ContextDB Memory CI

Run a hosted Memory CI suite from any CI shell and fail the job when memory
behavior regresses.

Memory CI is **Hosted Alpha**. This package is `0.1.0a1`, the reusable GitHub
Action tag is `0.1.0`, and there is no availability SLA.

## GitHub Actions (recommended)

```yaml
- name: Run ContextDB Memory CI
  uses: atomsai/contextdb-clients@memory-ci-action-v0.1.0
  with:
    token: ${{ secrets.CONTEXTDB_EVAL_TOKEN }}
    project-id: ${{ vars.CONTEXTDB_PROJECT_ID }}
    suite-id: ${{ vars.CONTEXTDB_EVAL_SUITE_ID }}
    upload-artifacts: "true"
```

The Action pins this package version, writes a safe GitHub summary, and uploads
content-free JSON and JUnit exports after every CLI result. See the
[Action guide](../action/README.md) and
[example workflow](../action/examples/github-actions.yml).

## Raw CLI

Use the CLI directly in other CI systems or when you need a custom shell step:

```bash
pip install contextdb-memory-ci==0.1.0a1

contextdb-memory-ci run \
  --project-id "$CONTEXTDB_PROJECT_ID" \
  --suite-id "$CONTEXTDB_EVAL_SUITE_ID" \
  --json-output memory-ci.json \
  --junit-output memory-ci.xml
```

Set `CONTEXTDB_EVAL_TOKEN` to a `cbe_` evaluation token before running the
command. Keep the token in a CI secret store. The client sends it only in the
`Authorization` header and never prints it.

The command starts one run, polls to a terminal status, then writes requested
safe exports atomically. It exits:

- `0` for `passed` plus `unchanged`
- `1` for failed behavior or a regression
- `2` for operational errors, cancellation, timeout, or no baseline

A passed run with no baseline exits `2` by default. Use
`--allow-no-baseline` only when that policy is intentional.

The public client accepts and displays aggregate status, opaque IDs, bounded
machine failure codes, and content hashes. It rejects undeclared status
fields. It does not receive suite names, queries, assertions, memory content,
raw failures, or case details.

## When to run Memory CI

- Before changing an embedding model, retrieval policy, or memory-formation
  prompt, run the pinned suite and block evidence or action-outcome changes.
- After adding a source backfill, verify that expected customer context appears
  without reviving forbidden stale evidence.
- Before releasing a voice or support agent, confirm the same memory cases
  still produce the expected `act`, `ask`, or `abstain` result.

Run and export retention can be bounded, and service failures exit `2` rather
than being reported as behavioral regressions.

## Frequently asked questions

### Does the CLI receive customer memory or test queries?

No. Polling returns aggregate status and opaque IDs. JSON and JUnit exports are
content-free. The CLI rejects response fields outside the published contract.

### Which exit code should block a pull request?

Exit `1` means observed memory behavior failed or regressed. Exit `2` means the
check could not make a behavioral determination, so fail the job and inspect
the operational error separately.

### What happens when a suite has no baseline?

A passing run exits `2` by default. Pin a successful baseline in ContextDB, or
use `--allow-no-baseline` only for an intentional first-run workflow.

### Is the evaluation token the same as a project API key?

No. `cbe_` tokens are project-bound automation credentials limited to Memory
CI run and export routes. Keep them in CI secret storage.

- [Memory CI](https://contextdb.ai/evals)
- [ContextDB documentation](https://contextdb.ai/docs)
- [Public API contract](https://github.com/atomsai/contextdb-clients/blob/main/openapi/evals-v1.yaml)

Apache-2.0. ContextDB trademarks are not licensed by Apache-2.0.
