Metadata-Version: 2.3
Name: engram-memory-system
Version: 0.6.0
Summary: Memory, hybrid retrieval, entity graphs, and project continuity for agents.
Project-URL: Homepage, https://github.com/raya-ac/engram
Project-URL: Repository, https://github.com/raya-ac/engram
Project-URL: Issues, https://github.com/raya-ac/engram/issues
Author: raya
License: MIT
Keywords: ai,cognitive,embeddings,hnsw,mcp,memory,rag,retrieval
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Database
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.11
Requires-Dist: aiosqlite>=0.19.0
Requires-Dist: fastapi>=0.115.0
Requires-Dist: hnswlib>=0.8.0
Requires-Dist: jinja2>=3.1.0
Requires-Dist: numpy>=1.24.0
Requires-Dist: psycopg[binary]>=3.2.0
Requires-Dist: pymupdf>=1.24.0
Requires-Dist: pyyaml>=6.0
Requires-Dist: sentence-transformers>=3.0.0
Requires-Dist: sse-starlette>=2.0.0
Requires-Dist: uvicorn>=0.30.0
Provides-Extra: anthropic
Requires-Dist: anthropic>=0.40.0; extra == 'anthropic'
Provides-Extra: api
Requires-Dist: anthropic>=0.40.0; extra == 'api'
Requires-Dist: google-genai>=1.0.0; extra == 'api'
Requires-Dist: openai>=1.30.0; extra == 'api'
Requires-Dist: voyageai>=0.3.0; extra == 'api'
Provides-Extra: dev
Requires-Dist: httpx>=0.27; extra == 'dev'
Requires-Dist: pytest; extra == 'dev'
Requires-Dist: pytest-asyncio; extra == 'dev'
Provides-Extra: docs
Requires-Dist: mkdocs-material>=9.5.0; extra == 'docs'
Requires-Dist: mkdocs>=1.6.0; extra == 'docs'
Provides-Extra: gemini
Requires-Dist: google-genai>=1.0.0; extra == 'gemini'
Provides-Extra: openai
Requires-Dist: openai>=1.30.0; extra == 'openai'
Provides-Extra: voyage
Requires-Dist: voyageai>=0.3.0; extra == 'voyage'
Description-Content-Type: text/markdown

<p align="center">
  <img src="assets/logo-512.png" alt="engram" width="160">
</p>

# engram

memory for work that continues after the conversation ends.

i built engram to keep the things an agent should be able to return to: decisions,
errors, project context, procedures, and the connections between them. it stores
memories in sqlite or postgres, searches them through several retrieval signals,
and exposes the same store through a CLI, an MCP server, and a web workspace.

it also has a lifecycle. memories can be edited, challenged, superseded, promoted,
or forgotten. retrieving something is not proof that it is still true. keeping
that distinction visible matters more than making every query return an answer.

## what lives here

- **retrieval:** dense embeddings, full-text search, entity relationships, and
  associative retrieval, combined with intent-aware ranking and optional rerankers.
- **memory:** working, episodic, semantic, procedural, and codebase layers, with
  separate fact, procedure, and narrative types.
- **continuity:** session diaries, structured handoffs, recent context, decisions,
  error patterns, and explicit negative knowledge.
- **maintenance:** deduplication, consolidation, retention, entity graph tools,
  filesystem drift checks, and status history.
- **inspection:** a web dashboard, retrieval explanations, query comparisons,
  activity views, exports, and direct access to the underlying database.
- **dormant recall:** an optional shadow experiment that looks for a neglected
  relevant connection without putting it into ordinary answers.

some write and maintenance paths use an LLM for extraction, enrichment, or
consolidation. local embedding search can run without an LLM service. choosing an
API embedding or LLM backend sends the relevant inputs to that provider.

## get it running

python 3.11 or newer is required. for the code in this checkout:

```sh
git clone https://github.com/raya-ac/engram.git
cd engram
python3 -m venv .venv
. .venv/bin/activate
pip install -e ".[dev]"
cp config.example.yaml config.yaml
```

the default embedding model is `BAAI/bge-small-en-v1.5`. local models need to be
available on first use; subsequent runs can use their cached weights. the default
backend selects an available local implementation. set
`embedding_backend: sentence_transformers` for an explicit CPU setup.

check `config.yaml` before storing anything. it controls the database, models,
LLM backend, web binding, and optional experiments. the example uses an API LLM
backend for write-time processing; configure it for your environment.

```sh
engram --config config.yaml status
engram --config config.yaml remember "the release requires a restore drill before activation"
engram --config config.yaml search "release preparation" -k 5 --json
engram --config config.yaml search "release preparation" --rerank --debug
```

`--config` comes before the command. without it, engram checks the current
working directory, the source checkout, then `~/.config/engram/config.yaml`.
explicitly supported `ENGRAM_*` environment settings take precedence over the
file; see [config.py](engram/config.py) and [the example](config.example.yaml).

## storage and models

sqlite is the default. it uses FTS5 and stores its database at
`~/.local/share/engram/memory.db`. postgres uses the same memory model with its own
full-text search and connection adapter.

```yaml
storage_backend: sqlite
db_path: ~/.local/share/engram/memory.db
embedding_backend: sentence_transformers
embedding_model: BAAI/bge-small-en-v1.5
embedding_dim: 384
```

for postgres, set `storage_backend: postgres` and provide `ENGRAM_POSTGRES_DSN`
or a private `postgres_dsn` config value. keep credentials out of commits.
`engram migrate-postgres --help` describes the existing sqlite migration tool.
back up the source first and check its verification output before switching.

local embeddings and cross-encoder reranking are included in the base install.
optional extras provide the API clients:

```sh
pip install -e ".[voyage]"
pip install -e ".[openai]"
pip install -e ".[gemini]"
pip install -e ".[api]"
```

model selection determines the embedding backend for recognized API models.
configure the corresponding provider key in the process environment. changing
embedding models requires compatible dimensions and re-embedding existing data;
use `engram reembed --dry-run` before `engram reembed`.

## use it from an agent

start a stdio MCP server:

```sh
engram --config /absolute/path/to/config.yaml serve --mcp
```

an MCP client can launch it with an explicit executable and config:

```json
{
  "mcpServers": {
    "engram": {
      "command": "/absolute/path/to/engram/.venv/bin/engram",
      "args": ["--config", "/absolute/path/to/config.yaml", "serve", "--mcp"]
    }
  }
}
```

use `recall_recent` for chronology and `recall_hints` for lightweight recognition.
use `recall` for a concrete semantic query. `remember`, `remember_decision`, and
`remember_error` store context; `session_handoff` and `resume_context` support
resuming work. tools also cover entities, timelines, memory status, codebase
scanning, drift, and consolidation. the server's `tools/list` is the current
interface; [mcp_server.py](engram/mcp_server.py) defines it.

`recall` supports `facts_only`, `facts_plus_rules`, and `full_context` profiles.
ordinary search records returned results as accesses. that is a retrieval signal,
not explicit evidence that someone used the result.

## native application integration

`engram --config /absolute/config.yaml api` runs a persistent local JSONL
process. it exposes project context, semantic search, session checkpoints and
structured evidence through a supported service boundary. a harness can launch
Engram in its own Python environment without importing the store or speaking MCP.

Kiln's native experience and Mythic's check/planning lifecycle belong to those
projects. Engram supplies reusable memory and evidence APIs. evidence records
carry caller provenance, timestamps, expiry and source references; they are not
independently verified merely because they were stored. referenced forgotten or
inactive memories cannot remain eligible supporting evidence. no observation
can instruct Engram to execute a command or fetch a URL.

[the native API contract](docs/native-api.md) distinguishes nonreinforcing scoped
context from ordinary semantic search, which records accesses. it also describes
immutable evidence, explicit unknown/stale states and supported operations.

## optional Codex compatibility

the optional separate adapter exposes `codex_context`, `codex_checkpoint`,
`codex_diagnostics`, and a scoped `codex_evidence` reader. it reads explicitly scoped active
memories, saves a deliberate task handoff, and reports its own process and
persisted index coverage. it doesn't alter core retrieval, collect transcripts,
install hooks, or reinforce memories when context is read.

```sh
engram --config /absolute/engram/config.yaml codex setup --project /absolute/project
```

that prints a supported `codex mcp add` command for review; it does not register
anything automatically. each adapter process is bound to one canonical project
directory. unscoped legacy notes and sibling projects stay outside its context.
checkpoints are separate from memory records and can be replaced or cleared.
the full Engram MCP server can remain connected alongside it for broader recall.

[adapter setup, task-start/resume workflow and limits](docs/codex-adapter.md)
includes tested registration syntax and explains what diagnostics can establish.
a fresh adapter's PID does not prove another running client has reloaded code.

## how retrieval works

```text
query → intent and query features
      → dense + full-text + entity graph + associative candidates
      → reciprocal rank fusion
      → date, importance, recency and access boosts
      → optional cross-encoder and trained deep reranker
      → noise and threshold gate → ordinary results

original query → optional independent dormant search → separate shadow log
```

the ANN index accelerates dense retrieval when available; a brute-force embedding
path remains available. the CLI makes cross-encoder reranking opt-in with
`--rerank`; MCP recall uses it. the deep reranker runs when a trained model is
available. debug output explains the ordinary ranking stages.

ranking scores depend on the stage and model. a cosine score, a cross-encoder
score, and a fused ranking score are not interchangeable confidence estimates.
small random ranking noise also means uncached scores can vary between runs.

## dormant recall

sometimes an old project stopped because a capability did not exist yet. later,
a new capability appears, but ordinary retrieval keeps returning the recent,
often-used material. dormant recall asks whether a neglected memory is relevant
to the task now.

this is an experiment. it is **off by default**. enabling shadow mode evaluates
at most one candidate per hybrid search, or none, and writes the result separately.
it does not add suggestions to recall output, hints, context, or normal answers.
there is no automatic visible mode in this version.

```yaml
dormant_recall:
  mode: shadow                 # use "off" to disable; quote it in YAML
  candidate_limit: 50
  dormancy_days: 30
  min_relevance: 0.75
  max_bonus: 0.05
  rerank_candidates: 12
  min_rerank_score: 0.6
  cooldown_days: 7
  feedback_cooldown_days: 30
  log_max_events: 1000
  log_retention_days: 30
```

the candidate search reads current active, dormant embeddings directly from the
database, independently of ordinary results and the ANN cache. a stale index
cannot hide a newer memory from this path. candidates must pass the cosine floor
and a separate query/content relevance check. a bounded
bonus from dormancy and existing importance can reorder close relevant matches;
age cannot bring an irrelevant item through the gate. literal word overlap is
supporting evidence, not a requirement, so differently worded connections can
still qualify. the explanation reports the evidence without inventing a story
about why an old problem is now solved.

only active, non-forgotten memories qualify, with the same memory-type profile
as the search. forgotten, archived, deleted, superseded, merged, or otherwise
inactive memories are excluded. the current forgotten flag covers both lifecycle
archival and explicit forgetting; this experiment does not try to separate them.

review it explicitly:

```sh
engram --config config.yaml dormant review --limit 20
engram --config config.yaml dormant inspect EVENT_ID
engram --config config.yaml dormant feedback EVENT_ID useful
```

`review` lists metadata. `inspect` fetches the eligible candidate and records a
separate exposure timestamp. feedback is `useful`, `irrelevant`, or `dismissed`;
`useful` means the connection was actually used. MCP provides `dormant_review`,
`dormant_inspect`, and `dormant_feedback` for the same workflow.

computing, inspecting, or giving feedback on a candidate never increments its
ordinary access count, updates its ordinary last-accessed time, or changes its
importance. silence does not mean useful. cooldown and explicit-use state survive
log rotation. the bounded log stores IDs, times, numeric signals, and feedback,
without copying query text, matching words, or memory content.

set `ENGRAM_DORMANT_RECALL_MODE=off` to override the file at process start.
existing MCP processes do not hot-reload these edits. the feature creates only
additive experiment tables and contains its failures so normal recall can finish.
see [the dormant recall guide](docs/dormant-recall.md) for ranking, persistence,
retention, and feedback details.

**what is verified:** isolated SQLite and PostgreSQL tests cover eligibility,
independent candidates, bounded ranking effects, cooldown, feedback, persistence,
concurrency, failure isolation, and absence of reinforcement. a separate CLI/MCP
smoke test uses synthetic memories and the cached local embedding model.

**what remains unproven:** usefulness in real work and threshold calibration.
in the initial cosine-only probe, a sentence with no literal query overlap scored
about 0.734. the 0.75 default abstained; a lower diagnostic threshold admitted it.
the added relevance check now rejects that thin sentence. the current interface
smoke test uses a more informative prototype note, which still falls below 0.75
and is admitted only in its separately labelled 0.70 isolated pilot. that is a
concrete precision/recall tradeoff, not evidence of broad usefulness.

the repair also recovered a real Junkstep release note with one previous access
and about 37 days of dormancy. it was missing from the old ANN index but ranked
first against current database vectors at 0.817. in an isolated snapshot it added
installer-packaging and release-verification context missing from ordinary
results, passed the relevance check, and remained unreinforced. this is a targeted
evaluator assessment; no positive user feedback was fabricated. exact scanning
trades additional database work for current coverage, so larger stores need
measurement before treating this as a scaling solution.

## keep the store understandable

```sh
engram ingest /path/to/notes/
engram entity "project name" --graph
engram drift --json
engram patterns --dry-run
engram index status
engram export memories.json --include-embeddings
engram import memories.json --skip-duplicates
engram consolidate
```

consolidation can cluster, summarize, promote, and soft-archive memories. drift
checks compare stored filesystem references with current files; `drift --fix`
changes memory state. inspect these operations before using them on a valued
store. they are maintenance actions, not substitutes for validating a claim.

the portable export includes active memories. it is not a lossless backup of
forgotten records, all history, or the dormant experiment tables. use native
sqlite or postgres backup tooling when you need the full store, and verify a
restore into an isolated database. the legacy sqlite-to-postgres migration does
not yet transfer dormant telemetry or its cooldown state.

## inspect it in the browser

```sh
engram --config config.yaml serve --web
```

the workspace binds to `127.0.0.1:8420` by default. its redesigned memory list,
navigation and inspector share a restrained archive layout. all eighteen original
views remain available, alongside dormant review. mobile Menu and Activity
drawers retain navigation and inspection; `/` focuses search, and view URLs can
be bookmarked. memory editing loads the full record rather than the list excerpt.

[web workspace controls](docs/guides/web-workspace.md) cover search, hints,
filters, continuity, graph and maintenance workflows. `web.auth_token`
configures bearer authentication. keep the database and config private, and
check authentication and network exposure before making the service reachable
outside the local machine. `serve --mcp-sse` is also available for HTTP MCP use.

## development and checks

```sh
python -m pytest tests/ -q
```

the ordinary suite expects cached local embedding weights; the shared fixtures
set Hugging Face offline mode. dormant contract tests use temporary sqlite files.
set `ENGRAM_TEST_POSTGRES_DSN` to a disposable test cluster to also run their
postgres cases; each creates and drops its own randomly named schema.

```sh
python -m pytest tests/test_dormant.py -q
python tests/dormant_smoke.py --work-dir /path/to/new/disposable/directory
```

the smoke test starts fresh CLI and stdio MCP processes against synthetic sqlite
stores. it compares ordinary result IDs with the experiment off and on, exercises
review/inspection/feedback, checks cooldown, and reads back access and importance
fields. it writes a JSON report in the supplied directory. it does not open a
production config or restart an existing service.

[config](engram/config.py), [store](engram/store.py),
[retrieval](engram/retrieval.py), [dormant recall](engram/dormant.py),
[lifecycle](engram/lifecycle.py), and [the MCP server](engram/mcp_server.py) are the
main entry points; the [native service](engram/service.py) is the application
boundary and [the optional Codex adapter](engram/adapters/codex.py) is a separate
integration layer. further guides and reference material live in [docs](docs/).

## license

[MIT](LICENSE).

![the memory workspace, using fictional verification data](docs/assets/workspace-library.png)
