Metadata-Version: 2.5
Name: livekit-plugins-voicemem
Version: 0.2.2
Summary: Long-term memory for LiveKit voice agents, backed by PostgreSQL and pgvector.
Project-URL: Homepage, https://github.com/mahimailabs/livekit-plugins-voicemem
Project-URL: Source, https://github.com/mahimailabs/livekit-plugins-voicemem
Project-URL: Issues, https://github.com/mahimailabs/livekit-plugins-voicemem/issues
Project-URL: Changelog, https://github.com/mahimailabs/livekit-plugins-voicemem/blob/main/CHANGELOG.md
Author-email: Mahimai Labs <mahimairaja3@gmail.com>
License-Expression: Apache-2.0
License-File: CHANGES-FROM-UPSTREAM.md
License-File: LICENSE
License-File: NOTICE
License-File: third_party/VoiceMem-LICENSE.txt
License-File: third_party/mem0-LICENSE.txt
Keywords: agents,ai,livekit,memory,pgvector,rag,voice
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Multimedia :: Sound/Audio :: Speech
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Requires-Dist: huggingface-hub>=0.26
Requires-Dist: livekit-agents<1.8,>=1.6.0
Requires-Dist: numpy>=1.26
Requires-Dist: onnxruntime<1.24,>=1.18; python_version < '3.11'
Requires-Dist: onnxruntime>=1.18; python_version >= '3.11'
Requires-Dist: openai<4,>=2.0
Requires-Dist: psycopg[binary,pool]>=3.2
Requires-Dist: tokenizers>=0.20
Description-Content-Type: text/markdown

<picture>
  <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/mahimailabs/livekit-plugins-voicemem/main/docs/assets/logo-dark.png">
  <img alt="VoiceMemory" src="https://raw.githubusercontent.com/mahimailabs/livekit-plugins-voicemem/main/docs/assets/logo-light.png" width="420">
</picture>

Long-term memory for [LiveKit Agents](https://docs.livekit.io/agents/) voice agents, backed by
PostgreSQL and pgvector. Your agent remembers what a caller told it last week.

[![PyPI](https://img.shields.io/pypi/v/livekit-plugins-voicemem?color=22d3ee)](https://pypi.org/project/livekit-plugins-voicemem/)
[![Python](https://img.shields.io/pypi/pyversions/livekit-plugins-voicemem?color=22d3ee)](https://pypi.org/project/livekit-plugins-voicemem/)
[![License](https://img.shields.io/badge/license-Apache--2.0-f59e0b)](LICENSE)
[![LiveKit Agents](https://img.shields.io/badge/livekit--agents-1.6%20%7C%201.7-f59e0b)](https://docs.livekit.io/agents/)

Every turn a caller speaks splits two ways. **Facts** are discrete and retrievable: they are
allergic to shellfish, their manager is Priya. **Traits** are continuous and never stated
outright: they get anxious before design reviews, they want the conclusion before the reasoning.

Most memory layers store only the first kind, which is why agents that "remember" still feel like
they are reading your file back to you. This stores both, and injects them differently: facts the
agent may repeat, traits it may only be shaped by.

Embeddings run in process by default: **2.2 ms p50**, against 232 ms for an OpenAI round trip.
No torch, no CUDA, no multi-gigabyte image. `onnxruntime` comes with Silero and the turn
detector already, so for most agents it costs nothing new.

> **Status: 0.2.2.** The schema may change between minor versions while 0.x. 0.2.0 changed the
> default embedding model, and therefore the vector width: existing databases keep the model they
> were built with, so nothing breaks, but they stay on it.

## Install

```bash
pip install livekit-plugins-voicemem
# plus whichever STT, TTS and LLM you use, which are separate LiveKit plugins:
pip install "livekit-agents[deepgram,openai,silero,turn-detector]~=1.7"
```

Requires PostgreSQL 14 or later with [pgvector](https://github.com/pgvector/pgvector) 0.8+,
an OpenAI key, and LiveKit Agents 1.6 or 1.7.

## Set up the schema

The plugin owns its tables in their own `voicemem` schema, so it never collides with yours.

```bash
export VOICEMEM_ADMIN_DSN=postgresql://admin@host/db   # a role with DDL rights
voicemem-db --dsn "$VOICEMEM_ADMIN_DSN" upgrade
voicemem-db --dsn "$VOICEMEM_ADMIN_DSN" status
```

Migrations do not run at startup, on purpose. Twenty workers booting at once and racing DDL is a
real failure, not a theoretical one: `CREATE TABLE IF NOT EXISTS` is not race-safe in PostgreSQL.
Use `voicemem-db sql` to print the DDL if your shop reviews schema changes.

## Use it

```python
from livekit.agents import Agent, AgentSession
from livekit.plugins import voicemem

class Assistant(Agent):
    def __init__(self, hooks):
        super().__init__(instructions="You are a helpful assistant.")
        self._hooks = hooks

    async def on_user_turn_completed(self, turn_ctx, new_message):
        await self._hooks.on_user_turn_completed(turn_ctx, new_message)   # the one line

async def entrypoint(ctx):
    await ctx.connect()
    runtime = await voicemem.build(voicemem.Config(
        pg_dsn=os.environ["VOICEMEM_PG_DSN"],
        openai_api_key=os.environ["OPENAI_API_KEY"],
    ))
    participant = await ctx.wait_for_participant()
    hooks = voicemem.MemoryHooks(runtime.session(user_id=participant.identity))

    session = AgentSession(
        stt=..., llm=..., tts=..., vad=...,
        turn_handling={"preemptive_generation": {"enabled": False}},   # see Limitations
    )
    hooks.attach(session)                     # once, here, not in Agent.on_enter

    @session.on("conversation_item_added")
    def _on_item(ev):
        if ev.item.role == "assistant":
            user_text = next((i.text_content for i in reversed(session.history.items)
                              if i.role == "user"), "")
            hooks.remember_turn(user_text, ev.item.text_content or "")

    ctx.add_shutdown_callback(hooks.aclose)
    ctx.add_shutdown_callback(runtime.aclose)
    await session.start(agent=Assistant(hooks), room=ctx.room)
```

See `examples/basic_agent.py` for the whole file.

## How it works

Two stores, following [VoiceMem](https://github.com/xzf-thu/VoiceMem)'s design.

**The left brain stores facts.** It extracts atomic statements from each turn, files them under one
of seven life domains, and retrieves them by vector search. "User is allergic to shellfish."

**The right brain stores what the person is like.** Preferences, habits, and emotional patterns:
"gets anxious before design reviews". These are injected as internal notes the agent must never
repeat aloud, and they are what stops the agent sounding like it is reading your file back to you.

**Reading** runs inside the voice turn: embed the query once, classify it into slots, narrow to
those candidates, rank, and fetch relevant traits concurrently. No LLM call.

**Writing** runs after the turn, never between the user finishing and the agent speaking. One
extraction call, plus a conflict-resolution call when there is anything to conflict with.

## Measured latency and cost

Real numbers from `scripts/bench_latency.py`, not from the VoiceMem paper. 8 stored turns,
12 queries, both backends measured on the same machine minutes apart.

**Read path**, the part inside the voice turn, p50 milliseconds:

| stage | local (default) | OpenAI |
|---|---|---|
| embed query | **3.2** | 237.2 |
| classify into slots | 0.2 | 0.6 |
| rank (pgvector + rerank) | 1.6 | 5.2 |
| right brain (traits) | 1.5 | 5.0 |
| **total** | **5.3** | **241.4** |
| **total p95** | **18.9** | **370.3** |

**Embedding was 97% of the read path, and it was a network call.** Running it in process removes
it: 5.3 ms against 241.4 ms, and the tail matters more than the median. `recall_budget_s` is
600 ms, so an OpenAI call that lands in its tail does not merely run slow, it overruns the budget
and injects nothing at all, silently. Ranking gets faster too, because 384-wide vectors are cheaper
to compare than 1536-wide ones.

The model is 118 MB of ONNX and loads once at startup, in about 0.8 s.

Retrieval is still prefetched on interim transcripts, which mattered enormously when a turn cost
240 ms and matters little now. It stays because it costs nothing.

**Write path**, background: 3.8 s p50, 5.5 s p95, at **2 LLM calls per ingested turn**
(1 when the store is empty and conflict resolution is skipped).

*Environment: same-host Docker `pgvector/pgvector:pg17`, Apple Silicon, Python 3.12,
livekit-agents 1.7.1. A managed database in another region will be slower; on the local backend
the `embed` row will not change, the others will. The OpenAI `embed` row is the one that moves
most, and not predictably: a separate 36-sample run of the same queries measured 232 ms p50 but
3209 ms p95.*

## Limitations

Stated up front rather than buried.

- **Emotion is inferred from text**, by the extraction model reading what was said. There is no
  prosody analysis, no acoustic emotion, no voiceprint and no speaker identification. Upstream
  ships those; this plugin does not, because they require torch, funasr and modelscope. Text
  catches "I'm frustrated". It cannot catch a flat "fine" said bitterly.
- **Memory injection disables LiveKit's preemptive generation.** Injecting changes the turn's chat
  context, so the framework's equivalence check fails and it cancels the speculative generation it
  had already started. Preemptive generation is **on by default**, so unless you turn it off you
  pay for a discarded LLM call on every turn. Disable it as shown in the quickstart. The plugin
  logs a warning once at startup if you have not.
- **Relative-date expansion covers English and Chinese only**, and its output format must match how
  the extractor writes dates. "next week" becomes `August 31, 2026`.
- **The plugin owns 24 tables** in your database, in the `voicemem` schema. The 0.x schema is not
  stable.
- **`on_user_turn_completed` does not fire** for realtime models using server-side turn detection,
  so memory is neither injected nor ingested on those turns.

## Multi-tenancy

Every table carries `tenant_id` and every query scopes by it. Migration `0002_rls.sql` adds
PostgreSQL row-level security on top, so a missed `WHERE` clause cannot cross tenants.

**This only works if you connect as a constrained role.** Superusers and `BYPASSRLS` roles walk
straight through row-level security no matter what the policies say. Migration 0002 creates
`voicemem_app` for this; give it a password and point `pg_dsn` at it.

```bash
voicemem-db --dsn "$DSN" status     # reports whether isolation is actually in force
```

## Swapping implementations

Four seams, defined as `typing.Protocol`, so your implementation never imports anything from here.

```python
class MyEmbedder:                       # satisfies voicemem.protocols.Embedder
    @property
    def model_name(self) -> str: ...
    @property
    def dimensions(self) -> int: ...
    async def embed_documents(self, texts): ...
    async def embed_query(self, text): ...
```

`Embedder`, `LLMClient`, `VectorStore` and `GraphStore`. `container.py` is the only file that names
a concrete class.

## Development

```bash
uv sync --group dev
docker compose up -d
export VOICEMEM_TEST_DSN=postgresql://voicemem_app:apppass@localhost:55432/voicemem_test
uv run pytest                 # integration tests skip without a DSN
uv run ruff check .
```

`pytest` on a fresh clone with no database and no API key passes: the contract suite runs the whole
retrieval chain against in-memory fakes.

## Attribution

Derived from [VoiceMem](https://github.com/xzf-thu/VoiceMem) (Apache-2.0). Prompt templates
originate from [mem0](https://github.com/mem0ai/mem0) (Apache-2.0). See `NOTICE` for attribution
and `CHANGES-FROM-UPSTREAM.md` for what was changed and why.

Apache-2.0. Not affiliated with, endorsed by, or sponsored by LiveKit, the VoiceMem authors, or
mem0.
