Metadata-Version: 2.5
Name: lmcache-kvblockd
Version: 0.2.1
Summary: LMCache remote-storage backend backed by kvblockd
Project-URL: Homepage, https://kvstash.com
Project-URL: Documentation, https://kvstash.com/integrations/lmcache/
Project-URL: Get the daemon, https://kvstash.com/docs/install/
Author-email: Rudray Mehra <rudraymehra@gmail.com>
License-Expression: Apache-2.0
Requires-Python: >=3.10
Requires-Dist: kvblockd>=0.2.0
Requires-Dist: lmcache<0.6,>=0.5.1
Requires-Dist: numpy
Provides-Extra: test
Requires-Dist: pytest>=8; extra == 'test'
Requires-Dist: torch; extra == 'test'
Description-Content-Type: text/markdown

# lmcache-kvblockd

[LMCache](https://github.com/LMCache/LMCache) remote storage plugin for
[KVStash](https://kvstash.com). vLLM deployments that already run LMCache can
keep KV in the KVStash daemon (`kvblockd://` scheme) and reload it after a
restart or from another server.

- Get the daemon (free download): https://kvstash.com/docs/install/
- Full guide: https://kvstash.com/integrations/lmcache/

Linux x86-64 only (LMCache ships manylinux wheels only).

## Install

```sh
pip install lmcache-kvblockd
```

## Configure

```sh
PYTHONHASHSEED=0 vllm serve MODEL --kv-transfer-config '{
  "kv_connector": "LMCacheConnectorV1", "kv_role": "kv_both",
  "kv_connector_extra_config": {
    "lmcache.remote_storage_plugins": "kvblockd",
    "lmcache.local_cpu": true,
    "lmcache.chunk_size": 256,
    "lmcache.extra_config": {
      "kvblockd_token": "YOUR_TOKEN",
      "remote_storage_plugin.kvblockd.module_path": "lmcache_kvblockd.adapter",
      "remote_storage_plugin.kvblockd.class_name": "KvblockdConnectorAdapter",
      "remote_storage_plugin.kvblockd.url": "kvblockd://STORE_HOST:9440?namespace=lmcache&streams=4"
    }
  }
}'
```

- The address goes in `remote_storage_plugin.kvblockd.url`. LMCache ignores
  `remote_url` when `remote_storage_plugins` is set, and setting both creates
  a second backend that doubles every write.
- Set `PYTHONHASHSEED=0` on every worker. LMCache's chunk keys depend on it,
  so workers with different seeds cannot share cache.
- Use a separate namespace per model revision, dtype and cache layout.

## LMCache's CUDA extension is required on a GPU

Without its compiled `lmcache.cuda_ops` extension, LMCache copies KV between
the GPU and CPU with a torch fallback that does not handle the fused K/V layout
vLLM uses since 0.26. The bytes it hands the plugin are then not the KV, and a
reload gives wrong answers. So when a GPU is present and the extension does not
import, the plugin refuses to start and logs why. LMCache then marks itself
unhealthy and, with its default recompute policy, stops caching (its own CPU
tier would take the same wrong copy); requests are still served. Check before
you deploy:

```sh
python -c "import lmcache.cuda_ops"
```

If that fails, install an LMCache build compiled for your torch version.

## Behaviour notes

- Cache tags must have string names and values without `@` or `%` (LMCache's
  text keys do not escape them). Unsupported tags become cache misses.
- GET and lookup deadlines include queueing and connection set-up.
- Lookup with `pin=True` reports only the leading run of keys whose
  time-limited leases were acknowledged. A lease can expire; LMCache's full
  asynchronous pin/unpin lifecycle is not implemented.
- Records with malformed headers or contradictory tensor sizes are removed so
  they can be written again. Other read problems stay misses without deleting
  valid records. Do not share a namespace between incompatible writers.
