Metadata-Version: 2.4
Name: modelkeeper
Version: 0.1.0
Summary: Keep apps on local model servers (Ollama, LM Studio) alive through silent model eviction.
Project-URL: Homepage, https://github.com/modelkeeper/modelkeeper
Project-URL: Issues, https://github.com/modelkeeper/modelkeeper/issues
Author: modelkeeper contributors
License: MIT
License-File: LICENSE
Keywords: keep-alive,llm,lm-studio,local-llm,ollama,reliability,watchdog
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development :: Libraries
Classifier: Topic :: System :: Monitoring
Requires-Python: >=3.10
Requires-Dist: httpx>=0.24
Provides-Extra: dev
Requires-Dist: pytest>=7; extra == 'dev'
Requires-Dist: ruff>=0.1; extra == 'dev'
Description-Content-Type: text/markdown

# modelkeeper

**Keep your app alive when a local model server silently evicts your model.**

You built an app on Ollama or LM Studio. It works on your machine. Then, mid-session, a request comes back with `model not loaded` / `unloaded` and your app falls over — because the server quietly evicted the model out from under you. modelkeeper makes that a non-event: it **prevents** the eviction it can (idle-TTL heartbeats), and **survives** the eviction it can't (warm-and-retry).

```python
from modelkeeper import ModelKeeper

keeper = ModelKeeper(models=["qwen2.5"], backend="auto")
reply = keeper.guard(client.chat, model="qwen2.5", messages=msgs)  # heals + retries once on eviction
```

One dependency (`httpx`). Python ≥ 3.10. Sync-first. MIT.

---

## The problem, with receipts

Local model servers unload your model when *they* decide to — not when you're done with it.

- **Ollama ignores / mishandles `keep_alive`.** Models unload earlier than asked, `keep_alive` doesn't survive restarts, and the timer interacts badly with memory limits and multiple loaded models. This is long-standing and still open:
  - [ollama/ollama#7773](https://github.com/ollama/ollama/issues/7773)
  - [ollama/ollama#9410](https://github.com/ollama/ollama/issues/9410)
  - [ollama/ollama#13227](https://github.com/ollama/ollama/issues/13227)
  - [ollama/ollama#16610](https://github.com/ollama/ollama/issues/16610)
- **LM Studio auto-evicts by design.** It ships an **idle TTL** (60 minutes by default) and an **auto-evict** guardrail that unloads a model when memory gets tight or a new model is loaded. See [LM Studio: TTL and Auto-Evict](https://lmstudio.ai/docs/api/ttl-and-auto-evict).

Both are reasonable server behaviors. Neither is something your *app* can assume away. The first request after an eviction is the one that fails — and if you have a background job, it's the job that eats the stall, silently. There is no canonical fix package for this. modelkeeper is that package.

---

## Prevent, then survive

modelkeeper works in two tiers. Use either or both.

**Prevention** — stop the eviction from happening:
- **Heartbeat** — the `Watchdog` sends each pinned model a minimal touch on a cadence well under the shortest TTL (default 10 min), so the idle timer never reaches zero.
- **`doctor`** — a config-hygiene checklist that flags the footguns *before* they bite (unset `OLLAMA_KEEP_ALIVE`, memory that's too tight for the models you've loaded, LM Studio's TTL, oversized context windows).
- **Right-sized context / RAM** — `doctor` tells you when your loaded footprint is close to the edge.

**Survival** — for the evictions prevention can't stop (OOM, a server restart, another app grabbing the GPU, the `keep_alive` bugs themselves):
- **`guard` / `@guarded`** — on the request path, warm the model and retry **once** when a call hits an eviction.
- **Heal** — the `Watchdog` proactively reloads an evicted pinned model before your next request notices.

Prevention handles the predictable. Survival handles the rest. Neither alone is enough; together your app just keeps working.

---

## Install

```bash
pip install modelkeeper
```

(From source: `pip install -e ".[dev]"`.)

---

## 30-second quickstart

### 1. Request-path guard (survival)

Wrap the call that talks to the model. On an eviction — a raised exception *or* an error field in the returned response — modelkeeper warms the model and retries once. Non-eviction errors are never retried; a second eviction is never retried.

```python
from modelkeeper import ModelKeeper

keeper = ModelKeeper(models=["qwen2.5"], backend="auto")

# imperative
reply = keeper.guard(my_chat_fn, messages=msgs)

# decorator
@keeper.guarded
def chat(msgs):
    return my_chat_fn(messages=msgs)
```

Guarding a function that returns a raw response dict? The default error-extractor
reads OpenAI-style `{"error": {"message": ...}}` envelopes. Plug in your own for
any shape:

```python
keeper = ModelKeeper(models=["qwen2.5"], error_extractor=lambda r: r.get("failure"))
```

### 2. Background watchdog (prevention + survival)

```python
from modelkeeper import Watchdog

wd = Watchdog(
    models=["qwen2.5"],
    backend="auto",
    interval=180,             # probe cadence
    heartbeat_interval=600,   # touch each model every 10 min so the TTL never fires
    on_event=lambda ev: print(ev.kind, ev.target, ev.detail),  # broke / recovered / healed
)
wd.start()   # daemon thread; idempotent
...
wd.stop()
```

The watchdog only calls `on_event` on **state transitions** — a server that stays down overnight produces one `broke` event, not hundreds. Routine heartbeats are silent. `wd.status()` returns a live snapshot (`healthy`, per-model `loaded`/`reloads`/`heartbeats`, and a 50-entry incident ring).

### 3. CLI

```bash
modelkeeper status                     # both backends: reachable? which models loaded?
modelkeeper warm qwen2.5 --backend ollama
modelkeeper watch qwen2.5 --interval 180 --heartbeat-interval 600
modelkeeper doctor                     # config-hygiene checklist (OK / WARN / INFO)
```

Every command supports `--json` for scripting. `watch` runs in the foreground and exits cleanly on Ctrl-C; add `--no-heartbeat` for heal-only mode.

---

## FAQ

**Can't I just set LM Studio's guardrails / disable the TTL, or set Ollama's `keep_alive`?**
Those dials help — turn them up. But they can't reach zero. TTLs still exist and can be re-enabled; servers restart and forget your settings; *another* app can load a model and evict yours under the memory guardrail; and Ollama's `keep_alive` handling is itself buggy (the four issues above). Configuration reduces the odds; it doesn't make your app immune. modelkeeper makes your app survive an eviction **regardless of** how the dials are set — and `modelkeeper doctor` tells you which dials to turn first.

**Does the heartbeat waste GPU/CPU?**
Barely. A heartbeat is a single 1-token / empty-generate request per model per interval (default 10 min). That's the cheapest possible "touch" — the same call used to JIT-load a model — and it's skipped entirely while the server is unreachable.

**Will `guard` hide real errors by retrying?**
No. It retries **only** eviction-class errors, and **only once**. A bad prompt, a 400, an OOM that isn't an eviction — all propagate immediately, unretried. A second eviction on the retry also propagates.

**Is it safe to run against my running LM Studio / Ollama?**
Yes. modelkeeper only *reads* state and *loads* models — it never unloads anything. `status` and `doctor` are read-only; `warm` and the heartbeat only load.

**Async?**
v0.1 is sync-first (threading watchdog, sync `guard`). An asyncio API is on the roadmap — see below.

---

## Scope & honesty

**v0.1 is deliberately small:** Ollama + LM Studio, synchronous, one dependency (`httpx`). It does one thing — keep your model resident and your calls alive — and tries to do it without surprises. It does **not** manage model *selection*, routing, or downloads; it does not proxy your traffic. It reads what a server exposes and is honest about what it can't see (it can't read another process's env or a footprint a server doesn't report — `doctor` says "unknown" instead of guessing).

## Roadmap

Candidates for v0.2 (contributions very welcome — see [CONTRIBUTING.md](CONTRIBUTING.md)):

- **Admission-control forecasting** — "loading model B will likely evict model A: you have X GB free, B needs ~Y GB." Turn the reactive memory WARN into a pre-flight check.
- **Asyncio API** — an `AsyncModelKeeper` / async watchdog mirroring the sync surface, for apps already on `asyncio`.
- **More backends** — vLLM, llama.cpp (`server`), LocalAI, Jan, and any OpenAI-compatible endpoint. The adapter contract is four methods (see CONTRIBUTING).

---

## License

MIT © modelkeeper contributors
