Metadata-Version: 2.4
Name: sotis
Version: 1.3.0
Summary: Reliability library for LLM agents you build (LangGraph / custom loops): detects meltdowns in real time and intercepts them before they spiral.
Requires-Python: >=3.10
Description-Content-Type: text/markdown
Requires-Dist: numpy>=1.26.0
Requires-Dist: pydantic>=2.6.0
Requires-Dist: python-dateutil>=2.9.0
Requires-Dist: openai>=1.25.0
Requires-Dist: anthropic>=0.25.0
Requires-Dist: rich>=13.7.0
Provides-Extra: dev
Requires-Dist: pytest>=8.2.0; extra == "dev"
Requires-Dist: pytest-benchmark>=4.0.0; extra == "dev"
Requires-Dist: pytest-cov>=5.0.0; extra == "dev"
Requires-Dist: langchain>=0.2.0; extra == "dev"
Requires-Dist: langgraph>=0.0.60; extra == "dev"
Provides-Extra: langgraph
Requires-Dist: langchain>=0.2.0; extra == "langgraph"
Requires-Dist: langgraph>=0.0.60; extra == "langgraph"
Provides-Extra: obs
Requires-Dist: streamlit>=1.35.0; extra == "obs"

# Sotis

**Sotis watches your LLM agent and catches it before it spirals.**

[![PyPI version](https://badge.fury.io/py/sotis.svg)](https://pypi.org/project/sotis/)
[![Python 3.10+](https://img.shields.io/badge/python-3.10+-blue.svg)]()
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)]()
[![Architecture](https://img.shields.io/badge/docs-Architecture-00F2FE)](https://github.com/Shaurya-34/Sotis/blob/main/ARCHITECTURE.md)
[![Usage](https://img.shields.io/badge/docs-Usage%20%26%20CLI-3B82F6)](https://github.com/Shaurya-34/Sotis/blob/main/USAGE.md)

![Sotis intercepting a live agent meltdown](https://raw.githubusercontent.com/Shaurya-34/Sotis/main/assets/demo.gif)

*Live run: a real LLM agent spirals on a buggy codebase. Sotis detects three meltdowns in real time, intercepts each one, and degrades gracefully (GDS 1.0 → 0.4) instead of letting the agent burn.*

<sub>📄 Full raw terminal transcript of this run: [`run_groq_llama70b_meltdown_20260601.txt`](https://github.com/Shaurya-34/Sotis/blob/main/ExperimentLog/circular%20import%20trap/run_groq_llama70b_meltdown_20260601.txt)</sub>

```bash
pip install sotis
```

Long-running agents fail in predictable ways — they loop on the same tool calls, flood their context with error traces, and spiral until the task collapses. Sotis detects these failure patterns in real time and transparently resets execution before they take hold.

*Based on ["Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents"](https://arxiv.org/abs/2603.29231) (arXiv:2603.29231, April 2026)*

---

## Who it's for

Sotis is a **library you add to agents you build.** It lives *inside* your
agent's execution loop, so it can do more than observe — it intercepts a
spiraling agent, rolls back the files it touched, distills the context, and
resumes it.

- ✅ **LangGraph agents** — drop in `SotisLangGraphGuard` as a guard node
  ([USAGE.md](https://github.com/Shaurya-34/Sotis/blob/main/USAGE.md))
- ✅ **Custom ReAct / tool-calling loops** — wrap your loop with `SotisGuard`
  for detection, and wire recovery with `CheckpointManager` + `ContextResetter`
  (or use the built-in `SotisRuntime`)
- ✅ **Any LLM provider** — OpenAI, Anthropic, DeepSeek, Google, or
  OpenAI-compatible endpoints (Groq, OpenRouter)

---

## Scope & limitations

Being clear about what Sotis does **not** do is as important as what it does:

- **Reliability, not adversarial security.** Sotis assumes the agent is *trying*
  to do the right thing and degrading. Entropy/loop detection is a statistical
  signal, and any threshold-based signal is gameable by definition — an attacker
  who knows the logic can craft an injection that stays under it. Sotis is a
  circuit breaker for *accidental* failure, not a defense against a deliberate
  adversary. Pair it with a real injection/intent layer if that's your threat
  model.
- **Loud failures are behavioral; quiet ones need invariants.** Entropy + loop
  detection read the *shape* of the tool-call stream, so they catch the visible
  spiral — thrashing, repeats, edit storms — but are blind to the *quiet* failure:
  an agent confidently proceeding on state that silently went wrong, with no loop
  and no entropy spike. For the slice of quiet failure an invariant can express
  (e.g. "a tracked file no longer parses"), `handoff_check=True` runs your
  invariant at each handoff and intervenes at the source as a `STRUCTURAL_FAILURE`
  before the bad state propagates. The harder, *semantic* case — where nothing
  syntactic is wrong but the reasoning is — still needs a world-state signal
  (goal-progress regression, contradiction detection) and remains on the roadmap.
- **Checkpoint "goodness" can be invariant-verified (opt-in).** By default Sotis
  rolls back to the last snapshot. Pass a `checkpoint_invariant` and rollback
  instead targets the most recent state that *passed* your invariant — so if a
  tool call silently corrupted state before the meltdown was visible, you don't
  roll back into the poison. A built-in `python_imports_cleanly` invariant ships
  (every tracked `.py` still parses); supply your own for other domains. See
  [Tuning](#tuning).
- **It bounds failure; it doesn't guarantee success.** Sotis stops the spiral
  and hands you a clean, recoverable state — it does not make a weak model
  finish the task.

See the [open issues](https://github.com/Shaurya-34/Sotis/issues) for the
roadmap addressing these.

---

## Architecture

An agent goal is decomposed into a DAG of subtasks and executed through the
Sotis ReAct runtime or the LangGraph guard. Every step is fed to the **core
layer** — the Shannon entropy monitor, the Jaccard/fingerprint loop detector,
and the workspace density guard. When any of them flags a meltdown, the
Checkpoint Manager rolls the workspace back to its last stable state, the
Context Resetter distills a compact resumption prompt, and execution continues.
Step telemetry streams to a JSON-L logger that the Streamlit dashboard renders.

> Full architecture diagram, module descriptions, and design decisions:
> **[ARCHITECTURE.md](https://github.com/Shaurya-34/Sotis/blob/main/ARCHITECTURE.md)**

---

## Usage

```python
from sotis import SotisGuard

guard = SotisGuard()

for step in range(max_steps):
    action = agent.decide()
    result = tools.execute(action)

    meltdown = guard.watch(action.name, action.args, result.summary)

    if meltdown:
        guard.reset()  # clears detector state; recovery is below
```

`SotisGuard` is detection only. `reset()` clears the detectors' sliding windows
so the next steps are judged fresh; it does not touch your files or your
agent's context. Recovery (rolling tracked files back to the last good
checkpoint, replacing the history with a distilled resumption briefing, and
enforcing the 2-reset cap) is done for you by `SotisLangGraphGuard` and
`SotisRuntime`. In a custom loop, wire it yourself with `CheckpointManager`
(`track()` before the subtask, `rollback()` on meltdown) and
`ContextResetter.distill()` to build the briefing.

### What it looks like in practice

*Illustrative output of a healthy intercept-and-recover cycle. For a real,
unedited run (where a weaker model spirals and Sotis intercepts 3 meltdowns),
see the [demo GIF](#sotis) and [raw transcript](https://github.com/Shaurya-34/Sotis/blob/main/ExperimentLog/circular%20import%20trap/run_groq_llama70b_meltdown_20260601.txt) above.*

```
[Step 22] write_file -> {"path": "src/main.py", "content": "import math"} | SUCCESS
[Step 23] run_tests  -> {"cmd": "pytest"} | FAIL (ImportError)
[Step 24] write_file -> {"path": "src/main.py", "content": "import math"} | SUCCESS
[Step 25] run_tests  -> {"cmd": "pytest"} | FAIL (ImportError)

[WARNING]   Anomaly detected: Workspace edit storm and exact argument loops
[INTERCEPT] Sotis Meltdown Interception Triggered!
[RECOVER]   Restored workspace files to stable baseline (step 22 diff)
[RECOVER]   Distilled session context history (86% token savings)
[RESUME]    Injecting resumption briefing into agent context...

[Step 26] grep_search -> {"query": "math"} | Execution resumed cleanly
```

---

## CLI

```bash
sotis dashboard    # Launch the Streamlit observability dashboard
sotis benchmark    # Run the empirical benchmark suite
sotis demo         # Run the built-in meltdown/recovery demo
```

The dashboard reads session telemetry from `logs/`. Point it elsewhere with
`sotis dashboard --logs <path>` or the `SOTIS_LOG_DIR` environment variable —
set the same value when running your agent so both write and read the same
directory regardless of where each is launched.

Full command reference and the LangGraph integration guide: **[USAGE.md](https://github.com/Shaurya-34/Sotis/blob/main/USAGE.md)**

---

## Tuning

The default entropy threshold (`1.5 bits`) is calibrated for agents that use 1-2 tools in tight loops. If your agent legitimately uses 3+ different tools in a short window, the default will fire false positives — `log2(3) = 1.585 > 1.5`.

Raise the threshold for multi-tool agents:

```python
from sotis import SotisGuard
from sotis.core.entropy import EntropyConfig

guard = SotisGuard(entropy_config=EntropyConfig(hard_threshold=2.7))
```

| Threshold | Behavior |
|---|---|
| `1.5` (default) | Catches tight loops fast. Will false-positive on diverse tool usage. |
| `2.0` | Good balance for agents using 3-4 tools regularly. |
| `2.7` | Permissive — only fires on genuine chaotic switching across 6+ tools. |

Validated in the [detection gauntlet](https://github.com/Shaurya-34/Sotis/blob/main/ExperimentLog/real_world_validation/test5_gauntlet_20260529_212356.txt): the default fired a false positive on healthy diverse work, raising to 2.7 eliminated it while still catching all 3 meltdown scenarios.

**Beyond the fixed threshold, three opt-in detectors** (full details in
[USAGE.md](https://github.com/Shaurya-34/Sotis/blob/main/USAGE.md)):

- **Adaptive threshold** — learns a per-agent baseline and triggers at `mean + 2σ`
  instead of a global number, so diverse agents stop false-positiving:
  `EntropyConfig(adaptive=True)`.
- **Token-spike corroboration** — flags a sudden jump in tokens/step (≥3× the
  rolling mean) as a *corroborating* signal, not a sole trigger:
  `SotisGuard(token_spike_factor=3.0)`.
- **Invariant-verified checkpoints** — rollback targets the most recent state that
  *passed* your invariant, not just the last snapshot (which may be the poisoned
  one): `checkpoint_invariant=python_imports_cleanly`. Ships a built-in Python
  invariant; supply any `Callable[[dict[str,str]], bool]`.

---

## Active Stabilization, Not Passive Tracing

Tools like LangSmith, Langfuse, and Helicone log what happened after your agent already spent $20 looping in production.

Sotis intervenes *during* execution. It intercepts spiraling tool calls, rolls back uncommitted file edits, distills conversation history, and redirects the model's reasoning loop — before the damage accumulates.

---

## Capabilities

| Capability | Description |
|---|---|
| **Meltdown Detection** | Sliding-window Shannon entropy + exact/semantic loop detection |
| **Adaptive Threshold** | Per-agent baseline (`mean + 2σ`) instead of a fixed global cutoff |
| **Token-Spike Signal** | Corroborates a meltdown when tokens/step suddenly jump |
| **Workspace Density Guard** | Detects infinite same-file edit cycles |
| **Verified-Checkpoint Reset** | Rolls back to a state proven good by your invariant, not just the last snapshot |
| **Transparent Reset** | Git-diff checkpointing + distilled context rebuild (up to ~86% token savings on longer runs) |
| **Graceful Degradation** | GDS scoring preserves partial progress across resets |
| **LangGraph Integration** | Native guard node — intercepts state, rolls back files |
| **LLM Support** | OpenAI, Anthropic, DeepSeek, Google, any OpenAI-compatible endpoint |
| **Observability** | Streamlit dashboard + structured JSON session logs |

---

## The Science

Sotis operationalizes the formal reliability framework from *["Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents"](https://arxiv.org/abs/2603.29231)* (arXiv:2603.29231, April 2026).

Four key findings from the paper that Sotis directly addresses:

**Meltdown Onset Point (MOP)** — the paper quantifies the transition from coherent planning to chaotic looping via sliding-window Shannon entropy. Sotis implements this as a live runtime monitor over a 5-step window, with either a fixed or per-agent adaptive threshold.

**Super-linear reliability decay** — agent success rates decay faster than mathematically expected because errors are positively correlated across steps. A confused agent stays confused. Sotis acts as a circuit breaker that resets the error correlation coefficient by starting fresh from a verified checkpoint.

**Episodic memory failures** — the paper demonstrates that naive memory scaffolds universally degrade long-horizon performance by accumulating context overhead. Sotis uses controlled checkpointed resets instead of continuous memory accumulation.

**Graceful Degradation Score (GDS)** — rather than binary pass/fail, Sotis scores partial task completion using weighted subtask graphs, preserving measured progress across reset boundaries.

---

## Performance

| Metric | Result |
|---|---|
| Entropy + loop detection latency | < 0.2ms per step |
| Context distillation token reduction | 67–86% on live runs of 8–16 steps (BPE cl100k_base, vs. the logged history). The briefing is a fixed ~250–440 tokens, so on runs shorter than ~5 steps a reset costs tokens rather than saving them |
| Test suite | 163 unit tests passing |
| Live recovery | Verified on circular-import and AST recursive-loop traps |
| Verified-checkpoint rollback (live) | On a real Groq Llama-3.3-70B run, rolled back to a verified-good checkpoint, not the corrupt snapshot ([log](https://github.com/Shaurya-34/Sotis/blob/main/ExperimentLog/circular%20import%20trap/run_groq_llama70b_verified_rollback_20260605.txt)) |
| Local model validation (mistral:latest via Ollama) | Caught a real TOOL_LOOP meltdown and corrected agent behavior after reset |
| Detection (6-scenario gauntlet) | Caught all 3 meltdown scenarios (0 misses); 1 false positive in 3 healthy scenarios at the default threshold, removed by raising it to 2.7 |
| Total API cost for full validation suite | < $0.01 (Groq free tier) |

Full empirical ledger: [`performance_metrics.txt`](https://github.com/Shaurya-34/Sotis/blob/main/performance_metrics.txt)

Real-world validation logs: [`ExperimentLog/real_world_validation/`](https://github.com/Shaurya-34/Sotis/tree/main/ExperimentLog/real_world_validation)

Local model run (Ollama — mistral:latest meltdown + gemma3:4b tool-binding notes): [`sotis_gemma_mistral_run_review.txt`](https://github.com/Shaurya-34/Sotis/blob/main/ExperimentLog/document%20handling%20loop/sotis_gemma_mistral_run_review.txt)

---

## Benchmarks

**Reliability decay in frontier LLM agents as task horizon grows** *(Khanal et al. 2026 — arXiv:2603.29231)*

![Reliability Decay](https://raw.githubusercontent.com/Shaurya-34/Sotis/main/charts/3_reliability_decay.png)

**Context distillation: token reduction after meltdown reset** *(measured with tiktoken BPE cl100k_base)*

![Token Reduction](https://raw.githubusercontent.com/Shaurya-34/Sotis/main/charts/5_token_reduction.png)

**Real agent experiments — meltdown intercepts and outcomes** *(Gemini 3.5 · Groq Llama 70B · Mistral local · OpenRouter Gemini)*

![Real Experiments](https://raw.githubusercontent.com/Shaurya-34/Sotis/main/charts/6_real_experiments.png)

---

## Project Structure

```
sotis/
  core/     # Entropy, loop detection, checkpoint, decomposition, GDS
  lib/      # ReAct runtime, LangGraph integration, LLM adapters
  obs/      # Streamlit dashboard + structured JSON logger
  bench/    # Benchmark harness and task generators
```

---

## License

MIT
