Metadata-Version: 2.5
Name: sibyld
Version: 1.4.3
Summary: Sibyl daemon - knowledge graph and task workflow server
Project-URL: Homepage, https://github.com/hyperb1iss/sibyl
Project-URL: Repository, https://github.com/hyperb1iss/sibyl
Project-URL: Issues, https://github.com/hyperb1iss/sibyl/issues
Author-email: Stefanie Jane <stef@hyperbliss.tech>
License-Expression: Apache-2.0
Keywords: developer-tools,graph-rag,knowledge-graph,persistent-memory,semantic-search,task-workflow
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.13
Classifier: Typing :: Typed
Requires-Python: >=3.13
Requires-Dist: anyio>=4.14.2
Requires-Dist: argon2-cffi>=25.1.0
Requires-Dist: arq>=0.26.3
Requires-Dist: authlib<1.8,>=1.7.2
Requires-Dist: crawl4ai>=0.9.2
Requires-Dist: email-validator>=2.3.0
Requires-Dist: fastapi>=0.141.1
Requires-Dist: google-genai>=2.23.0
Requires-Dist: itsdangerous<3,>=2.2
Requires-Dist: mcp<3,>=2.2.0
Requires-Dist: mistune>=3.3.4
Requires-Dist: pydantic-settings>=2.7
Requires-Dist: pyjwt[crypto]<3,>=2.13.0
Requires-Dist: pyyaml>=6.0
Requires-Dist: resend<3,>=2.47.0
Requires-Dist: rich>=13.0
Requires-Dist: sibyl-core[bedrock,embeddings,graph,graphrag,llm]==1.4.3
Requires-Dist: slowapi>=0.1.9
Requires-Dist: starlette>=1.4.1
Requires-Dist: surrealdb<3.0,>=2.0.0
Requires-Dist: tomli-w>=1.2.0
Requires-Dist: typer>=0.27.1
Requires-Dist: uvicorn[standard]>=0.52.4
Description-Content-Type: text/markdown

# Sibyl API Server

`sibyld` is the FastAPI + MCP SDK server behind Sibyl's knowledge graph, agent memory loop, task
workflow, search, synthesis, and real-time updates.

## Quick Reference

```bash
# Install the embedded daemon without the web UI
curl -fsSL https://raw.githubusercontent.com/hyperb1iss/sibyl/main/install.sh | sh -s -- --daemon

# Start server from the monorepo
moon run api:serve        # or: uv run sibyld serve

# Start worker (Redis coordination only)
moon run api:worker       # or: uv run sibyld worker

# Quality checks
moon run api:test         # Run tests
moon run api:lint         # Lint
moon run api:typecheck    # Type check
```

## What's Here

- **MCP Server:** thirteen tools for search, context packs, exploration, bounded traversal, capture,
  memory, synthesis, and management
- **REST API:** 31 routers covering entities, tasks, teams, projects, experience, memory, synthesis,
  sources, auth, settings, and admin
- **Auth System:** JWT sessions, GitHub OAuth, OIDC enterprise SSO (see `docs/admin/`), API keys
  with scopes, MCP OAuth clients, RBAC, SMTP-backed password reset
- **Background Jobs:** in-process local runtime or Redis-backed `arq` workers, including the nightly
  reflection dream-cycle
- **WebSocket:** Real-time updates for entities and tasks

## Architecture

```
Sibyl API (port 3334)
├── /api/*              → FastAPI REST endpoints
├── /api/openapi.json   → OpenAPI schema
├── /mcp                → MCP server (streamable-http, 13 tools)
├── /api/ws             → WebSocket for real-time updates
└── Lifespan            → Background jobs + coordination broker
```

## Key Directories

| Directory       | Purpose                                                                                                             |
| --------------- | ------------------------------------------------------------------------------------------------------------------- |
| `api/routes/`   | REST endpoints (31 routers: tasks, entities, teams, experience, memory, synthesis, crawler, ingestion, auth, admin) |
| `ai/`           | DB-backed LLM settings, model validation routes, runtime invalidation                                               |
| `auth/`         | JWT, sessions, API keys, RBAC, MCP OAuth clients                                                                    |
| `persistence/`  | SurrealDB-native runtimes for auth, content, graph, and backups                                                     |
| `crawler/`      | Documentation crawl and ingestion pipeline                                                                          |
| `ingestion/`    | Source import pipeline (mailbox and other adapters)                                                                 |
| `jobs/`         | Background jobs (reflection dream-cycle, crawl, backups)                                                            |
| `coordination/` | Local and Redis brokers for jobs, locks, and pub/sub                                                                |
| `email/`        | Transactional email delivery                                                                                        |
| `generator/`    | Synthetic test-data generation                                                                                      |

## Configuration

**Required:**

```bash
SIBYL_JWT_SECRET=...              # Auth (required in production; dev auto-generates)
SIBYL_ANTHROPIC_API_KEY=...       # Required when LLM provider=anthropic
# SIBYL_OPENAI_API_KEY=sk-...     # Required when LLM provider=openai
# SIBYL_GEMINI_API_KEY=...        # Required when LLM provider=gemini
# LLM provider=bedrock needs no key: AWS credentials plus AWS_REGION

# Embeddings: choose OpenAI, Gemini or Amazon Bedrock (Cohere Embed v4)
SIBYL_EMBEDDING_PROVIDER=openai   # openai | gemini | bedrock
SIBYL_OPENAI_API_KEY=sk-...       # Required when embedding provider=openai
# SIBYL_GEMINI_API_KEY=...        # Required when embedding provider=gemini
```

**Optional:**

```bash
SIBYL_STORE=surreal                   # the only supported value
SIBYL_COORDINATION_BACKEND=auto       # auto | local | redis
SIBYL_SURREAL_URL=ws://127.0.0.1:8000/rpc
SIBYL_SURREAL_USERNAME=root
SIBYL_SURREAL_PASSWORD=root
SIBYL_REDIS_HOST=127.0.0.1            # only needed for Redis coordination
SIBYL_REDIS_PORT=6381
SIBYL_LLM_PROVIDER=anthropic          # anthropic | bedrock | openai | gemini
SIBYL_LLM_MODEL=claude-haiku-4-5
SIBYL_LLM_CRAWLER_MODEL=claude-haiku-4-5
SIBYL_LLM_SYNTHESIS_MODEL=claude-sonnet-4-6
SIBYL_LLM_TEMPERATURE=0
# A shared timeout wins over every surface default, so the memory surface needs
# its own value; consolidation sends a whole cohort in one request.
SIBYL_LLM_TIMEOUT_SECONDS=60
SIBYL_LLM_MEMORY_TIMEOUT_SECONDS=600
SIBYL_EMBEDDING_MODEL=text-embedding-3-small
SIBYL_EMBEDDING_DIMENSIONS=1536
SIBYL_GRAPH_EMBEDDING_PROVIDER=openai
SIBYL_GRAPH_EMBEDDING_MODEL=text-embedding-3-small
SIBYL_GRAPH_EMBEDDING_DIMENSIONS=1024
# Amazon Bedrock (provider=bedrock) signs with the AWS credential chain
SIBYL_BEDROCK_REGION=us-west-2        # falls back to AWS_REGION
SIBYL_BEDROCK_INFERENCE_SCOPE=us      # us | global | regional
```

Gemini keys can also be supplied through `GEMINI_API_KEY` or `GOOGLE_API_KEY`. Changing embedding
provider, model, or dimensions changes vector spaces; re-crawl sources and rebuild graph indexes
before mixing old and new search results.

LLM settings are instance-wide. Environment variables win over database settings field by field;
env-backed fields return `409 LOCKED_BY_ENV` on write. Database settings are managed under:

```text
GET  /api/settings/ai/llm
PUT  /api/settings/ai/llm/{surface}
POST /api/settings/ai/llm/{surface}/test
POST /api/settings/ai/keys/{provider}/test
POST /api/settings/ai/models/{model_alias}/test
GET  /api/settings/ai/registry?kind=llm
```

Crawler extraction and synthesis generation call `sibyl_core.ai` rather than provider SDKs directly.
Custom model IDs are accepted as database settings with an `unverified_model` warning; validate them
with the model test endpoint before using them in production flows.

## CLI Commands

```bash
sibyld serve              # Start the HTTP server
sibyld serve -t stdio     # Start a stdio server (MCP subprocess mode)
sibyld worker             # Start the job worker (local mode exits cleanly)
sibyld up                 # Start data services + API
sibyld down               # Stop all services
sibyld db backup          # Back up the graph database
sibyld migrate import ... # Import a migration archive
sibyld generate realistic # Generate sample data
```

## Runtime Modes

For single-machine Surreal development, run `sibyld serve` or `sibyld up` with
`SIBYL_STORE=surreal`. The default `coordination_backend=auto` resolves to `local`, so background
jobs, pending state, locks, pub/sub, and schedules all stay in-process with no Redis requirement.

Redis remains available for distributed or multi-process dev. Set `SIBYL_COORDINATION_BACKEND=redis`
when you want the `arq` worker model, then run `sibyld worker` or `moon run api:worker` separately.

## Archive Checks

The authenticated archive API records an immutable check before migration applies data:

| Endpoint | Purpose |
| --- | --- |
| `POST /api/archive-imports/check` | Upload an archive and save its checked plan |
| `GET /api/archive-imports/{run_id}` | Read the caller's saved counts and digests |

Send the check as `multipart/form-data` with exactly two parts: `archive` contains the exported
archive bytes, and `options` contains a JSON object. The options use this shape (replace the owner
and actor placeholders with the source owner and authenticated destination actor):

```json
{
  "mappings": {
    "source_private_owner_id": "<source-owner>",
    "projects": {},
    "teams": {},
    "quarantine": {
      "memory_scope": "private",
      "scope_key": "<authenticated-actor>"
    }
  },
  "conflict_policy": "additive"
}
```

Project and team mappings bind source labels to current writable destinations. Eligible private
rows belong to the authenticated actor. Protected or retired rows and their dependents are
quarantined. Current membership and credential restrictions gate each destination.

An optional `Idempotency-Key` header binds the request to the actor and organization. Retrying the
same request preserves its original plan and credential ceiling. A changed request or originating
API key returns `409`. Invalid archives return `422`; requests exceeding configured budgets return
`413`. Successful responses contain the run identity, checked status, digests and per-kind counts.
The status endpoint accepts any currently valid same-actor session or API key with read access,
even after the originating key is revoked. Status reads do not grant permission to replay or apply.

The check stores an inert archive artifact and plan in one native transaction. Applying data and
the client migration command are subsequent protocol layers. Resource settings use the
`SIBYL_ARCHIVE_IMPORT_` prefix; the complete metadata request defaults to 4 MiB for HTTP and 64 MiB
for WebSocket or embedded transport. Set `SIBYL_ARCHIVE_IMPORT_METADATA_TRANSACTION_BYTES` only to a
budget supported by the configured native server.

## Key Patterns

**Multi-tenancy:** Every operation requires org context.

```python
manager = EntityManager(client, group_id=str(org.id))
```

Write concurrency: the SurrealDB driver serializes WebSocket operations per client. Clone graph
drivers per organization rather than sharing one driver across org scopes.

**SurrealDB access model:** The API server, worker, CLI, and schema bootstrap flows use configured
SurrealDB system credentials (`SIBYL_SURREAL_USERNAME` / `SIBYL_SURREAL_PASSWORD`) so they can run
migrations, background jobs, and admin workflows. Route code must keep explicit org, project, and
principal predicates because system users sit above table-level permissions.

Auth and content schema migrations also define table permissions for future scoped Surreal record
users. Tenant-owned tables accept rows where `organization_id` (or `organizations.uuid`) matches
either `$token.org` from an external JWT access method or `$auth.organization_id` from a Surreal
record session. Secret-heavy and global tables, such as API keys, sessions, OAuth tokens, system
settings, and telemetry rollups, remain `PERMISSIONS NONE` for direct scoped DB access.

**Request context:** Auth middleware injects user and org.

```python
from sibyl.auth.dependencies import get_current_user, get_current_organization
```

## Dependencies

Depends on `sibyl-core` for models, graph client, AI substrate, and tool implementations.
