Metadata-Version: 2.5
Name: cowork-server
Version: 0.26.11.10.1rc2
Summary: FastAPI backend for the MindsHub Cowork desktop app
Project-URL: Homepage, https://github.com/mindsdb/cowork-server
Project-URL: Repository, https://github.com/mindsdb/cowork-server
Project-URL: Issues, https://github.com/mindsdb/cowork-server/issues
Author-email: MindsDB Inc <admin@mindsdb.com>
License-Expression: AGPL-3.0-only
License-File: LICENSE
Classifier: Framework :: FastAPI
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Requires-Python: <3.14,>=3.12
Requires-Dist: aiohttp>=3.14
Requires-Dist: alembic>=1.17.0
Requires-Dist: anton-agent<3,>=2.26.11.9.1
Requires-Dist: anyio>=4.14.2
Requires-Dist: beautifulsoup4==4.15.0
Requires-Dist: cryptography<49; sys_platform == 'darwin' and platform_machine == 'x86_64'
Requires-Dist: cryptography<51,>=48.0.0
Requires-Dist: fastapi>=0.136.1
Requires-Dist: google-re2>=1.1
Requires-Dist: httpx>=0.27
Requires-Dist: lxml==6.1.3
Requires-Dist: markdown==3.11.2
Requires-Dist: openai-codex==0.147.0
Requires-Dist: pillow==13.0.0
Requires-Dist: psutil>=7.0
Requires-Dist: psycopg2-binary>=2.9
Requires-Dist: psycopg[binary]>=3.2
Requires-Dist: pydantic-settings>=2.14.2
Requires-Dist: pydantic>=2.12.5
Requires-Dist: pyjwt>=2.15.0
Requires-Dist: pypdf==6.21.1
Requires-Dist: python-docx==1.2.0
Requires-Dist: python-dotenv>=1.2.2
Requires-Dist: python-multipart>=0.0.28
Requires-Dist: redis<6,>=5.3.1
Requires-Dist: reportlab==5.0.1
Requires-Dist: sqlmodel>=0.0.38
Requires-Dist: structlog==26.1.0
Requires-Dist: tinycss2==1.5.1
Requires-Dist: tzlocal>=5.0
Requires-Dist: uvicorn>=0.47.0
Requires-Dist: xhtml2pdf==0.2.26
Provides-Extra: channels
Requires-Dist: aiohttp>=3.9.0; extra == 'channels'
Requires-Dist: discord-py>=2.3.0; extra == 'channels'
Requires-Dist: slack-sdk>=3.27.0; extra == 'channels'
Description-Content-Type: text/markdown

# Cowork Server

FastAPI backend for [MindsHub Cowork](https://github.com/mindsdb/cowork). Manages projects, conversations, files, scheduling, memory, and agent orchestration with a SQLite-backed data layer.

This repo is the **Python backend**. The **frontend** (Electron shell + React SPA) lives in a separate repo: [`mindsdb/cowork`](https://github.com/mindsdb/cowork). They are developed and released independently. At runtime, the frontend spawns `cowork-server` as a local sidecar and communicates over HTTP (`127.0.0.1:26866`).

## Quick Start

Requires Python 3.12+ and [uv](https://docs.astral.sh/uv/).

```sh
# Install and run

uv tool install cowork-server
cowork-server
```

The server starts on `http://127.0.0.1:26866`. Confirm with:

```sh
curl http://127.0.0.1:26866/api/v1/health/
```

## Reviewed release dependencies

`uv.lock` is the runtime selection for this repository. CI, Docker and release smoke tests refuse a stale lock. `cowork/release-requirements.txt` is its generated runtime export, with environment markers and artifact hashes. Every wheel and source distribution carries that exact file; publishing checks both artifacts against the lock.

The release workflow selects Anton from `uv.lock`. Publishing a newer Anton alone does not change this server release. To move a dependency deliberately:

```sh
uv lock --upgrade-package anton-agent
python scripts/release_contract.py
uv sync --locked --group dev
bash scripts/check_wheels_available.sh
make test/unit
```

Review both the lock and export diffs, run the release checks, and publish a new server release. Change a declared compatibility constraint first when the intended version is outside it. Use the same process for any other dependency. Do not run a general upgrade while preparing a source-only release.

Cowork's release installer consumes the contract from the selected wheel or Git commit. Direct `pip install` and `uv tool install cowork-server` still resolve the public library compatibility ranges; they do not automatically consume this export. For a standalone server with this repository's reviewed selection, check out a reviewed release tag and run `uv sync --locked --no-dev`, then `uv run --locked --no-dev cowork-server`.

The eight platform/Python wheel checks use release constraints. When the lock selects an Anton release candidate, the required wheel gate waits for that exact version to reach PyPI's Simple index before resolving it; the publish runner repeats the wait before building. The weekly `--floating` check separately probes the public compatibility ranges and never rewrites the release lock. Build backends, the Python interpreter and operating-system packages are separate inputs; this policy fixes the application dependency selection, not every byte of an image.

## Development

```sh
# Run from source (auto-manages virtualenv + deps)
uv run --locked cowork-server
```

When running alongside the Electron app in dev mode, the app spawns the server automatically — no manual start needed. The Electron app looks for a sibling `cowork-server/` directory by convention (override with `COWORK_SERVER_DIR`).

cowork-server imports anton's artifact folder constants (`anton.core.artifacts.internal_files`) at startup, with no fallback. An anton without those constants (`NON_CONTENT_NAMES` and `REVISIONS_DIRNAME`, which [mindsdb/anton#521](https://github.com/mindsdb/anton/pull/521) added, first in 2.26.10.2.1rc2 and first in a stable release in 2.26.10.4.1) makes the server fail to start with `ImportError`.

### Dev setup helper

```sh
uv run --locked cowork-dev-setup
```

Initializes the database and validates configuration.

### Testing

```sh
uv run --locked pytest
```

Tests use an isolated in-memory database and temporary directories — no side effects on your local `~/.cowork/` data.

Every unit test runs under a guard for blocking work where it stops other requests (`tests/_blocking_guard.py`). A test fails when cowork code checks out a pooled database connection on the event loop. It also fails when, in org tenancy, cowork code reads, lists, writes, moves, deletes or closes a file under `COWORK_HOME` on the event loop, or on a worker thread that `run_efs` didn't start, such as the file a returned Starlette `FileResponse` opens on anyio's threads. Move such work into `run_db` or `run_efs` (`cowork/db/units.py`). The sites the code has today are listed in the guard with their reasons, each by the function that blocks and the cowork call nearest the work, past the shared file helpers in `cowork/common/paths.py`, so new work below a listed function fails too. A listed site that no test reaches fails a full run, so remove its entry when you move the work. A run that can't tell, such as one test file or `--setup-only`, says in one line that it skipped that check.

The guard sees only what the tests drive. File store work counts only in a test that sets org tenancy in the environment, as the `org_deployment` fixture does, or on the settings the app already holds. A test that patches one module's own check, such as its `_org_mode`, switches that module alone, so the guard doesn't see its file work. The guard also sees only the calls it wraps: the `os` calls in its `_PATH_PARAMETERS`, `open` and `io.open`. A read on a file object opened earlier, `fcntl.flock`, `os.readv`, `os.writev` and `os.sendfile` go unseen. On a worker thread it watches anyio's threads and the event loop's default executor, not a plain `threading.Thread` or an executor of your own.

A few tests need a real Redis or PostgreSQL, because hosted Cowork runs on both: two replicas reading one Redis (`tests/integration/test_two_replicas.py`), the Redis clients through a killed connection, a demoted primary, a connection closed during its handshake and a lost reply (`tests/integration/test_redis_failover.py`), and the migration and launch-lock contracts on PostgreSQL. They carry the `redis_service` or `postgres_service` marker and skip when their service is absent. `make test/services` runs only them. Point it at a disposable Redis and database. The failover tests empty the database `COWORK_TEST_REDIS_URL` names and run `REPLICAOF`, `CLIENT KILL` and `ACL SETUSER` on that Redis, putting the server back afterwards, so they run only when the URL is set and skip otherwise, even with a Redis on `localhost:6379`. The migration test downgrades the database to an empty schema:

```sh
COWORK_TEST_REDIS_URL=redis://localhost:6379/1 \
COWORK_TEST_POSTGRES_URI=postgresql://postgres@localhost:5432/cowork_test \
make test/services
```

On every pull request and every push to `staging`, `.github/workflows/tests-services.yml` runs them against Redis 8.2.3 and PostgreSQL 16.13 service containers with `COWORK_REQUIRE_SERVICES=true`, so a missing service fails the job instead of skipping. The desktop wheel release doesn't wait on that job.

A run that requires every prerequisite (`COWORK_REQUIRE_INTEGRATION=true` or `COWORK_REQUIRE_SERVICES=true`) fails when a test skips, or is left out with `--deselect`, unless `tests/skip_allowlist.yaml` names it with a reason (`tests/skip_guard.py`). Any run that sets either flag lists what it left out at the end of its report and in the GitHub job summary. The allowlist names two tests: the remote-worker disconnect check, an operator-run proof described under [Streaming & scheduling](#streaming--scheduling), and the post-deploy cancel check, which skips when the model ends its turn before the turn can be seen running, so there is nothing to cancel.

Post-deploy integration runs use the target cluster's self-hosted runner. Dev and staging obtain the fixed `cowork` test suite through MindsHub's cluster-only service URL. Production must not mutate that shared `@emailsink.dev` identity while the fixed password remains committed. The production test path therefore accepts a dedicated `COWORK_TEST_API_KEY` secret, a reviewed `COWORK_TEST_USER_EMAIL` variable on the non-staff `@mindshub.ai` domain, and the immutable dedicated organization id in `COWORK_TEST_ORG_ID`. The suite resolves and matches that principal and organization through production MindsHub, and refuses an employee-classified or MindsHub-admin identity before testing. A missing or mismatched identity fails the required prod run instead of falling back to provisioning or reporting skipped tests. Standing-identity mode also requires the test target to be exactly `https://cowork.mindshub.ai` before the first network call, so a changed environment file or workflow cannot send the production key to another origin.

Do not store or use `COWORK_TEST_API_KEY` yet, or configure its paired email and
organization id for a production run. The live `prod` GitHub Environment has no
protection rules or deployment-branch policy. Although `publish.yml` refuses to
enter its production build/deploy job from a non-main ref, a manually selected
branch runs that branch's workflow text and can remove the check. Workflow code
is therefore defense in depth, not the authority that protects an Environment
secret.

Before `COWORK_TEST_API_KEY` is stored or used, or any production evidence run
starts, the `prod` Environment must have a nonempty required-reviewer rule with
`prevent_self_review: true`, `can_admins_bypass: false`, and a selected-branches
deployment policy whose only entry is the `main` branch, created as an exact
Branch rule (no tag, wildcard, or second branch rule). Verify the live settings
without reading any secret value:

```sh
gh api repos/mindsdb/cowork-server/environments/prod \
  --jq '
    [.protection_rules[]?
      | select(.type == "required_reviewers")
      | {
          prevent_self_review,
          reviewers: [.reviewers[]?
            | {type, name: (.reviewer.login // .reviewer.slug)}]
        }] as $required_reviewers
    | {
        can_admins_bypass,
        required_reviewers: $required_reviewers,
        deployment_branch_policy
      }
  '
gh api 'repos/mindsdb/cowork-server/environments/prod/deployment-branch-policies?per_page=100' \
  --jq '[.branch_policies[] | {name, type}]'
```

The first command must show exactly one required-reviewer rule with at least one
named reviewer and `prevent_self_review: true`, plus `can_admins_bypass: false`,
`protected_branches: false`, and `custom_branch_policies: true`. The second must
print exactly `[{"name":"main","type":"branch"}]`. Only after both checks pass
may an operator store `COWORK_TEST_API_KEY`, `COWORK_TEST_USER_EMAIL`, and
`COWORK_TEST_ORG_ID` on that Environment. Evidence must come from a fresh
main-branch run started after the
protection was active, with `run_attempt: 1`; an eligible reviewer other than the
run's `actor` and `triggering_actor` must approve its `prod` Environment gate. A
rerun of an attempt that began before protection does not count. Keeping the
values at Environment scope prevents non-prod jobs from receiving them, but that
scope is safe only when these Environment controls are active.

#### Nightly production read-only smoke

The production nightly reads production on `mdb-prod` at `43 7 * * *` once its
schedule is live. It is held today; the Environment prerequisites below say what
turns it on. It uses the same guarded standing identity, but selects only
`tests/integration/test_production_read_only.py`. That selection is GET-only:
it reads health, conversations, schedules, files, and pins. It never
provisions an identity and does not create conversations, schedules, files,
artifacts, or model turns. The broad integration target excludes the
`production_read_only` marker, so release and staging callers cannot include
this production-only selection by accident.

Failures and the next recovery use the shared engineering-channel notifier.
The nightly is an alert, not a release gate. Review any new endpoint in
`READ_ONLY_ENDPOINTS` together with
`tests/test_production_read_only_workflow.py`, which rejects mutating HTTP calls
and pins the complete request list.

The unattended nightly cannot reference the existing `prod` Environment.
That Environment must retain its required-reviewer gate for production deploys,
and GitHub pauses every job that references such an Environment until a reviewer
approves it. The monitor instead uses a dedicated `prod-read-only` Environment
with unique `COWORK_PROD_READ_ONLY_API_KEY`,
`COWORK_PROD_READ_ONLY_USER_EMAIL`, and
`COWORK_PROD_READ_ONLY_ORG_ID` inputs. Unique names prevent a missing
Environment value from falling back to a repository or organization credential.

Before promoting this workflow to `main`, an operator must create
`prod-read-only` with no required-reviewer or wait-timer rule and a custom
deployment policy whose only entry is the exact `main` branch. The repository's
`main` protection must continue to require a pull-request approval, resolve
review conversations, and apply to administrators. These controls let the
scheduled job start without weakening the separately protected `prod`
Environment:

```bash
gh api --method PUT repos/mindsdb/cowork-server/environments/prod-read-only \
  --input - <<'JSON'
{
  "wait_timer": 0,
  "prevent_self_review": false,
  "reviewers": [],
  "deployment_branch_policy": {
    "protected_branches": false,
    "custom_branch_policies": true
  }
}
JSON
gh api --method POST \
  repos/mindsdb/cowork-server/environments/prod-read-only/deployment-branch-policies \
  -f name=main -f type=branch
```

Only after those controls exist may the operator copy the already reviewed
standing identity into the dedicated names. The commands prompt for the secret
and variable values and do not print them:

```bash
gh secret set COWORK_PROD_READ_ONLY_API_KEY \
  --repo mindsdb/cowork-server --env prod-read-only
gh variable set COWORK_PROD_READ_ONLY_USER_EMAIL \
  --repo mindsdb/cowork-server --env prod-read-only
gh variable set COWORK_PROD_READ_ONLY_ORG_ID \
  --repo mindsdb/cowork-server --env prod-read-only
```

Verify policy and names without reading credential values:

```bash
gh api repos/mindsdb/cowork-server/environments/prod-read-only \
  --jq '{protection_rules, deployment_branch_policy}'
gh api \
  'repos/mindsdb/cowork-server/environments/prod-read-only/deployment-branch-policies?per_page=100' \
  --jq '[.branch_policies[] | {name, type}]'
gh api repos/mindsdb/cowork-server/branches/main/protection \
  --jq '{required_pull_request_reviews, enforce_admins, required_conversation_resolution}'
gh secret list --repo mindsdb/cowork-server --env prod-read-only \
  | rg '^COWORK_PROD_READ_ONLY_API_KEY\b'
gh variable list --repo mindsdb/cowork-server --env prod-read-only \
  | rg '^COWORK_PROD_READ_ONLY_(USER_EMAIL|ORG_ID)\b'
```

The Environment response must have no `required_reviewers` or `wait_timer`
entry and must enable only custom branch policies. The branch-policy response
must be exactly `[{"name":"main","type":"branch"}]`. The branch-protection
response must show at least one required approval, administrator enforcement,
and required conversation resolution. Do not dispatch until every check passes
and the cowork-server#472 prerequisite has landed.

The nightly schedule is held out of the workflow until those checks pass.
GitHub arms a `schedule:` trigger as soon as the file reaches the default
branch, so shipping one before the Environment exists does not buy a monitor, it
buys a nightly page that reports a missing credential in the same shape as a
production outage. `workflow_dispatch` is the only trigger until an operator has
provisioned `prod-read-only` and watched one dispatched run go green. Restore the
`schedule:` block with `- cron: "43 7 * * *"` and delete this paragraph in the
same change; a contract test fails if only one of the two happens.

### Logging

Set `LOG_LEVEL` (default `WARNING`) to control verbosity. Enable file logging with `ENABLE_FILE_LOGGING=true` (writes to `LOG_DIR`, defaults to `~/.cowork/logs/`).

Database exception records omit SQL, parameter values, driver details and tracebacks, in the owned console and file handlers and in everything Uvicorn logs. `setup_logging()` attaches the database and provider filters to the `uvicorn`, `uvicorn.error`, `uvicorn.access` and `uvicorn.asgi` loggers, and to any handler a launcher already installed on them, so the policy holds under every launcher: the `cowork-server` CLI, `python -m uvicorn cowork.server:app`, or `uvicorn spa_wrapper:app` with Uvicorn's default log config. Uvicorn's levels stay as the launcher set them, so lifecycle lines such as `Finished server process` still print. The shared filter replaces each such record with `Database operation failed: error_type=<class> sqlstate=<code or unknown> site=<module>.<function>:<line>`, keeping only valid driver SQLSTATE codes. `site` names the logging call. When that call only relays the error, as Uvicorn's request runner and the server's lifespan guard do, `site` names the deepest Cowork frame the error passed through outside the session wrappers (`cowork.db.scoped`, `cowork.db.session` and `cowork.db.units`), such as the service line that ran a route's failing statement. Session logs retain error types and client HTTP status, including the 503 for a full pool. A 5xx `HTTPException` logs its status. Its own detail is not logged, but its cause's class, message and traceback are, unless the cause chain holds a database or provider error, which the filters replace. Any other error, including one that carries a `status_code` without being an `HTTPException`, logs only its type in the session line, then reaches Uvicorn, which logs it through the same filters. Exceptions and HTTP errors returned to callers stay unchanged.

A database error while the server starts or stops (a migration, the seed rows, or Postgres not yet reachable after a reboot) logs that safe line through the owned handlers. Uvicorn then prints only `RuntimeError: Database operation failed during server startup` (or `shutdown`), because Starlette hands it the failure as plain traceback text, which no filter can read. Other startup errors keep their traceback.

A call site passes the ids that locate a failure as `extra=log_context(...)` from `cowork.common.logger`, which sets record attributes: `request_id`, `project_id`, `schedule_id`, `conversation_id`, and `artifact_slug` for one artifact or `artifact_slugs` for several. The filters never change them, and the console and file formatters render them before the message, for example `[Req:<id>][Project:<id>][Schedule:<id>][Conversation:<id>][Artifact:'<slug>']` or `[Artifacts:'<slug>', '<slug>']`. Artifact slugs are folder names, so they render quoted. A site that carries only a request id passes `extra={"request_id": ...}` directly.

OpenAI and Anthropic SDK errors (any `OpenAIError` or `AnthropicError`) and Anton's typed provider errors are found, in the same handlers and Uvicorn loggers, through each exception's cause, then its context (even a suppressed one), then its exception group members. The record becomes `Provider operation failed: error_type=<the record's exception class> provider_error=<the first provider error's class> status=<that provider error's HTTP status or unknown>`; a wrapper's own status is never used. Cowork's request ID (`[Req:<id>]`) stays on the line; the provider's own request ID is not logged. Provider messages, bodies, requests and tracebacks are omitted. Routing failure warnings also omit body-derived type, code and parameter fields. Original exceptions, probe verdicts and client HTTP/SSE errors stay unchanged.

Uvicorn's access line keeps each request's method, path and status, and `setup_logging()` replaces its query with `?[redacted]`, because a query can carry an OAuth callback's code or a webhook's verify token.

SQLAlchemy query and connection diagnostics stay at `WARNING`, even when application logging is set to `DEBUG`. SDK request diagnostics and HTTP client logging stay at `ERROR`, including Anthropic, `httpx2` and the whole `httpcore` and `httpcore2` namespaces, because they can include request bodies, URLs or response headers such as `set-cookie`.

## Releasing

Releases are automatic on merge; there is no version to bump by hand (the
package version comes from the tag).

- Push to `main`: [`publish.yml`](.github/workflows/publish.yml) runs the unit
  tests, cuts a CalVer tag and GitHub release (`v0.<yy>.<m>.<d>.<seq>`), then
  builds and publishes to [PyPI](https://pypi.org/project/cowork-server/) via
  OIDC trusted publishing.
- Push to `staging`:
  [`publish-staging.yml`](.github/workflows/publish-staging.yml) does the same on
  the rc pre-release stream (`v0.<yy>.<m>.<d>.<seq>rc<n>`, GitHub and PyPI
  pre-release). When PyPI's newest `anton-agent` upload is an rc, it pins that rc
  into the wheel so the pair installs exactly. When the newest upload is a
  stable release, the wheel keeps the `anton-agent` range from `pyproject.toml`.

Both take their version, tag, and release from the shared `calver-release.yml`
reusable in [mindsdb/github-actions](https://github.com/mindsdb/github-actions)
(`prerelease: true` selects the rc stream). The publish jobs stay in these two
workflows: PyPI trusted publishing matches the OIDC claim on the workflow
filename and does not support reusable workflows.

### Nightly staging integration

The cowork-server maintainers own the deployed integration signal and its
staging prerequisites. [`nightly-staging-integration.yml`](.github/workflows/nightly-staging-integration.yml)
runs every day at 06:41 UTC and can also be dispatched by hand. It calls the
same `tests-integration.yml` reusable workflow as a deployment on `mdb-dev`,
then reports a failure or the first recovery through the shared
engineering-channel notifier. It is a standalone monitor and never gates a
publish, release, or deployment.

**The nightly runs staging's copy of the tests, not main's.** GitHub starts every scheduled run on the default branch, so the checkout would otherwise take `main`. `main` trails `staging` by every commit waiting for the weekly release. Any behavior change among those commits fails `main`'s tests against staging's pods, even though staging works as intended. So the job passes `ref: staging`, and `tests-integration.yml` hands that input to `actions/checkout`. The deploy callers leave `ref` empty and keep testing the commit they just deployed. [`test_nightly_integration_workflow.py`](tests/test_nightly_integration_workflow.py) fails if a scheduled caller's `ref` differs from its `deploy-env`.

### Code scanning

[`codeql.yml`](.github/workflows/codeql.yml) runs CodeQL on every pull request and push to `staging` and `main`, and weekly on `main`. Each event's pipeline calls it as a job (`dev-build-deploy.yml`, `publish-staging.yml`, `publish.yml`), so its results sit in that event's one run. It scans Python, JavaScript/TypeScript and the workflows themselves with the default query suite, and needs no build. It replaces CodeQL's default setup, which scans only the default branch and branches under classic branch protection. `staging` is protected by rulesets, so default setup never scanned a pull request onto it. Default setup must stay off in the repository's code scanning settings, since GitHub refuses a workflow's results while it is on.

The suite may create and delete test conversations, schedules, files, and agent
turns in staging. The fixed test tenant is reserved for the `cowork` suite, and
the workflow sets `COWORK_REQUIRE_INTEGRATION=true` for staging so a missing
target, identity source, replica, or port-forward fails instead of becoming a
green skip.

In the packaged Electron app, a background updater checks PyPI on every launch and upgrades automatically (with rollback on failure). See [`server-updater.ts`](https://github.com/mindsdb/cowork/blob/main/src/main/server-updater.ts) in the frontend repo.

## Architecture

```
cowork/
  api/v1/endpoints/   # FastAPI route handlers
  services/           # Business logic
  models/             # SQLModel / DB models
  schemas/            # Pydantic request/response schemas
  db/                 # Database session and migrations
  common/             # Shared utilities, settings
  harnesses/          # Agent adapters (Anton)
```

The server is designed to be **agent-agnostic** — core features (projects, conversations, files) are shared across agents, while agent-specific behavior lives in harness adapters. See [docs/DESIGN.md](docs/DESIGN.md) for the full architectural rationale.

### Server modes and terms

The server runs in one of three modes, set by two settings ([Configuration](#configuration)):

| Mode | Settings | What it is |
|------|----------|------------|
| MindsHub mode | `COWORK_TENANCY_MODE=org` | The cloud, multi-tenant server. Every request is an organization and a person, from the identity headers the gateway sets, and every resource is kept per organization. Sharing a skill or an artifact with people, teams or everyone at the organization works only here. |
| Single-tenant mode | `COWORK_TENANCY_MODE=local` and `COWORK_TRUST_IDENTITY_HEADERS=true` | One organization with many people, each with their own space (their partition, `cowork.principal.personal_partition_id`), as the self-hosted VM runs. A person shares one of their own skills with everyone, read-only ([Sharing one skill with everyone](#sharing-one-skill-with-everyone)). |
| Single-user mode | `COWORK_TENANCY_MODE=local` alone | The desktop app: one person, no identity headers. |

The code names MindsHub mode `TenantScope.org_mode`, and the app learns it from `/health`'s `org_mode`.

- **MindsHub**: the MindsDB platform service this server asks about organizations, people, teams, permissions and artifact rules. Its public reads go out with the caller's own credential (`cowork/services/mindshub_client.py`); its internal artifact-access routes take a shared secret (`cowork/services/artifact_access.py`).
- **MindsHub UI**: the web app's build flag `VITE_PLATFORM_UI`, which shows MindsHub's model, provider and workspace controls. It is set in the app's build, not here, and it is separate from the server's mode: the cloud build turns both on, and neither implies the other.
- **Audience**: who a skill or an artifact is shared with, under MindsHub's artifact rule names: `org_allowed` (Everyone at ‹org›), `allowed_team_ids` (Specific teams), `allowed_user_ids` (Specific people in the organization) and, for an artifact only, `allowed_emails` (anyone else, by address). Nothing set is "Only me".
- **Default team**: the organization's team MindsHub marks `is_default`, whose members are everyone in it. Naming it is stored as `org_allowed`, never as a team.
- **Always on**: a skill shared with everyone or with a team that the agent applies on every task in the turns of the organization or the team, instead of recalling it when a task needs it. It stays on demand for people a skill is shared with personally ([Always-on skills](#always-on-skills)).
- **Off for me**: a person's own switch for one skill (`PUT /api/v1/skills/{skill_id}/me`), which leaves it out of their turns only. A skill's `enabled` turns it off for everyone who reads it; a turn uses a skill only when both are on.

### Harness system

A **harness** adapts an external agent library (Anton today) to the cowork-server interface. All harnesses implement the `HarnessProvider` protocol (`harnesses/base.py`), which exposes streaming responses, skill sync, and memory operations. The active harness is selected via the `harness` user setting. To add a new agent, implement the protocol and register it with the `@register` decorator.

**Each turn's scratchpads exit when the turn ends.** The Anton harness builds a fresh `ChatSession` for every turn, and each scratchpad that session starts is a child process of the server. When the turn completes, fails or is stopped, `close_session_scratchpads` (`cowork/common/chat_session.py`) kills those processes. A Stop or an opted-in client disconnect also interrupts a running cell and ends its turn; the harness does not start another model step. The next turn starts a new process, which reloads the variables the old one saved to the conversation's namespace snapshot after every cell. Only the scratchpads close: full-stack backends the agent launched keep running. Anything a cell starts in the scratchpad's process group, such as a default `subprocess.Popen` child or a thread, dies with it, so a long-lived service belongs in `launch_backend`. A child started with `start_new_session=True` or `setsid` escapes: it keeps running and holds two of the server's pipe descriptors until it exits. The connector probe closes its scratchpads the same way, and shutdown waits up to 5 seconds for closes still running. A scratchpad runs as `python <tmp>/anton_scratchpad_<random>.py`, so `pgrep -f anton_scratchpad_` counts the live ones; it should drop to 0 once no turn is running.

### Streaming & scheduling

Agent responses stream to clients via **Server-Sent Events** (SSE) on `POST /responses/`. The server tracks in-flight streams and supports cancellation (`/responses/cancel`) and late-join tailing (`/responses/tail`).

A streaming API question can set `"cancel_on_disconnect": true` to stop its own run when the
originating POST connection closes before the answer finishes. The option is a strict boolean,
defaults to `false`, and requires `"stream": true`. Browser questions leave it off, so a reload
still reconnects to the detached answer. Closing a `/responses/tail` observer never cancels a run.
If a client leaves during routing or setup, that request's cleanup prevents it from leaving a
live producer behind. Completion and server shutdown retain their existing outcomes.

A question can set `"memory": "off"` to read no learned memory and learn nothing, while other questions keep memory, for example a scored evaluation run on a deployment whose testers have it on. The turn runs as every turn does with `MEMORY_ENABLED=false`: two empty memory tiers in its scratch folder, no background memory passes, and the operator's managed rules still read. A later turn sent without the key in the same conversation learns from the whole conversation, its earlier memory-off turns included, so send the key on every turn of the conversation. The field takes `"on"` (the default) or `"off"`. A deployment that doesn't trust identity headers, such as the desktop app, still reads stored memory with memory off, so it refuses the question with 501.

Each answer also gets one `response.memory_read` event, before its `response.figure_sources`, if any, and its `response.completed`. `read` says whether the turn read learned memory, the person's own or the shared store, and `entries` counts the rules, lessons, profile lines and scratchpad tips anton put in front of the model (`ChatSession.memory_read`). The operator's managed rules aren't counted. A turn on two empty tiers, as every question with `"memory": "off"` runs, reads nobody's learned memory, so it gets `"read": false` and `"entries": 0` whether or not the deployment sets `COWORK_DEPLOYMENT_KIND`. So does a direct answer from the routing gate, which runs no agent. The harness sends the event with the first event anton streams after building the turn's prompt, which anton does before its first model call, or as the turn fails, is stopped or ends, if anton streams nothing after it. So a turn that fails after building its prompt carries it, and so does a turn the user stops after that. The event is saved with the answer's events. An answer that ended before anton built its prompt gets none, and so do a turn on the remote worker and a turn on an anton without `ChatSession.memory_read`.

The root of `response.created` carries a `run_id` UUID for that execution. An API client or proxy
can send `{"conversation_id": "...", "expected_run_id": "...", "reason": "client_gone"}` to
`POST /api/v1/responses/cancel`. The guard must match the authorized run; a stale or foreign ID
returns 404 and cannot stop a successor. Turn numbers can repeat, so they are not cancellation
guards. Existing Stop calls without `expected_run_id` still cancel the conversation's current
authorized run. A 200 response reports whether cancellation was requested; the stream's terminal
event and persisted answer determine the final outcome. Database or worker failures can prevent
successful finalization.

A proxy that keeps reading upstream after its client leaves must relay the disconnect explicitly.
The self-hosted recording proxy consumes the option, keeps its upstream connection detached for
audit records, and sends an exact-run cancel. Its final record comes from the answer's events.
See [Disconnect ownership and exact-run cancellation](docs/DESIGN.md#disconnect-ownership-and-exact-run-cancellation).

`tests/test_disconnect_transport.py` covers real sockets and a local executing scratchpad.
For the deployed remote-worker check, use the standard post-deploy identity and `COWORK_BASE_URL`,
then set `COWORK_RUN_DISCONNECT_WORKER_TEST=true` and `COWORK_DISCONNECT_NAMESPACE` before running
`uv run pytest -q tests/integration/test_disconnect_worker.py`.
`COWORK_DISCONNECT_CONTEXT` optionally selects the Kubernetes context. The runner needs read
access to pods and permission to exec the observer in the worker container. The check starts
its own bounded cell and observes process/container identity and heartbeat cessation; a
cancelled SSE event alone cannot pass it. It skips unless explicitly enabled.

`/responses/cancel` takes an optional `reason`. The UI sends `"stalled"` when it heard nothing for its idle window, and the turn then saves as a `response.failed` with code `stalled`, so a reload shows the stall card. Without a reason, or with any other value, the cancel is a Stop and keeps the partial answer with no error row. While anton waits on a model call that sends no output, the server (in cloud, the anton pod) writes a `model_wait` progress frame every 20 to 25 seconds so the turn's idle bounds do not end it. A model call that sends no output for 600 seconds ends the turn with code `model_timeout`; anton's `ANTON_MODEL_CALL_IDLE_TIMEOUT_S` sets that limit, and 0 or less turns it off. [Turn liveness and idle bounds](docs/DESIGN.md#turn-liveness-and-idle-bounds) lists every bound a quiet turn meets and what happens when each one fires.

`POST /responses/` reads what it needs (the turn's settings, its attachments, the conversation, whether an always-on skill reaches the turn and the routing gate's history) as one short database unit in a worker thread (`run_db` in `cowork/db/units.py`). A turn that carries attachments, or names no project by id and so may start its conversation in the general project, whose folder is checked, runs that unit on the file store's threads instead (`run_efs_unit`). Deciding whether an always-on skill reaches the turn can take its skill file, which that unit doesn't read: when one of the organization's skills is always on, the file is read in a unit of its own on the file store's threads (`run_efs_unit`), and when none reaches the turn, the gate's history is read in one more database unit after it. The request holds no pooled connection while the gate or the stream runs, and a wait for one never stalls the event loop. When no connection frees within `POOL_TIMEOUT` (5 s by default), the request is refused with `503`, a `Retry-After` header and `{"detail": "Cowork is busy. Try again in about 5 seconds.", "code": "server_busy"}`. An answer that meets the same wait after its stream started ends with one `response.failed` frame with code `server_busy` and the `retry_after` and `retry_at` fields `rate_limited` carries. The question sent again after that frame gets a stream of its own.

In a turn this process runs (`COWORK_TURN_BACKEND=inprocess`), scratchpad call-end events retain their complete input in the stream and saved conversation, including code longer than 65,536 characters. The next turn's write check can therefore read bindings from a long earlier cell. Results and every other tool's input retain their 65,536-character cap. cowork-server adds no cap of its own to a remote pod's scratchpad call input: the pod's own 65,536-character cap (`MAX_STEP_CHARS` in anton's `anton/cloud_turn/__main__.py`) is the only one, so a remote turn's saved call-end event can hold a cut call. Remote turns replay no cells and refuse `ANTON_WRITE_APPROVAL`, so no write check reads it. Replay preserves an explicit cell error; when a truncated result loses that field, only its saved `cell_status="ok"` establishes success, and the cell otherwise says it failed, timed out or has an unknown outcome. The agent reads that outcome too: its notebook view and compaction summary show such a cell as failed, and anton's end-of-turn memory pass counts it as one. A failed or unconfirmed cell keeps its code for binding analysis but cannot establish ownership of a file while it stays in the pad. A cell that compaction folds into a summary keeps that flag, so an earlier failed write there can't make its file the agent's own either. Plain-text refusals are never replayed as executed cells. Old events whose call and result both lost their code cannot be recovered.

An answer produced in this process (`COWORK_TURN_BACKEND=inprocess`, the default) runs its own database work as units too. One at the start saves the question (pending) and loads the conversation. The harness reads the history, the attachments and each `recall_history` lookup in units of its own. One at the end saves the answer and clears the question's pending flag in one commit. Both hold a per-conversation lock that keeps a conversation's message writes in order. So such an answer holds no pooled connection while the model answers or the stream runs. A save that fails at the end of a finished or stopped answer ends the stream with `response.failed` (`server_busy` when no connection freed in time), never `response.completed`. An interrupted or failed answer whose save fails keeps its own `response.failed` frame, and the next boot's recovery seals an interrupted one into history. A Stop, a second Stop or a shutdown that arrives while a turn saves its question or its answer waits for that save, so the stream's last frame always matches what the database holds. The compacted history summary and new artifact index are database units too. The harness waits for its cleanup through `run_to_completion`, so a Stop cannot abandon these writes, and a full pool never blocks the event loop. A shutdown can still lose one that has not started when the event loop closes ("Work on the shared file store" below). When the pool stays full, a Stop waits up to `POOL_TIMEOUT` for each pending write (the answer, a compacted summary and a new artifact's row), so `/responses/cancel` can take up to three times `POOL_TIMEOUT`. A Stop during a running cell returns within 5 s only while the pool frees connections. An answer produced by the remote backend (`COWORK_TURN_BACKEND=remote`) works the same way. Saving the question, saving the compacted summary the pod reports, indexing the artifacts it created and saving the answer are each a unit, and the artifacts' owners are looked up in a unit before they are published. Staging the pod's workspace, reading what the pod is seeded with (its history and the project's memory), building the skills it gets and saving the memory it reports work on the file store, so each runs as one unit on the file store's threads (`run_efs_unit`, "Work on the shared file store" below). So it holds no pooled connection while the pod answers, and its settings come from the snapshot the request loaded. The turn indexes the artifacts the pod created once, however it ends. A shutdown waits up to 8 s at the end of its teardown for that write, and loses it if it is still running then ("Work on the shared file store" below). A Stop waits for that write and then for the answer's save, so with the pool full `/responses/cancel` can take up to twice `POOL_TIMEOUT`. A memory or summary save that is still waiting for a connection when a Stop lands is dropped.

The connector form's stream (`POST /api/v1/connectors/submissions/`) reads its conversation and the settings that build the probe's model client, and saves the probe's narrative, as units too. So it holds no connection while the credential probe waits on the model, and it never waits for one on the event loop. When no connection frees in time for the conversation read or the settings read, the form shows `Cowork is busy. Try again in about 5 seconds.` and the probe does not start. The stream ends at that refusal and saves nothing to the conversation. A save of the probe's narrative that fails still ends the stream with `response.completed`, because the form carries the probe's outcome.

Every other coroutine route reads and writes through database units as well, and takes the caller's tenant scope (`get_tenant_scope`) instead of the request's session. Search, settings, the MindsHub workspace selector, the channel configuration routes, a model comparison's usage, delete and continue routes, and the read of a skill's stored rule before its sharing save therefore wait for a connection in a worker thread, and a full pool refuses them with the same `503`. `tests/test_coroutine_routes_hold_no_session.py` fails when a coroutine route takes the request's session, and when a coroutine dependency of any route does, since that dependency runs on the event loop even under a sync route. The routes still allowed to are listed there with their reasons, and all four are desktop-only. A channel turn, an inbound Slack, Discord or Telegram message, runs the same way. Its binding, its conversation, its harness, its question, its answer and the artifacts it created are each read or written in a unit, so it holds no pooled connection while the agent answers. The turn's first unit reads its harness and loads its settings. The rest of the turn runs with that snapshot bound, as a browser turn does (`use_turn_settings`). The pod's seed is read in one unit by the same reader a browser turn uses (`ResponsesHandler._read_remote_turn_inputs`), less the project memory. That read and each memory save the pod reports take their settings from the snapshot, so no unit waits for a second connection while it holds the first. The workspace staging before the pod starts, and the memory and compaction the pod reports, go through the same helpers a browser turn uses (`stage_remote_workspace`, `save_turn_memory` and `save_turn_compaction` in `cowork/handlers/responses.py`). The staging, the skills block the pod gets and each memory save work on the file store, so each runs as one unit on the file store's threads (`run_efs_unit`). A full pool or a busy file store refuses the turn at its staging or its skills block, but a memory or compaction write it refuses is dropped with a WARNING, and the turn goes on to its reply. When the pool or the file store refuses a channel message before the agent answers it, the sender gets one short reply, "I'm overloaded right now and couldn't start on that. Please send it again in a minute.", and the event is still recorded `failed`. A refusal while the binding is read or made answers only a direct message or a message that mentions the bot, since the channel's trigger rule is not read yet. Once the agent has answered, a full pool never costs the sender the reply: the saves of the question and the answer each try 3 times, 1 and then 2 seconds apart, and a save still refused is dropped with a WARNING naming the conversation, and the reply is delivered anyway. The turn then sends the files of the artifacts it created or touched, as its own artifact cards name them, never another conversation's from the project's shared artifacts folder: at most 3, each a regular file inside the project's `.anton/artifacts`, read in one `run_efs` call through descriptors that do not follow a link. A file larger than the adapter's own upload cap (`max_attachment_bytes`: 10 MiB for Discord, 50 MiB for Slack and Telegram, 100 MiB for WhatsApp, and 100 MiB for an adapter that states none) is skipped on its size before any of it is read. A file store that frees no slot in time sends no files, with a WARNING, and the turn still counts as delivered. Each adapter's `send_attachment` takes the bytes and the file name, and refuses a file past its own platform's cap. A channel attachment keeps only the last component of its platform file name, as an upload does (`upload_basename` in `cowork/services/files.py`); a name that is empty, `.` or `..` is stored as `file`. In org tenancy a channel turn saves no memory at all: it runs for its organization with no member, and neither memory tier takes an entry without one, so the turn logs one INFO line instead. The webhook records each inbound event in a unit before it schedules the event. A cancel of the intake, such as a credentials change, a lost lease or a shutdown, waits for the event in hand to be both recorded and scheduled, so a recorded event is never left for its redelivery to be dropped as a duplicate. Telegram's poll loop polls a batch again when its intake fails, so a full pool delays those messages and the loop keeps running. A stream (the Discord gateway, Slack Socket Mode) has no offset to resend, and a new connection does not bring a message back, so a batch the pool refuses is taken in again on the same connection, 3 seconds after each refusal, until it is recorded; dedupe drops the events of it already recorded. An event is marked `failed` only when the runtime itself fails on it. Its status is a record for operators that dedupe does not read, so a status write the pool refuses is dropped with a WARNING. The ingress reconcile, which runs every 30 seconds on every replica, reads every organization's installations in one unit.

A question sent into a conversation whose streamed turn is still answering is refused with `409` and `{"detail": "Another question is still being answered in this conversation. Wait for it to finish, then send yours again.", "code": "turn_in_progress"}`. It is refused before anything reads or saves it, and the running answer goes on. A turn whose stream has already ended counts as finished, so a question queued behind it is accepted. The check runs before any database read, and `RunRegistry.start` repeats it under a lock for that conversation before it opens the new turn's buffer. Two questions racing through the gate read the same history, so both compute the same turn number, which names the turn's buffer and which boot recovery reads as the question's position in the history. Once a turn saves its question, the registry refuses a later start at that turn's number or an earlier one, even after that turn has ended. So the second of two racing questions is refused with the same `409` whether it reaches the registry before or after the first turn ends, and two questions never share a buffer. A question that follows a turn that saved nothing (refused for a full pool, say) keeps that turn's number and is accepted. A queued follow-up waits for the preceding turn's internal terminal record once its final client frame is being written; slow buffer creation for one conversation does not hold up other conversations.

With several replicas (`COWORK_STREAM_BACKEND=redis`), a send that reaches another replica than the one running the conversation's turn gets the same `409`. Each turn takes its conversation's turn claim in Redis before it opens its buffer, renews it every 10 s, and releases it once its stream has a terminal record. Only the run that holds the claim writes the conversation's turn index entry, and a channel turn writes it only while no run holds the claim, so Stop and answers keep reaching the run that is answering. A turn also notes there that it saved its question as soon as it saves it, so the second of two racing questions is refused with `409` on every replica, also when it arrives after the first turn ended. A claim that is not renewed lapses 90 s after its last renewal, longer than the stall the liveness probe forgives (up to 65 s) plus the time a claim can have run down when that stall starts. A send that arrives while another replica's turn writes its final frame waits up to 10 s for that turn to release its claim, as a follow-up on the same replica does. When Redis cannot say whether another run holds the conversation, the send is refused with `503`, `Retry-After: 2` and `{"detail": "Cowork is busy. Try again in about 2 seconds.", "code": "server_busy"}`. A claim that a refused send may have taken, or that a finished turn could not release, is released again every second until Redis answers, so it is gone about a second after Redis is back. A process that starts in a pod releases the claims an earlier process in that pod left, and writes the controller's cancel flag for each of their turns, which relies on one server process per pod. That covers a container restarted in place, after an OOM kill or a crash, that starts again within 90 s of its last renewal. A liveness restart kills a stalled process only after its claims have lapsed, so it counts as a force-kill here. A pod that is force-killed or lost releases nothing, so a send into a conversation whose turn it was relaying is refused with `409` until that turn's claim lapses, up to 90 s, and is admitted after that. Nothing ends that dead turn's stream or saves it yet: a reader that stays attached ends the stream after its idle cut, and its question stays pending. Where turns run in a sandbox, the dead turn's sandbox turn runs on until it ends or reaches the controller's turn limit, and the next send in that conversation waits behind it, unless a Stop on the dead turn, sent before that next send, cancels it. A conversation delete stops the turn it finds running, on whichever replica it lands. The replica running the turn cancels it and releases its claim. Another replica reads the claim and writes the turn's cancel flag with the cause `discarded`, so the turn stops without saving into the deleted history or writing to its deleted stream, and releases its claim; the next send is admitted once it has stopped. A delete that cannot write that flag logs `turn_discard_cancel_failed`, and the next send is then refused with `409` until the turn ends. A conversation delete that cannot reach Redis to forget the conversation's turn index entry logs `turn_index_forget_failed`, and a send at or below the deleted turn's number is then refused with `409` on every replica until 5 minutes after that turn released its claim. A turn whose process died before its release left no release time, so such a send is refused until the entry expires, an hour after the turn started. That is how long a turn's saved question goes on refusing a send at its number that read the history before the save, far longer than a request spends between reading the history and starting its turn. The claim does not order database writes either: a save that outlives its turn at shutdown can still be writing when a turn on another replica starts. `cowork/streaming/turn_claim.py` describes the keys. The file backend never reads or writes them.

Where turns run remotely (`COWORK_TURN_BACKEND=remote`), a replica that has begun ending its turns for a shutdown refuses a new one with `503`, `Retry-After: 1` and `{"detail": "Cowork is busy. Try again in about 1 second.", "code": "server_busy"}`, the refusal a full database pool gives. The retry reaches another replica. `deployment/cowork-server/README.md` describes what a shutdown does to the answers in progress.

A turn that fails ends with one `response.failed` frame carrying a stable `code` and the user-facing `error` (`cowork/handlers/turn_errors.py`), and the conversation's saved events keep the same payload, so a reload shows the same card. For `included_allowance_exhausted` and `free_serving_paused` the frame also carries `reset_at`, the instant the free way forward comes back. An in-process turn passes the gate's `X-MindsHub-Reset-At` header through as sent. A hosted turn takes it from the anton worker's `turn_failed` reply, which scratchpad-controller forwards beside `error`. The producer keeps it only for those two codes and only when it parses as an ISO-8601 instant with a UTC offset. An older worker sends none, and its frame has no `reset_at`.

In-process turns pick the billing code from anton's exception type, the same way hosted turns do: `WalletEmptyError` gives `token_limit`, `AllowanceExhaustedError` gives `included_allowance_exhausted`, and `FreeServingPausedError` gives `free_serving_paused`. This holds when a self-hosted stack sets `MINDS_URL` to `https://minds.invalid`, because anton builds these types only for a response from a MindsHub host. The `X-MindsHub-Reason` header counts unless the response provably came from another host, and the body `code` counts only from the configured `minds_url` host. In-process, a `reason` attribute or a class with the same name never selects a code; hosted turns map anton's own type name.

The server never picks a billing code from a 429's status alone: the code comes from a recognized reason or from anton's exception type. anton retries a gateway 429 it cannot classify, and the turn ends as `provider_overloaded`, in-process and hosted alike. A 429 that reaches the server unclassified ends as `anton_error`. From the configured gateway, a 402 with no recognized reason gives `token_limit` and a 503 gives `policy_unavailable`.

A turn that stops before it finishes, at a limit, a content filter or another hand-back, still ends with `response.completed`, so every client sees the answer end and keeps its text. Its `response.status` is then `incomplete`, and `response.incomplete_details.reason` says why: `spend_ceiling`, `round_cap`, `handback_budget`, `handback_verifier_failure`, `handback_stuck`, `retry_exhausted`, `content_filter` or `question_unanswered` (anton's `ended_by`, described on `IncompleteDetails` in `cowork/schemas/responses.py`). A turn that finished has `status: "completed"` and `incomplete_details: null`. `stream: false` returns the same status and reason in its JSON body.

An answer whose text holds figures gets one `response.figure_sources` event just before `response.completed`. The event lists each figure: its place in the answer (UTF-16 offsets, the unit JavaScript strings index by), its text, its origin (`step`, `question`, `earlier_answer` or `none`) and the steps it came from. Each step it names is listed once, with its kind, a short label, and the project file or attached file, web page, SQL and requests it shows. A cloud pod sees only its conversation's folder and the project's artifacts, so a step there names a file in one of them or an attached file. The event also names the project the turn ran in (`project_id`), which a step's file path is relative to. The formatter (`cowork/harnesses/anton_harness/stream_formatter.py`) finds them in a worker thread once the answer is final, and `figure_sources.py` does the matching. It matches each figure only against this turn's steps: the scratchpad's cells, the results of the web tools anton runs itself (a provider's own web search never reaches the server, so a figure from it has no source), the files `read_text_file` read, other than the agent's memory files in `.anton/memory`, and the results of the connector tools that return the customer's records. Those memory reads, like every other tool, hand back the agent's own text or the user's (an earlier answer from `recall_history`, an artifact, a memory), so their results hold no figure. A cell that reads a memory file never names it as its file, and a cell that makes no request and whose only source would be the memory files it or an earlier cell read is not a step. That rule misses what the cell reader can't see: a cell that also computes a figure from its own numbers or loads data through a call the reader doesn't classify (a library's download, a pickle) is left out, and a memory path built from a name or found by listing the folder reads as any other file. A connector tool is an MCP tool anton registers as `{engine}__{tool}`, or `{engine}__{connection}__{tool}` when the turn has two connections of one engine. anton's `returns_records` (`anton.core.mcp.access`) says which ones return records, such as HubSpot's `search_crm_objects`. Such a step is a query labeled by its engine and tool (`hubspot: search_crm_objects`), with the connection when the name carries one (`hubspot (crm2): search_crm_objects`). Its result is read as the tool returned it, less the ISO date-times in its records (`2026-03-04T09:12:44.512Z`): a record's timestamps are not figures, and the minutes of the created and last-changed times every HubSpot record carries would source almost any small number. A connector call that carries a `sql` argument, as HubSpot's `query_crm_data` does, lists that argument as the step's SQL as the call sent it, through the credential sweep and the length cap (`MAX_QUERY_CHARS`) every step's SQL passes. It is not split into statements or filtered to read statements, as a cell's SQL is, so an argument that also holds a write statement is listed whole. A connector tool that returns docs, schema or account details (HubSpot's `tool_guidance` or `get_properties`), a write tool and a tool of an engine anton has not classified are not steps. Neither is any connector tool when the installed anton has no `returns_records`, as with a local editable anton older than [mindsdb/anton#585](https://github.com/mindsdb/anton/pull/585), which added it. A call of a web tool, `read_text_file` or a connector tool is a step only when its `tool_done` event said `ok=True`, since the turn's tool rows hold a failed call's error as an ordinary result and that error can quote numbers. A figure written in a time unit (`12h`, `22 hours`, a `13.08-day` lead time, `22 business days`, or the first end of `22-25 hours` or `22 to 25 hours`) is not held by a step whose output names time units in the plural (`lead_time_days`, `hours=21.98`) or right after a number (`36h`, a `30-day` window) but never the figure's, so a `22` in a days column doesn't source a `22 hours` the model worked out. Only the output's units rule a step out: a window in its code or label (`interval '12 months'`, "over the last 30 days") doesn't, and a unit its code or label names keeps the figure. A step whose output names no time unit can still hold such a figure, and a rate (`$42 per hour`, `10 units/day`) has no time unit. The rule drops a true source whose output prints the figure's value but names only another unit, such as a cell that turns days into hours and prints the bare hours beside a days column, so that figure has no source. It doesn't fire on a days column named only in the code (`SELECT lead_time_days ...`) whose output prints a bare row, nor on a range joined by "and" (`between 22 and 25 hours`), so such a figure keeps its step. A cell that prints a figure also names the earlier cells that read the data it was computed from, such as the cell that loaded a file into a frame the printing cell sums. That cell may be an earlier turn's, since each turn's cells pass on the names and data they bound. The formatter reads the earlier turns only when the answer has figures and this turn ran a cell, and then only their last 50 cells. That never changes a figure's origin; past that, a figure can only lose the earlier cell that loaded its data, and its step the SQL or request of a function or string bound that far back. An earlier turn's step never holds a figure itself. A figure that no step of this turn holds and the question doesn't write gets the origin `earlier_answer` when one of the conversation's last 20 answers wrote it in the same form with an origin other than `none`. It names no source, and nothing of that answer is copied. The same form is the same value, exactly, with the same sign, currency sign, scale and percent, and the same unit written right after the digits (`3x`, `40kg`, `12h`). A unit after a space (`12 hours`) and an ISO currency code (`4,600 EUR`) are not part of it, so `12 hours` repeats `12 days`, while `12h` and `12hrs` differ. The turn reads those answers' saved events when it starts, in a database unit of its own (`read_earlier_answers` in `cowork/services/conversations.py`), and only for a conversation the caller owns. An answer with no event (one with no figures, a stopped or failed turn) still takes one of the 20 places. When the read fails, the server logs it and no figure is marked as repeated. The cowork client shows that origin's mark; a client that doesn't know an origin drops that figure, so it shows unmarked. The scratchpad's results arrive as stream events, and the other tools' arrive in the turn's tool rows. A call whose result anton marked `is_error` (one it refused to run, or one an error cut off) is not a step. A figure no step holds counts as the question's when the user typed it, in the question or in an answer to the agent's question during the turn. Context the client appends to the question for the agent (a connect form's state, picked Drive files, attached Notion pages) does not count. The event is saved with the answer's events, so a reload shows the same marks. A direct answer from the routing gate gets the event too, with no steps, so each figure in it is the question's, an earlier answer's or has no source. A non-streaming response carries the same steps and figures as `output[0].figure_sources`. A turn the user stops, a failed turn and a stalled turn get no event, and neither does an answer whose last text is anton's failure reply ("An unexpected error occurred: ..."). A turn that stops early at a limit or another hand-back (`status: "incomplete"`) gets the event as a finished turn does, since its answer is whole. The number in the sentence anton opens a hand-back with to name the limit it reached ("This task reached its limit of 25 steps per request", from `ChatSession._limit_reached`) is anton's setting, so it is no figure and gets no mark. A turn that compacts or hard-truncates its context mid-way still hands the formatter every tool row it wrote, the ones the compaction folded into its summary included, when its anton has `ChatSession.turn_messages()`. In process the server calls it only when the installed anton has it; a cloud pod with it sends those rows in its `history` frame. Its figures are then matched like any turn's. None of those rows are saved: the turn ran near or over the model's context window, so the next turn replays it as text only. A cloud pod or a local editable anton older than [mindsdb/anton#585](https://github.com/mindsdb/anton/pull/585), which added `turn_messages`, sends no rows after a compaction. When such a turn called `read_text_file` on a file outside `.anton/memory`, a web tool anton runs itself or a connector tool that returns records, and that call's `tool_done` said it succeeded, a figure that no step holds, and that neither the question nor an earlier answer gives, is left out of the event, so it gets no mark rather than "no source". When finding the sources fails, the server logs it and the answer goes out without them.

A step's label, web page, SQL and requests pass through a best-effort credential sweep: `_redacted` in `cowork/harnesses/anton_harness/figure_sources.py`, whose docstring lists every key name and token shape it knows and the forms it still misses, and where a connector with its own secret fields adds them. A secret under a key name the sweep doesn't know, or passed by position, is sent as written, as the step's code in the scratchpad events already shows it. A cell's SQL lists only read statements, each on its own, so a statement after one in the same string is left out; a string that two databases' quoting, comment, bracket or dollar-quote rules would split differently lists no statement, and `cell_queries.py` names the forms the split still misses, such as a `;` inside a bracket before any quote, where Postgres ends a statement, a `'` inside Oracle's `q'[...]'` quoting, and a `;` between backticks, which Postgres reads as an operator character. The sweep masks the password a listed statement sets with `IDENTIFIED BY`, with a quoted value after a secret key's name such as `PASSWORD '...'`, `SECRET '...'` or `SESSION_TOKEN '...'`, or after `REPLACE`, as it does in a string literal that holds one. A request carries its method and URL template, never headers, bodies or query values.

A background **scheduler** loop polls the database every 30 seconds for due schedules, supporting `once`, `every_15_minutes`, `every_30_minutes`, `hourly`, `daily`, `weekdays`, and `weekly` cadences. The poll and each run's reads and writes are database units, so a full pool refuses them after `POOL_TIMEOUT` rather than stalling the server. A run whose first write is refused does not start, and a cron slot then stays due for the next poll. So does a run refused before its turn starts by a busy pool, a busy file store or an auth outage, and the empty conversation it made for the turn is deleted. Only its rows go: the run made no folder for it, and the delete runs in a database unit, which a slow file store must not hold. A run's last two writes, its outcome and its finish, try again while the pool stays full, for up to 10 minutes, so a busy moment never leaves a run at `running` or its slot due. Every replica runs this loop against the same database, and one cron slot still starts one run. A run's row names the slot it fires for (`schedule_runs.slot_at`), and a unique index on the schedule and that slot makes inserting the row the claim. So when two replicas find the same slot due, the second one's run does nothing. A run renews a lease on its row every 30 seconds while it lives. A run with no renewal for 2 minutes belongs to a server that died without recording how the run ended, and the next poll on any replica fails it. Its slot fires again when its turn never started, and is used up when its turn may have run, so a lost replica neither skips a slot nor runs one twice. A run that leaves its slot due, because its turn never started, a shutdown cut it, or its schedule was paused before it failed, gives the slot back for the next poll to claim. Every run records its `outcome`: whether its agent acted before it failed, was stopped, or left no record of how it ended. Before handing off execution, the run saves `execution_claimed_at`; deleting its chat never clears that claim. After a crash, a claimed run is treated as potentially executed. The upgrade conservatively marks older runs as claimed because a missing chat link cannot prove they never started. A schedule has one run going at a time. A process runs at most `COWORK_SCHEDULED_RUNS_PER_PROCESS` scheduled runs at once (unset: 3 locally, 16 with Redis-backed turns); a due slot over that stays due for a later poll, and Run now answers 409 while the task is running or the process is full. A recurring schedule that fell behind, because the app was not running, runs its latest missed slot once if that slot is no more than seven days old; the slots it passed over are added to `missed_runs`. A one-off runs up to an hour late; from an hour on it is marked missed and runs only on Run now, which uses a one-off up. Pause records `paused_at` and stops what would start on its own; a run already going finishes and nothing follows it. Resume clears the pause and the failure streak. A slot that a run gave back before its agent acted, before or during the pause, runs once more after resume, unless a later slot came due meanwhile; otherwise the schedule waits for its next slot. Three cron slots in a row that settle failed set `auto_paused_at` and pause the schedule; a success resets the streak. Confirmed server-shutdown interruptions leave the streak unchanged; timeouts and interruptions without a shutdown marker still count. Each schedule reports its `state`: `active`, `paused`, `auto_paused`, `missed` or `done`. Run now marks a one-off done only after confirmed successful completion; other endings preserve its previous schedule state. Each run creates a conversation and is tracked in `schedule_runs`. Deleting that conversation does not delete the run: the run keeps its status, timings, and error as audit history, and only its link to the conversation is released. A channel binding pinned to the conversation is released the same way, so the external chat stays bound to its project and the next inbound message starts a fresh conversation.

Schedule responses include `lastRun`, the latest finished manual run or cron run that retained its slot. Abandoned cron attempts remain in run history. Each run reports its `trigger`, outcome, and a readable failure reason. Internal exception details stay in server logs. An ending without reliable evidence neither extends nor resets the failure streak. Terminal records are read with a bounded wait, including when Redis makes the record visible before the producer finishes closing its buffer.


`COWORK_SCHEDULED_RUNS_PER_ORG` optionally caps running tasks per account across replicas (1–256); unset, no account cap is imposed. Hosted limits from auth are tracked in [ENG-4064](https://linear.app/mindsdb/issue/ENG-4064/source-hosted-scheduled-run-organization-limits-from-auth). A full account is skipped before process admission; claims still check the limit atomically. These deployment settings require a restart.

`COWORK_SCHEDULED_RUN_MAX_SECONDS` bounds a run (default 30 minutes, 5 minutes to 24 hours, with a floor covering the remote controller's turn and setup budget). A reaped cloud run has its abandoned turn cancelled. This process excludes runs it is actively driving from its own reaper, including after laptop sleep. Settlement and finalization share a locked transaction, so a late writer cannot count a reaped failure again. Deleting a task stops its active turn on the next lease check; pausing it lets the current turn finish.

Run now accepts `?trial=true` to try a task without consuming its scheduled occurrence or changing its failure streak. Runs report `trigger` as `cron`, `manual`, or `trial`; the existing `isManual` and `trial` fields remain available. Trial migration `6f1b8d3a5c27` follows the last-run index migration `a9e2b6c4d810`.

### Redis through a failover

Hosted Cowork reaches Redis through one URL, `COWORK_TURN_REDIS_URL`, whose Service follows the primary. `cowork/turnqueue/redis_client.py` builds the server's two Redis clients so that a dropped connection, a Redis restart or a primary failover doesn't fail the call in flight:

- A call that fails with a connection error, a timeout, a refused or timed-out connect, or a `READONLY` or `LOADING` reply is sent again with jittered backoff (0.25 s base, 2 s cap), until 30 s after the call started. The blocking client, which conversation delete uses from a threadpool thread, stops after 10 s, and so does one delete's whole Redis cleanup: a conversation or turn delete runs its calls under one 10 s deadline, and a project delete runs every conversation's under one, so a Redis that does not answer holds the request for about 10 s. No connect, write or read waits past a deadline: each is cut to the time left, though to no less than 50 ms, so a call ends at most a few of those 50 ms waits past its deadline, and a call that starts after its group's deadline has passed fails at once without reaching Redis. Each new attempt opens a new connection, so after a demoted primary answers `READONLY`, the next attempt goes back through the URL and reaches the new primary.
- A new connection's handshake (`AUTH`, `SELECT`) is retried with its connect, under the same deadline. redis-py 5 retries only the TCP connect, so a Redis that accepts a connection and closes it mid-handshake, as a restarting Redis or a Sentinel `CLIENT KILL` does, would otherwise fail the call at once.
- A rejected password (`WRONGPASS`, `NOAUTH`) or a refused permission is raised at once and never sent again.
- The 30 s is one deadline per call. redis-py reconnects inside a command's own retry loop, and that reconnect spends what is left of the call's 30 s rather than another 30 s. `redis_deadline` shortens it for a group of calls: `/answer`'s wait for the pod's verdict uses it, so its reads stop retrying when the 15 s wait ends. A transaction whose connection closed while it watched keys ends without a reconnect: the watch ended with the connection, and redis-py 5 would reconnect to send `UNWATCH` after the call, outside its deadline.
- A connection opens within 2 s and a reply arrives within 5 s, and neither waits past the call's deadline, or that attempt fails. Every blocking read blocks for 2 s, under the 5 s, so a reader waiting on a quiet stream is never cut as a dead socket. A primary that is demoted answers its blocked readers `-UNBLOCKED`, and the reader reads again.
- A pooled connection idle for 15 s is PINGed before its next command, so one a failover left half-open is replaced before a write goes out on it.
- redis-py 5's async pool holds its lock while a new connection connects. The server's pool connects after it releases the lock, so callers don't queue behind one caller's connect retries.

The turn claim (`cowork/streaming/turn_claim.py`) does not ride through a failover. Each of its calls stops after 5 s, and a dropped connection or a `READONLY` reply inside its `WATCH` transaction ends that transaction, so a send that arrives during a failover is refused with `503` and `Retry-After`, and a renewal that fails tries again 10 s later.

The job `XADD` that queues a remote turn is sent at most once (`send_once`), since the controller would run a job queued twice. When its reply is lost, or does not arrive within the 5 s read timeout, nobody knows whether Redis applied it, so the producer reads the newest 5 entries of the conversation's job stream and looks for the turn's correlation id. When the job is there, the turn goes on as if the `XADD` had answered. When it is not, or that read fails too, the producer writes the turn's cancel flag, so a copy the controller picks up later stops at its first check, and the turn ends with `response.failed`. A `READONLY` or `LOADING` reply to the `XADD` is still retried, since Redis refused the write without running it. Every other write is retried. The `RPUSH` that delivers an `ask_user` answer can then queue its entry twice, with one `answer_id`: the pod answers the question once, nothing renders its verdict on the copy, and `/answer` returns at the first verdict. A failed `/answer` would be worse: the client would retry, get 409 and resend the typed text as a new message. Inside the answer's `MULTI`, a refused `RPUSH` aborts the `EXEC`, since Redis runs `MULTI` and `EXEC` while it loads. A turn buffer append is retried, so Redis can hold one record twice, and a reader of the buffer yields each `seq` once. The controller numbers each turn's replies with `n`, counting up by one. The relay drops a reply whose `n` equals the last one it applied, which the controller wrote twice. A reply whose `n` skips a number is applied, and the relay logs `remote_reply_gap` with both numbers, since a reply never reached the stream. A reply whose `n` is below the last one comes from a second run of the same job, and the relay fails the turn rather than mix the two runs' answers. A controller that sends no `n` has every reply applied.

Conversation, turn and project delete clear a conversation's Redis buffers and its turn index entry only with `COWORK_STREAM_BACKEND=redis`, so desktop and the single-container server never call Redis on a delete. The buffer walk (`SCAN`) holds one connection of the pool for all its pages, and starts again from cursor 0 when a page went out on a new connection, since after a failover that connection can reach a server the cursor means nothing to. Holding the connection keeps another threadpool thread from reconnecting it unseen between two pages.

## Data Layer

Data lives in two places: a **SQLite database** for structured records and the **filesystem** for project files and agent workspaces. Understanding both is essential.

> The `~/.cowork` paths below are the default (prod) home. The desktop app runs one of several build channels, each with its own isolated home (`~/.cowork-dev`, `~/.cowork-stable`, etc.) selected via `COWORK_HOME`. See the [Cowork frontend README → Build Channels](https://github.com/mindsdb/cowork#build-channels) for the full mapping.

### SQLite database

- **Location**: `~/.cowork/cowork.db` (override with `DATABASE_URI`)
- **ORM**: [SQLModel](https://sqlmodel.tiangolo.com/) (SQLAlchemy + Pydantic)
- **Migrations**: Alembic (`cowork/db/alembic/versions/`). Startup runs `alembic upgrade head` (singular), so the graph must have exactly ONE head: if two branches each added a migration on the same parent, every fresh boot aborts with "Multiple head revisions". After merging or rebasing, check `alembic heads`; if it prints two revisions, add a no-op merge revision whose `down_revision` is the tuple of both heads (see `f4e2c1a9d3b7` for the pattern).

Key tables:

| Table | Purpose |
|-------|---------|
| `projects` | Project metadata and filesystem path |
| `conversations` | Conversation threads, linked to a project |
| `messages` | Individual messages with role, content (JSON), and harness tag |
| `message_events` | Streaming event payloads for a message |
| `files` | Metadata for uploaded files (path points to filesystem) |
| `schedules` / `schedule_runs` | Recurring prompts and their execution history |
| `settings` | Key-value user settings; sensitive values Fernet-encrypted |
| `pins` | User-pinned items (conversations, artifacts, etc.) |
| `channel_*` | Channel installations, bindings, sessions, and events |
| `shared_resource_attributions` | Creator and latest-editor user IDs for org-shared filesystem resources |
| `shared_resource_mutations` | Append-only actor trail for allowed org-shared mutations, including deletes |

All models use UUID primary keys with auto-tracked `created_at`/`modified_at` timestamps.

### Filesystem storage

```
~/.cowork/
├── cowork.db                       # SQLite database
├── .master_key                     # Fernet encryption key for settings
├── skills/                         # COWORK_SKILLS_DIR — canonical SKILL.md store
│   └── <slug>/SKILL.md             # one folder per skill (see docs/SKILLS.md)
├── projects/                       # COWORK_PROJECTS_DIR
│   ├── general/                    # Default project (always exists)
│   └── <project-name>/
│       ├── <user & agent files>    # Working directory visible to agents
│       ├── skills/                 # symlinks to skills enabled for this project
│       │   └── <slug> -> ~/.cowork/skills/<slug>
│       └── .anton/                 # Private agent workspace
│           ├── artifacts/          # Agent-produced outputs (HTML apps, docs, etc.)
│           │   └── <slug>/
│           │       ├── metadata.json
│           │       └── <files>
│           ├── memory/             # Persistent agent memory by category
│           └── context/            # Project context for agent runs
├── files/                          # COWORK_FILES_DIR — uploaded files
│   └── <file-id>/<filename>
└── data-vault/                     # COWORK_VAULT_DIR — encrypted connector creds
    └── <engine>/<connection-name>/
```

### How the two layers relate

The **database** holds structured metadata and relationships (which messages belong to which conversation, which conversation belongs to which project). The **filesystem** holds the actual content agents work with — project files, artifacts, memory entries, and uploaded documents. The `files` and `projects` DB tables store filesystem paths that point into the directory tree above.

This split comes from an ongoing migration from a purely filesystem-based architecture. Structured data that benefits from querying and relationships, such as conversations, messages, settings, and schedules, lives in SQLite. Components that are inherently file-based, such as project working directories, agent artifacts, harness-managed memory, the connector vault credentials outside MindsHub mode, and skills, remain on the filesystem by design. In MindsHub mode, OAuth credentials live per user in MindsHub's Data Vault instead. Skills are canonical `SKILL.md` files, so they can be edited, uploaded, and distributed per project; see [docs/SKILLS.md](docs/SKILLS.md). See [docs/SERVER_MIGRATION.md](docs/SERVER_MIGRATION.md) for the full migration story.

Agents (via their harness) have read/write access to their project's working directory and the private `.anton/` subdirectory. They do **not** access the SQLite database directly — all DB interaction flows through the service layer.

### Work on the shared file store

On a hosted deployment `COWORK_HOME` is a shared NFS mount (EFS), mounted `hard`. A `stat` that the file server doesn't answer blocks its thread with no time limit. On the event loop such a call stops every request on the pod, liveness included. On anyio's default threads it holds threads that sync routes and every request's sync dependencies share.

The artifact routes' lookups and reads, and a turn's artifact bookkeeping, run their file store work through `run_efs` in `cowork/db/units.py`, beside `run_db`. Other file store work still runs on other threads, and a stuck mount can hold those too. On anyio's default threads: the live-app share and copy routes through `run_in_threadpool`, and the sync handlers and `FileResponse` bodies of desktop-only and loopback-only routes. In org tenancy no response body reads the file store there, and neither does any sync route but the loopback-only ones, which `require_local` refuses to any peer but loopback. On asyncio's default executor through `asyncio.to_thread`, for example: the publish uploads, an artifact delete's take-down of its live pages, the live-app reads and the startup owner backfills. `run_efs` works like this:

- `run_efs` has its own 100 threads per process by default (`EFS_SLOTS`). A started call keeps its thread until it returns, so a stuck mount holds that many threads at most.
- One person holds at most 10 of them at once by default (`EFS_SHARE`). A heavy client is then refused before anyone else waits. One organization's work that runs for no person, such as a channel turn, shares 10 the same way. Nothing caps an organization: ten busy people fill the slots, and anyone else, in any organization, then waits for one.
- A caller waits for a slot until `POOL_TIMEOUT`, then gets `FilesystemBusy`, which the app answers with `503` and `Retry-After`.
- A turn's artifact bookkeeping (the snapshot at turn start, the diff, autopublish and the cards at turn end) passes `deadline=False` and waits as long as it takes. In MindsHub mode nothing records a turn's artifacts later, so a refusal there would lose them. The scheduled self-heal of a project's artifacts (below) waits the same way, outside any turn: at most one runs at a time per member and project, and one more per project for its channel turns, and a shutdown cancels it.
- A caller cancelled after its work started waits for the work to return, as with `run_db`, so a lock the work holds is released before the caller unwinds.
- A caller cancelled while it waits for a slot leaves at once, and its work never runs. So a remote turn's turn-end diff and the database record of what it found run as one step under `run_turn_end_record`, which is `run_to_completion` tracked for shutdown: a Stop or a shutdown that lands while either waits or runs waits for both, and the rows are written once. The in-process harness's cleanup at turn end runs under `run_to_completion` the same way.
- During a file store stall, a Stop waits for a slot with no bound, and so does a `/responses/cancel` that reached the replica running the turn. A shutdown waits only until the end of the lifespan teardown, whose last step waits up to 8 s (`TURN_END_RECORD_DRAIN_SECONDS`) for the turn-end records still running (`drain_turn_end_records`). A stall that clears within that wait keeps the turn's artifact rows. The event loop's close then cancels every task still running. It cuts a diff that has not returned and a record still waiting for its database slot, so that turn's artifact rows are lost, and the last step logs `shutdown_turn_record_lost` at ERROR with the conversation, the project, `touched` and `count`. `touched` holds the new artifacts' slugs its diff found, or, when the diff had not returned, the artifacts the pod's `turn_completed` named (`artifacts_touched`, below; empty for a pod that sends no usable list). It holds the first of them in sorted order, none longer than 255 characters (`MAX_ARTIFACT_NAME_CHARS` in `cowork/turnqueue/models.py`): at most 50 (`TURN_END_RECORD_NAMES_LOGGED` in `cowork/db/units.py`), and only as many as fit in 12 KiB as the line encodes them (`TURN_END_RECORD_TOUCHED_BYTES` there), where a multibyte character costs up to 4 bytes and a control character 6. So the line stays under the 16 KiB a container runtime splits a line at, whatever characters the names hold. `count` says how many the record held. A web turn whose record is cut there still saves its answer, marked `turn_lost`, before the loop closes. A web turn whose pod had finished, and which was still recording its artifacts when the shutdown ended it, starts its answer's save once that record ends; a save that starts as the loop closes may not finish. A file store call already running on the stuck mount keeps its worker thread, which is not a daemon thread, so the process stays alive until the kubelet's SIGKILL.
- Autopublish releases each publish lock through `run_efs_cleanup`, under `run_to_completion` too, including when a cancel lands while it takes the lock, and when a cancel or a failure lands while it mints its publish key. It keeps one lock on purpose: that of an artifact whose upload it handed to its thread, running or still queued, when a cancel or its publish timeout lands. A running upload may still record the publish, and autopublish cannot tell a queued upload from a running one. So the lock stays until it is 180 s old, three times the 60 s publish timeout, counted from when it was taken. Until then later turns skip that slug as `lock_busy`. At a shutdown during a file store stall, the event loop's close can also cut a release still waiting for a slot, and that lock stays until the same age.

The artifact routes read in turn. One `run_db` unit reads the project row, the conversations the caller owns and the artifact's location row (below). One `run_efs` call then opens the folder that row names, or finds the artifact roots and the folder. The caller's own folder answers; for a reviewer, a co-member's folder answers behind the owner's grant, checked in that same call. A lookup that listed the roots records what it found in one more `run_db` unit when that can change the row, and a refusal there is only logged. The artifacts list finds its roots and reads its cards in one `run_efs` call. A route a reviewer may use too reads what it needs in the lookup's `run_efs` call, once the lookup has let the caller in (`review_artifact_and_read`), so it waits for a file store slot once. Those routes are the review entry, the draft link, the private draft preview (`/drafts/{project}/{artifact}/{path}`) and the draft host's preview. A preview checks the file's path and opens the file in that call. It builds a UTF-8 HTML preview there too, and streams anything else from the open file after the call. A route only the owner may use checks ownership in another `run_db` unit after the lookup. Only then does it read the folder, in one more `run_efs` call. So a read never runs for a caller that check refuses, and that second wait can still answer `503`. A save or a restore that changed the source then takes the artifact's publish lock, to republish the artifact if it is live. The access change takes the same lock. Each try at that lock is another `run_efs` call, and a busy file store counts as a busy lock. The route tries again every 0.1 s for up to 60 s. If it gets no lock, the access change answers `503`, and a save or a restore keeps its change without the republish. The request's own session never takes a connection while a file store call waits. A delete runs in three steps. A file store call pins the folder, checks it and lists its live pages. The pages come down on asyncio's default executor, so a slow publish service holds no file store slot. Where the server keeps owner records for published pages (`COWORK_TRUST_IDENTITY_HEADERS`), they come down in that first call instead, since each page's owner check holds the publish record's lock across its take-down. A second file store call then records them as unpublished and removes the folder. In MindsHub mode the delete first revokes the artifact's draft review rule, and once that is gone the delete's own file store calls wait for a slot as long as it takes, rather than answer `503` and leave the folder's review marker without its rule. An upload whose database row is refused answers at once, and its folder is removed in the background once a file store slot frees.

In org tenancy each artifact id has a row in `artifact_locations`: its project, its conversation for a legacy per-conversation root, and its folder name. A lookup reads the row with the project, then opens that one folder inside a root the caller may already read, and serves it only when the folder's `metadata.json` names the requested id. A `metadata.json` that is not a regular file, such as a FIFO an agent left in its place, holds no id, and no read of it waits for a writer. Without a row, or when the folder no longer holds the id, the lookup lists the project's roots as before. That scan reads every folder of the project that holds the id, a co-member's legacy roots included, and records what it found: the folder, when one holds the id. An id held by two folders of one project is marked `conflicted` instead: its lookups keep listing the roots and answer 409 when two of its folders are in the roots that decide the lookup (the caller's own, or every member's when the caller's own hold none), as before the table. While the row's folder holds the id, such a lookup reads the caller's own roots first, without a stat of any co-member's workspace, reads every member's only when the caller's own hold no folder, and records nothing. The mark clears when such a lookup finds the row's folder alone and a strict count of every root of the project, each co-member's legacy roots included, agrees. Each replica runs that count at most once per id every 15 minutes, so once the other folder is gone, the mark clears at the first such lookup on a replica that has not counted the id in the last 15 minutes. The backfill below clears a mark too, but only while its pass runs. A folder whose name is not valid UTF-8 counts toward a conflict, but no row names it, since no row can store it. Only a caller who reads the row's root as their own moves or drops a stale row, and a folder that fails to read leaves the row as it is. There is one row per id in an organization, so an id held in two of its projects names one of them, and each lookup from the other project lists that project's roots and leaves the row as it is. Only such a scan, or the backfill below, writes a row, since each reads every root of the project. A turn end sees only its own folders, so it writes none, and a new artifact's first lookup lists the roots once. A turn's folder whose id has a row naming another folder of the project marks the id conflicted. A copy no turn end claims, such as one a shell `cp -r` makes, is caught where the project's whole artifacts folder is still listed: at a remote turn end whose pod sends no list of the artifacts it touched, and at the project's scheduled self-heal (below). Each reads the id of every folder it lists, and marks an id conflicted when its row names another folder of the project. A row whose folder the listing read and found gone or holding another id, as after a shell `mv`, is stale instead, and the id's next lookup moves it. Each listing sorts its own errors by errno: an entry that is gone, is not a folder, or is a link loop (`ENOENT`, `ENOTDIR`, `ELOOP`) holds no artifact, and any other error leaves the entry unreadable, whatever Python's own `Path.is_dir` would answer (3.12 and 3.13 raise on `EIO`, 3.14 answers False). The id reader never follows a link, so an entry that is a link, or whose `metadata.json` is one, is never counted unreadable. Neither is the artifacts folder itself, or any entry below it, when that folder is a link, even one to a folder the server cannot list: no listing of it can ever find an id. No listing includes the project's `.locks` folder. A folder the listing could not read is never taken for one that is gone, so its row still marks the copy. A listing that could not read a folder, or the project's artifacts folder itself, logs one warning (`event` `artifact_location_listing_unreadable`, with the project and a `reason`: `folders_unreadable`, with the `count` of folders it could not read, or `artifacts_folder_unreadable`, with no `count`, since it then read no folder below it) and asks for the project's location check again, even when it found no id. So a folder the listing could not read is listed again at the next self-heal, and a copy in it is marked at the first listing that reads it. That check publishes nothing. Each replica keeps its own requests and runs its own poll, and a listing asks again on the replica that made it, so a folder that stays unreadable costs one listing of the project every 15 minutes (`SELF_HEAL_INTERVAL_S`) on each replica that has listed it, for each member and project, and for each project's channel turns. Those listings stop when the folder reads again or the replica restarts. With the pod's list, a copy made in a turn, a channel turn's included, is marked within about 15 minutes of that turn when its reads succeed, and otherwise at the first later self-heal whose listing reads the copy's folder, whether or not autopublish is on, even when that turn's diff fails or a Stop cuts its turn end, and its id's lookups then answer 409. Without the pod's list, the turn end's own listing marks it; a Stop waits for that listing and its record, and a turn end whose diff or record fails asks for the project's self-heal instead. Rows move with a task's folders and are deleted with the artifact. A one-time pass at startup (`cowork/services/artifact_location_backfill.py`, sentinel setting `_artifact_location_backfill_v1`) records the artifacts made before the table, holding one file store slot while it walks. A project whose roots or folders it cannot all read gets no rows from it: only a folder that is gone, is a link, holds no metadata that parses, or is one the server may not read at all (`EACCES`, `EPERM`) counts as holding no id. The pass walks the projects in id order and records each one it finishes, and each one it could not read, in the `_artifact_location_backfill_v1_progress` setting row. An id whose row the database refuses, or whose row another writer deletes mid-write, is skipped and logged with `event` `artifact_location_backfill_skipped`. An id whose one folder has a name that is not valid UTF-8 is skipped too. A stop at shutdown, a project it could not read, or any other database error leaves the sentinel unwritten, and the next start walks only the projects the pass has not finished and the ones it could not read that still exist. The pass writes its sentinel, and deletes the progress row, once it reaches the last project with none left unread. Its summary lines (`event` `artifact_location_backfill_summary`) count the ids `recorded`, `duplicated`, `conflicts` and `skipped`, and the projects `unread`. To walk every project again, delete both setting rows and restart cowork-server. Shutdown sets the walk's stop and waits at most 2 s for it (`LOCATION_BACKFILL_SHUTDOWN_SECONDS`), then goes on. The desktop writes no rows.

At the end of a remote turn the pod's `turn_completed` names the artifacts its tools touched (`artifacts_touched`, at most 1,000 names: `MAX_ARTIFACTS_TOUCHED` in `cowork/turnqueue/models.py`, which anton and scratchpad-controller mirror; a longer list counts as none, and so does a list holding a name longer than 255 characters, `MAX_ARTIFACT_NAME_CHARS` there). The turn claims a listed folder as created only when it appeared during the turn and its provenance in `metadata.json` names no other conversation. It reads only the listed folders' content for edits. The list alone is not enough: anton also lists a sibling's folder whose files changed during one of this turn's cells. The turn end opens each listed name as one folder directly below the artifacts folder, never a link. A listed folder it cannot read is left out of the turn and logged as a warning. A pod that sends no list gets the older diff: provenance in `metadata.json` decides what the turn created, and every folder's content is read for edits. The repair route records each folder it queues an agent repair in (`queued_agent_repairs`, per conversation and keyed by org), and takes the row back when it refuses the repair. A turn reads its conversation's rows when it starts, so with the pod's list the diff also checks those folders for a repair the turn should finish, and lists nothing else. When the rows can't be read, the diff lists the folder only to find a repair queued to the conversation. Once its cards are out, the turn end drops the rows it read whose folders it checked, by id: a repair recorded again during the turn gets a new row, which stays for the next turn, and so does the row of a folder the diff could not read or whose queued repair the turn end could not finish, such as one whose capture found the artifact's lock held. A pod with no list gets the full listing, and every folder is checked for a queued repair.

With the pod's list, autopublish at the turn end also opens only the turn's own folders, publishes those that need it (never the project's `.locks` folder), and lists nothing. The rest of the project waits for its self-heal. In org tenancy such a turn end asks for one as its record starts, before the diff (`request_turn_end_self_heal`), a channel turn's too, whether or not the organization's autopublish (`artifact_autopublish_enabled`) is on. So a turn end whose diff fails, or that a Stop cuts, still asks, though it never reaches its publish. The scheduler's poll loop (`cowork/scheduler.py`, every 30 seconds) runs it at most once every 15 minutes per member and project, and once per project for its channel turns (`SELF_HEAL_INTERVAL_S` in `cowork/services/artifact_autopublish.py`). A self-heal lists the project's artifacts and checks each one's id against its location row (above). One whose listing could not read a folder, or whose check fails outright, asks for the next location check, which publishes nothing. Then, only while autopublish is on and for a member's self-heal, it publishes those the member owns that need it, such as an artifact whose publish failed, under a turn end's limits: five publishes and a 60-second budget. Each replica keeps its own requests in memory, so a restart drops them until the project's next turn. A pod with no list asks for a self-heal only when its turn end's own check of the project's locations falls short: its listing could not read a folder or the project's artifacts folder (above), which asks for the location check alone, or its diff, or the record of what the diff listed, fails, which asks for the whole self-heal. Otherwise its turn end lists the project itself, checks every id there, and, while autopublish is on, publishes what needs it.

Routes that work on the file store run their work as one unit on the file store's threads (`run_efs_unit`): the project file, skill, memory and upload routes, a skill's sharing save, its caller's own switch and the read of its stored rule before the save, a model comparison's create, continue and delete, the deletes of a conversation, a turn, a file or an attachment, an attachment's move into a project, a task's move, and the shared-project reads. A remote turn's workspace staging, its skills block and each memory save its pod reports run there too, for a browser turn and a channel turn alike, and so does a browser turn's seed read, which reads the project's memory. An artifact's access routes read the keys of its rule from the file store through `run_efs`, then ask MindsHub for the rule on one of anyio's default threads. The Drive un-pick (`DELETE /api/v1/connectors/connections/{engine}/{name}/picked-files/{file_id}`) reads no database and runs its vault work through `run_efs`, and so does the OAuth callback (`GET /api/v1/connectors/oauth/{service}/callback`), which any caller reaches and which reads and clears its pending sign-in in `COWORK_HOME`. A conversation list or create, or a schedule create, that names no project by id, and a conversation update that names its project by name, may resolve the general project, which checks that project's folder, so they run on `run_efs_unit` too. A request that names its project by id, lists every project, or changes only a title needs only the database and runs as a `run_db` unit. In org tenancy the project list reads rows only once the organization has its general project: the organization's first list makes that project and its folder on `run_efs_unit`, and later lists don't check the folder, which a project file route recreates when it is missing. A new channel thread's binding may start in the general project, so it is made on `run_efs_unit` too; a known thread's binding is a `run_db` read. In org tenancy the sync routes left on anyio's default threads touch only the database; the others answer only the desktop (desktop-only or loopback-only routes). A unit opens its own session and closes it when the work returns. After its file store slot it takes one of the file units' own slots (`EFS_UNIT_SLOTS`, 16 by default) and then a database unit slot, as `run_db` does, by the same `POOL_TIMEOUT` deadline, so it is refused about `POOL_TIMEOUT` after it asked, with `FilesystemBusy` or `DatabaseBusy`. Such a unit holds its connection while the file store answers, so units stuck on a slow store hold at most `EFS_UNIT_SLOTS` connections, and database units keep the rest of the pool. Its thread waits inside a full pool only while a checkout outside the units, such as a sync route's request session, holds the pool's connections; the pool's own timeout then refuses it with `DatabaseBusy`, up to about two `POOL_TIMEOUT`s after it asked. On SQLite, the desktop's database, a file store unit takes no unit slot: that one slot orders the database units, and a long upload or delete holding it would refuse every database unit behind it. A 5xx that a unit raises as an `HTTPException` is logged with its cause (`Session error: ... status=503 cause=...`), as the request session logs one.

The files these routes serve go out through `EfsFileResponse` (`cowork/api/v1/efs_file_response.py`), which opens and stats the file in one call on the same threads before it sends a header, so a busy file store answers `503` there, and a file that is gone by then answers `404`. A file or attachment download reads only the file's row before that, in a `run_db` unit, so it waits for a file store slot once. Project files and drafts stream from their pinned descriptor through `PinnedStreamingResponse` (`cowork/api/v1/pinned_response.py`), which reads each chunk on the same threads. A project file's open never waits on what sits at the path, so a FIFO an agent made there answers `404` at once, as anything but a regular file does, rather than holding its file store slot until the pod restarts. Once a response's headers are out, a refusal could only cut its body short, so each read waits for a slot as long as it takes. A client that leaves stops the reads: a send after a disconnect raises on an ASGI 2.4 server, and below 2.4, as uvicorn reports, the response listens for the disconnect and cancels its body. A response's close runs on the file store's threads even when a disconnect, a Stop or a shutdown cancels it (`run_efs_cleanup`). Starlette's `FileResponse` reads on anyio's default threads, so only desktop-only routes use it. The request's identity dependencies (`get_principal`, `get_tenant_scope`, `get_scoped_session` and `get_session`) are coroutines, so authenticating a request takes no thread.

Each turn restores the caller's vault secrets, which history is scrubbed against by exact value, through `run_efs`. A turn that gets no file store slot within `POOL_TIMEOUT` is refused, since its history would otherwise reach the model and the pod scrubbed by key shape alone: a web turn answers `503` with `Retry-After`, a channel turn fails, as it does when the database pool is full, and a scheduled run fails, leaving its cron slot due for the next poll. A vault that can't be read falls back to the shape-only scrubbing.

A daemon thread, `cowork/common/loop_watchdog.py`, watches for what slips through:

- `event_loop_stalled` (ERROR) means the event loop didn't answer for 2 s. The line carries the loop thread's stack and the requests in flight.
- `event_loop_recovered` (WARNING) gives the stall's length once the loop answers again.
- `limiter_saturated` (WARNING) means anyio's default threads, the `run_db` slots, the `run_efs` slots or the file units' `EFS_UNIT_SLOTS` (`run_efs_unit`) stayed full for 5 s.

Each boot logs `startup_finished` (WARNING) with the boot's length in `duration_ms`, counted from the server process's start, and each lifespan step's length in `steps`. A lifespan step still running 10 s after it starts (`_SLOW_STEP_S` in `cowork/server.py`) logs `startup_step_running` (WARNING) once, from a timer thread, with its name in `step` and its time so far in `duration_ms`. A step that took 10 s or more logs `startup_step_slow` (WARNING) as it ends, with the same fields. A step's timer stops when the step ends. A startup that fails stops the running step's timer when the task running the lifespan ends, so a step that raised is not later logged as still running. On a hosted deployment the liveness probe's delay has to cover that boot (`deployment/cowork-server/README.md`, "Probes").

**Settings** use a hybrid approach: user preferences and API keys are stored in the `settings` DB table (with Fernet encryption for secrets), while desktop connector credentials live in the filesystem vault (`data-vault/`). In MindsHub mode the OAuth routes proxy the caller's private connection in MindsHub's Data Vault instead.

**The MindsHub credential is the exception, and it is stored nowhere.** On the
desktop it is the user's own session token, which lives ten minutes, so the
Electron app hands it over at runtime through `PUT /api/v1/runtime-credential/minds`
and this process keeps it in memory. `SettingService._raw_data` overlays it onto
the stored rows, so every reader of `get_user_settings()` sees it without
knowing where it came from, and a value handed over beats any stored row.
The desktop app's own browser is handed over the same way, through
`PUT /api/v1/runtime-credential/browser` ([The agent's browser](#the-agents-browser)).

Two properties follow. **Nothing survives a restart**, so the desktop app re-pushes on every start of this process. And a key the user supplied by hand travels the same way, which is what keeps a long-lived `mdb_` key out of both `.env` and the settings table. The route is loopback-only and refuses in MindsHub mode: there the turn producer mints a per-turn credential and the pods are never handed one.

**A live turn re-reads it per request, with one exception.** A turn that copied the credential into its provider once would keep sending the token it started with across a hand-over, and take a gateway 401 that looks like a dead account. So `build_llm_client` passes an `api_key_provider` into MindsHub-backed providers, and each outbound model call reads the current in-memory value. Two limits are deliberate. Static MindsHub-mode and user-supplied keys keep their construction-time value, because nothing rotates them. And the **scratchpad subprocess keeps the token it was started with** for the rest of its turn: `export_connection_info()` hands it a string once and it has no supplier, so a pad-side model call still runs on that value. Refreshing it needs a pad IPC contract, which is out of scope here.

`build_llm_client` capability-gates the kwarg on `inspect.signature`, so an
older Anton keeps the static key and logs the degradation rather than failing
every turn.

## API

All endpoints live under `/api/v1/`. Key resource groups:

| Path | Description |
|------|-------------|
| `/health` | Server status: versions, capabilities and configuration state, which the desktop app and the web client poll |
| `/health/live` | Liveness probe: answers whenever the event loop does, and reads no database, file store or settings |
| `/health/ready` | Readiness probe: the same, for the kubelet's readiness check |
| `/projects` | Project CRUD and working-folder management |
| `/conversations` | Conversation threads and message history |
| `/responses` | Streaming agent responses (SSE) |
| `/files` | OpenAI-compatible file uploads |
| `/schedules` | Recurring task scheduling |
| `/skills` | Agent skill definitions |
| `/memory` | Persistent agent memory |
| `/artifacts` | Agent-produced file previews |
| `/publish` | Publish artifacts to MindsHub (view.mindshub.ai in prod) |
| `/connectors` | Third-party service connections and OAuth |
| `/settings` | User preferences and API keys |
| `/runtime-credential` | Desktop hand-overs, held in memory: the MindsHub credential, and the address and launch key of the app's own browser (write-only, loopback, outside MindsHub mode) |
| `/browse` | The agent's browser: whether this deployment offers it to the caller and, on the web, a warm-up, viewer links, the person's approval of a held step and signing out of every site. See [The agent's browser](#the-agents-browser) |
| `/hub/workspaces` | Which MindsHub workspace this person is working in |
| `/hub/usage` | The caller's free Air allowance (as a proportion), balance, auto top up and credit spend, for the desktop's usage warnings. An organization with no free grant reads `freeTokens.limit` 0 and `freeTokens.resetsAt` null, since nothing refills |

### Declaring who may call a route

Every route declares its permission at its own definition, and `create_app()` refuses to start if one does not. Adding a route means picking the class that names the credential the route actually takes:

| Declaration | The credential is |
|-------------|-------------------|
| `OpenByDesign` | nothing at all. Pinned route by route, with a written reason, in `tests/test_open_by_design_pin.py` |
| `LoopbackOnly` | the caller being on this machine (`require_local`) |
| `DesktopOnly` | not MindsHub mode (`require_local_tenancy` 403s the route there) |
| `LoopbackDesktopOnly` | both of the above |
| `PlatformSignature` | the calling platform's HMAC over the body (channel webhooks) |
| `Authenticated` | a verified `Principal` |
| `AuthenticatedInOrgMode` | a verified `Principal`, in MindsHub mode only; a no-op on desktop |
| `AuthenticatedOrgAdmin` | ... plus org-admin standing |

```python
from cowork.api.v1.permissions import AuthenticatedInOrgMode, require

@router.get("/thing", dependencies=[Depends(require(AuthenticatedInOrgMode))])
```

Declare it on the `APIRouter` when every route on it takes the same
credential, with one exception: never declare `OpenByDesign` on a router.
FastAPI *adds* a route-level `dependencies=[...]` to its router's rather than
replacing it, so a router carrying the open marker hands it to every route
added later, and the walker cannot tell that apart from a deliberate choice.
Every other class refuses, so inheriting one is safe.

`COWORK_TENANCY_MODE=org python -m scripts.dump_routes` prints the current
surface with each route's declaration.

### The agent's browser

The agent drives a browser the person can watch beside the chat and take over. Which browser depends on the deployment, and this server holds no browser credential of its own (`cowork/services/browser.py`).

- **On the web (MindsHub mode)** the browser is the browser router at `COWORK_BROWSER_ROUTER_URL`, which starts one browser per person. Each `/browse` route calls it as the caller, with their own MindsHub credential. Before the producer queues any web turn, it scopes the turn's key with the router (`POST {router}/_scope`): to the task's session and the project's saved tabs, which also starts the person's browser, or, for a turn with no browser, to no session (`{"session_id": null}`), so code in the turn that holds the key can't scope it to a session of its own. The router refuses a key it hasn't scoped, and takes a scope only with `X-Browser-Scope-Key`, read on every scope from the file `COWORK_BROWSER_SCOPE_KEY_FILE` names, where the chart mounts the router's own Secret, `browser-proxy`, key `scope_key`. The job's `params.browser` is `{base_url, profile, project?}` and carries no credential: the pod reaches the router with the turn key. A turn whose scope fails runs without a browser and logs `browser_scope_failed` with the router's code and the wait in `elapsed_ms`; a person the browser isn't for (their organization switched it off, or their role lacks it) logs `browser_scope_not_entitled` at info instead. No alert reads either yet.
- **On the desktop** the browser runs inside the Cowork app. The app hands over its loopback address and per-launch key through `PUT /api/v1/runtime-credential/browser`, with the body `{"url": "http://127.0.0.1:<port>", "token": "<43 base64url characters>"}`, or both blank to clear it. This process keeps them in memory and hands an in-process turn's anton that browser. The route is loopback-only and refuses MindsHub mode, and a deployment that trusts identity headers ignores the hand-over, so one person's browser never answers another person's turn.
- **No browser** for a channel turn, which comes from a chat others can write to while the browser holds one person's sign-ins; for a person who turned `browser_enabled` off; for a task whose Browser switch is off (`"browser": "off"` on `/responses`); and on a MindsHub-signed-in desktop whose organization's admin turned the browser off. That desktop reads the switch from its MindsHub entitlements before each turn and keeps the last answer when MindsHub can't answer; a desktop on the person's own model key asks nothing.

`GET /browse/status` answers `{enabled, available, status}`. `enabled` is the person's switch. `available` says whether the deployment offers them a browser, whatever the switch says, so they can turn it back on. `status` is one of these:

- `none`, `starting` or `running`: the router's answer for the caller's own browser on the web, or `running` for a registered browser on the desktop;
- `unavailable`: no router on the web, or no registered browser on the desktop;
- `organization_off`: an admin of the caller's organization turned the browser off, on the web or on a signed-in desktop;
- `not_entitled`: the `browser` block of the caller's MindsHub entitlements (`GET /v1/entitlements/me/`, read through the same cache as their permissions) says their account may not use it. That read fails closed: when MindsHub can't say, status answers 503 `mindshub_unreachable`.

`POST /browse/provision` warms the caller's browser with their time zone and language, and answers like status. `POST /browse/embed` takes `{session_id}` and answers a single-use viewer link, `{session_id, view_url, ticket_expires_at, lease_expires_at}`, which is never stored. `POST /browse/approve` takes `{session_id, pending_id}` from the commit-hold card and records the person's approval in the browser, so the agent's one resend of that step runs; it doesn't answer the card, which the app answers after a 200. `POST /browse/decline` takes the same body when the person skips that card, and the browser drops the step and any approval already recorded for it, so a late approval can't let the resend run. `DELETE /browse/logins` signs the person out of every site in their browser. These five are web-only and answer 409 elsewhere. A refusal from the router keeps its 401, 403 or 429 and its code in `detail`; any other router failure is 424 with the router's code, since the ingress swaps a 5xx body for its own page.

The `response.browser_session_opened` event carries the session id only. Whoever opens the pane mints their own viewer link, so a stored event opens nothing.

On the web, sites see the browser's traffic come from MindsDB's own addresses; the desktop app's browser goes out on the person's own connection.

`tests/fixtures/browser_router/` pins what this server sends the router and reads back, and `tests/fixtures/browser_contract/` holds anton's fixtures for the turn block and the session event.

### The MindsHub workspace selector

`/api/v1/hub/workspaces` backs the workspace selector at the bottom of Cowork's sidebar, which Cowork draws only when the listing carries two or more workspaces: one workspace means nowhere to move to. A **MindsHub Workspace** is an org-internal container that owns MindsHub resources (API keys, artifacts, model entitlements) and lives in MindsHub. It has nothing to do with the filesystem directories this repo calls workspaces, which is why the stored key is `hub_workspace_id`.

The selector has no feature gate, and its listing reads no entitlements: it shows whenever the listing carries two or more workspaces.

Four things about it are worth knowing before changing it.

**The sidecar makes the call, not the renderer.** MindsHub's ingress allows three console origins per environment and no Cowork host, and a per-PR Cowork host cannot be added to a static allow-list because its name carries the PR number. The packaged Electron app would get away with a direct call because it runs with `webSecurity: false`; the web SPA would not. Reading here works for both shells. The outbound host is derived by `default_minds_auth_host()` from `ENV`, which the desktop propagates when it spawns this process, and the credential is the caller's own, never the stored provider key and never `minds_url`, both of which an org admin can set.

**The credential arrives in its own header, `X-MindsHub-Authorization`.** It cannot use `Authorization`: Electron's main process overwrites that on every request to the loopback server with the server's own token, so the caller's Keycloak JWT can never arrive under that name in the desktop shell. `Authorization` is still the fallback **in MindsHub mode only**, where the ingress put the caller's JWT there; on a desktop install that header holds this server's own bearer, and forwarding it to MindsHub would leak the one credential the main process scopes to the loopback origin. `mindshub_credential` reads both, and it is deliberately a different function from `caller_bearer` so a client cannot steer the credential on the org model-catalog fetch by setting a header.

Skill and artifact sharing never use `mindshub_credential`, for the same reason: a caller can put a credential from another organization of theirs in that header, and MindsHub would answer with that organization's permissions, teams and members. Sharing reads all three with `caller_bearer`. Three routes still take `mindshub_credential`: the workspace selector's `GET /hub/workspaces/` and `PUT /hub/workspaces/active`, and `GET /hub/usage/`. `PUT /active` (`set_active_mindshub_workspace` in `cowork/api/v1/endpoints/mindshub_workspaces.py`) answers 403 for a workspace missing from the listing that credential returns, so the listing decides what it stores. The pick it stores grants nothing: the menu shows it only while the listing has it (`resolve_active` in `cowork/services/mindshub_workspaces.py`), and every key mint sends it for the gateway's user and organization, where MindsHub refuses a workspace they hold no grant on and the mint falls back to the default workspace and clears the pick (`_request_turn_key` in `cowork/turnqueue/auth_keys.py`). Every MindsHub read shares one transport and one per-caller cache (`cowork/services/mindshub_client.py`).

**The listing cache is keyed on the credential, not just the caller.** MindsHub answers the listing per caller: an owner or admin sees every workspace in the organization, a member only the ones they hold a grant on. So a cache keyed on the organization alone would serve one admin's menu to every member for the whole TTL, and the grant check on `PUT /active` reads the same entry. The key is `(MindsHub host, organization, user, credential digest)` (`CallerCacheKey` in `cowork/services/mindshub_client.py`). The digest is not belt-and-braces: `user_id` comes from the gateway-set principal and is `None` on every desktop request, because `scope_from_principal` returns `LOCAL_SCOPE` outside MindsHub mode, so identity alone collapses to one shared entry and a sign-out/sign-in as another account would be served the previous one's workspaces. A new session means a new token means a new entry. Entries are swept on write, since nothing re-reads a departed caller's key and the dict would otherwise grow for the process lifetime. The TTL follows whether **MindsHub answered**, not what it said: a reachable listing keeps the long TTL, and an unreachable one the short failure budget, so a degraded MindsHub does not add a round trip to every menu open without also going stale for a long stretch.

**Two refusals on `PUT /active`, and neither may read the stored pick.** A
workspace missing from the caller's listing is a 403; one in the listing but
stamped archived is a 409, so the client can say retrying will not help instead
of offering a loop with no exit. The archived check reads the target row's own
`archived_at` rather than asking whether it is in the set the menu offered, and
that distinction is load-bearing. `hub_workspace_id` is an untagged
`UserSettings` field, so `PUT /api/v1/settings/hub_workspace_id` writes it with
no gate and no listing check, and `selectable` keeps the active row even when
archived. A refusal phrased against the offered set would therefore have been
talked into accepting an archived workspace by one call to the settings route.
Nothing that refuses a request may read a value any caller can write.

**Picking a workspace changes what the client shows, not what a turn is billed
to.** Neither turn credential carries a workspace: a desktop turn presents the
user's own session credential, whose organization comes from the token's
active-organization claim, and a cloud turn presents a minted key whose request
body has no workspace field. So nothing on the turn path reads
`hub_workspace_id`, and a test asserts that.

The pick is stored as an untagged `UserSettings` field, so it lands per `(org, user)` in MindsHub mode and in the single global row on a desktop install. That is interim: the shared per-user preference the console reads has no route in MindsHub yet, and moving this onto it is follow-up work.

### Who can read what in MindsHub mode

Single-user mode has one person, so none of this applies: every check below is inert on the desktop.

In MindsHub mode a request acts as a pair, an organization and a user, and it never carries less. The gateway authenticates the credential, asks MindsHub whether that user is still a member of that organization, and injects `X-User-Id` and `X-Organization-Id`. cowork-server validates the shape of those headers and then trusts them, so **the gateway being the only route to the pod is what makes them trustworthy**, and that is a NetworkPolicy rather than anything in this codebase. `deployment/cowork-server/templates/network-policy.yaml` is that policy, on in staging and prod, and it admits only the nginx ingress controller pods in the `infrastructure` namespace on port 9010. It is off in PR environments, which take base values, so a PR environment does not enforce this boundary and a forged-header caller inside one is served. A request with no valid pair is answered 401 before any route runs, except on `/api/v1/health/`, `/api/v1/health/live` and `/api/v1/health/ready`, which clients and the kubelet probe with no headers, and the channel webhook paths, which third parties call.

### What the lists read in MindsHub mode

The project, skill, artifact and conversation lists run the same number of database statements whatever the organization's size. The rows a statement reads still grow with the organization. Tests count the statements at two sizes: `tests/test_org_list_scaling.py` for the project, artifact and conversation lists, and `test_the_skill_list_runs_the_same_statements_at_3_and_30_skills` in `tests/test_org_skill_sharing.py` for the skill list.

- `GET /api/v1/projects/` and `GET /api/v1/skills/` read every attribution row of their kind in one query (`SharedResourceAccess.snapshots`), after they read the projects or skills. A create or a skill upload commits its claim row (`SharedResourceAccess.reserve_claim`) before the project row or the skill's folder, so every listed item's claim is in that read. A skill with no sharing row takes its owner from the same attribution rows, read once per list (`SkillService.access_rule`).
- The lists answer from committed state. While a create, upload or rename is still in flight, its item lists as pending, or under its new name with no attribution, until the change commits.
- A create that crashed leaves its claim pending. Once the claim's 5-minute lease runs out, the next list recovers it: it finalizes the item for its creator, or releases the name when nothing was written. Only an item whose claim has expired, or has no expiry, takes the locks. A skill takes the organization's project-references lock and then the skill's own lock, the order every skill writer that takes both uses. A list with no such claim takes no lock.
- Every token check under a lock reads the row as the database holds it now (`SharedResourceAccess._find_for_update` re-reads with `populate_existing`), and so does recovery. A request's session keeps the rows it read past a commit, and another replica may have recovered the same claim meanwhile. A decision from the session's earlier copy would finalize the item a second time and reset its `updatedBy`, or a skill's sharing.
- `GET /api/v1/artifacts/` reads the caller's own conversation ids once (`ConversationService.owned_ids`). It then filters each project's legacy per-conversation workspaces by name before it stats any of them, so a co-member's workspace is never opened, and it skips a workspace with no `.anton/artifacts` folder. Project-root owners come from one query and legacy workspaces' owners from one more (`resolve_artifact_owners_by_source`). Each project's artifacts folder is still opened from the trusted root, one folder at a time without following links, so the list's file system work grows with the number of projects.
- `GET /api/v1/conversations/` reads the page's projects in one query and each row's last activity, the time of its newest message, in another (`ConversationService.last_messages_at`). That read returns one row per conversation, though a turn's tool rows and its answer share one `created_at`.

### How the browser stays in one organization

Canonical Cowork web sends `X-Cowork-Expected-Organization-Id` with every authenticated browser API request. `TrustedHeaderMiddleware._organization_boundary_response` compares it with the normalized `X-Organization-Id` the gateway supplies before the route runs. It applies only to Keycloak-shaped bearer JWTs. MindsDB API keys, opaque service credentials, requests without a bearer, CORS preflights, health checks, and channel webhooks keep their existing behavior.

The boundary always refuses. A missing header returns 426 and a malformed or
changed value returns 409. Both responses carry
`X-Cowork-Organization-Reload: required`, a JSON `code` and `detail`, and
`Cache-Control: no-store`. The browser reloads instead of letting an old
document continue under a new Keycloak organization. A request already inside a
route keeps the `Principal` created at its start, so a concurrent session change
cannot retarget that in-flight operation.

There is no setting that relaxes this. The boundary once had an `audit` position
that logged a violation and served the request anyway; it and the staged rollout
it existed for are gone. An environment that still carries the retired
`COWORK_ORGANIZATION_BOUNDARY_MODE` key boots normally and ignores it. Deleting
the field is what makes it unreadable: `AppSettings` sets no `env_prefix` and
gives every field an explicit `validation_alias`, so the environment source only
looks up names a field still claims. `extra="ignore"` only decides what happens
to an unknown key arriving in a `.env` file, which is not how the overlays
deliver this one. Both halves are pinned, the settings half by
`tests/test_app_settings.py::test_stale_organization_boundary_mode_env_var_is_inert`
and the request half by
`tests/test_principal.py::test_no_environment_variable_reopens_the_organization_boundary`.

`GET /api/v1/capabilities/organization-switch` is authenticated and returns protocol version 1. It reports `expectedOrganizationEnforced: true` in MindsHub mode with identity enforcement active. Identity enforcement is in that answer because the picker should stay hidden while identity is only audited, not because it gates the boundary: `TrustedHeaderMiddleware` runs the boundary for every browser JWT request it builds a `Principal` for, in MindsHub mode alone. It reports `enabled: true` when those hold and `COWORK_ORGANIZATION_SWITCH_ENABLED=true`.

`COWORK_ORGANIZATION_SWITCH_ENABLED` is the product enable for the picker, not a
safety switch, and it is the lever to reach for to hide the picker without a
code change. `COWORK_IDENTITY_ENFORCE=audit` hides it too, by dropping
`expectedOrganizationEnforced`, but it reopens the no-principal path and leaves
the boundary refusing anyway.

Backing enforcement out needs the code and values from before cowork-server#524.
[deployment/cowork-server/README.md](deployment/cowork-server/README.md#back-out-enforcement)
records the procedure and why a Helm rollback cannot do it.

Inside one organization, two different rules apply, and which one you get
depends on the resource:

- **Shared with the organization:** projects, project files at the project root,
  skills, and project memory. Every member reads them.
- **Private to whoever created it:** conversations and their history, scheduled tasks, personal memory, uploaded files, and everything under a conversation's own workspace at `conversations/<conversation_id>/`. Live artifacts are in that last group, because the agent writes them into the conversation it is running in. Cloud OAuth connections are also user-private, held in MindsHub's Data Vault rather than here. cowork-server stores none of it and relays the caller's own credential to MindsHub for the whole lifecycle: the connect handshake and its status poll, the connection catalog, one connection's detail, the Google Picker file grant, a short-lived picker access token, and disconnect. MindsHub resolves the caller from that credential, so each route only ever reaches that user's own connections.

Shared visibility does not imply shared destructive access. Project creators and organization admins can rename or delete a project; the General project is immutable. Any member can create a skill, while only its creator or an admin can edit, disable, or delete it. Packaged skills are immutable in MindsHub mode for every role (desktop keeps its existing editable-copy behavior). The first member to write a non-empty project-memory slot becomes its author, and only that author or an admin can subsequently change it. Project instructions follow the project creator/admin rule. Empty memory writes do not claim a slot, and ordinary project files remain member-writeable.

The API derives these decisions from the trusted principal and returns typed
`attribution` and `capabilities` objects for the client. Stable user IDs drive
authorization, and they are the only identity these rows keep: no email address
is stored, and the API fills one in only when the actor is the viewer
themselves, so an attribution never discloses another member's address.
File-backed resource ownership lives in `shared_resource_attributions`, outside
the agent-writeable project tree, and every allowed protected mutation appends a
`shared_resource_mutations` row. A delete removes the current attribution but
keeps its mutation event. Unattributed legacy resources fail closed to
admin-only mutation.

The private rule is enforced by the service layer rather than by the routes:
`ConversationService._owned`, `FileService._owned_select` and
`ScheduleService._owned_select` each add `created_by == <the caller>` when a
request is org-scoped. Two places extend the same rule to the filesystem.
`_conversation_workspace_ok` in the project-file routes refuses a path under
another member's conversation directory, and `artifact_roots` drops another
member's conversation directories before the artifact list or delete ever sees
them, because those routes are addressed by project and slug and never receive a
conversation id.

One artifact at a time can leave the private group, and only its owner can put it there. `POST /artifacts/workspace/{project}/{artifact}/comments-access` records a grant in that artifact's `.revisions/draft-review.json` and mints the matching rule in MindsHub. A co-member then resolves that one artifact by id (`artifact_scope.review_artifact_for_request` searches other members' workspaces only after the caller's own, and only accepts a folder that carries the grant) and gets the draft preview and comments, never the source, the edit routes or the delete. Without a grant the answer is 404, so a private draft still cannot be told apart from one that does not exist; with a grant the owner-only routes answer 403 instead, because to a client already looking at the draft a 404 would read as deleted. The artifacts list is unaffected either way: a co-member's artifact never appears in it, so review starts from the link the owner shares.

Outside MindsHub mode the route provisions nothing and answers `enabled` with `"scope": "owner"`. Its comment routes keep each artifact's threads in that artifact's `.revisions/comments.json` (`services/local_artifact_comments.py`), and find the artifact only among the caller's own artifacts roots. In single-tenant mode, while projects are personal, those roots are the person's own partition, so another person's calls to the draft answer 404.

An artifact's journal, `.revisions/`, holds its revisions, a pending source write, agent repairs, the local comments, the draft-review grant and the lock. The agent's code can write it, so every read and write goes through `services/artifact_journal.py`: the artifact folder and every folder below it open without following a link (the folders above it were opened the same way when the artifact was found), a file the server reads must be a regular file with one name and under a size cap, and a `.revisions`, `blobs/` or `repairs/` that isn't a real folder reads as an empty journal until the first write replaces that entry, never its target, with a real folder. An open that fails for any other reason, such as running out of file descriptors, is an error, never an empty journal, so no write replaces a history it couldn't read. The turn-end capture and the local comment routes run off the event loop, so a journal read or a wait on an artifact's lock holds one worker thread, not every request on the server. Each record the agent's code can write (a manifest entry, a pending source write and a repair record) is checked by its model in `services/artifact_revision_records.py` before the server uses a field of it, and again where a field names a file. A record that holds a value JSON can't encode, such as one nested too deeply or a lone surrogate, fails its model too. A revision's text is read only as `blobs/<sha256>` with bytes that match the name. A manifest entry that names any other path, or otherwise fails its model, is left out of the history and answers 404 `Revision not found`, as a missing revision does, and the next write of the manifest drops it. A repair record that fails its model answers as a missing repair and is never written back, nor is one under any name but its own UUID. A pending source write that fails its model is abandoned. A record keeps any key its model doesn't declare and is written back with the keys it had, so a journal reads and writes the same across builds. Every journal file, and each rewrite of an artifact's `metadata.json` by the identity and export services, is replaced through `dir_atomic_write` (`common/paths.py`): the new bytes go to a file beside it, which is synced and renamed over the name, and then the folder is synced.

MindsHub-mode artifact authorization uses a separate server-owned identity. An agent can edit `metadata.json`, so its local UUID cannot reserve a global comment key. The `artifact_identities` SQL table binds each `(organization, local UUID)` to its immutable owner and a MindsHub-issued canonical key. Draft grants, publishing, and policy deletion use that key; the comments REST/SSE proxy translates the unchanged local UI key through the same table. Read requests never allocate. Existing identities are adopted only when MindsHub confirms a matching durable owner and organization binding. New issuance uses a persisted random request ID so concurrent calls and lost replies cannot allocate different identities.

This requires MindsHub's internal `artifact-access/claim/` and `allocate/` endpoints and the matching services publisher ownership checks. Apply MindsHub's ownership history migration before enabling new policy writes, then deploy these clients with the coordinated MindsHub changes. Authorization failures preserve files and stop sharing. SQL aliases survive artifact deletion and schema downgrades; they are ownership history, not a cache that can be cleared during rollback.

Filesystem authorization decisions start with a resolved path. The route carries
that decision through opening the file: every
component below the project directory is opened `O_NOFOLLOW`, and a symlink
planted anywhere in the chain is refused. A pod mounts its own workspace
read-write, so without that a swapped directory component between the check and
the open reaches another member's tree, or another organization's.

**A refusal on a private resource answers 404, not 403**, with the same body a
genuine miss returns. Telling the two apart would confirm that another member's
file exists, which is most of what an attacker wants to know. Policy refusals
that reveal nothing personal answer 403 instead: the desktop-only routes say
`not available in org deployments`, and an organization-settings write without
the admin role says so plainly.

**The HTML preview hands out a bearer token in a URL** (`preview-mount-file`
returns one, `preview-asset` spends it), because an iframe cannot send an
`Authorization` header. The token is random, it is bound to the member and
organization that minted it, and it expires after 30 minutes, so a token that
escapes into a log or a screenshot is not a way in for anyone else.

Being the minter is not enough on its own, because **a mount grants a
directory**. The gate runs on the file the caller named, and an `.html` at the
project root sits in a directory every member's `conversations/<id>/` hangs off,
so a token minted on a shared file would otherwise read every workspace under it.
A mount therefore reaches only the workspace its own file lived in, and a mount
taken on a shared file reaches no workspace at all, not even the minter's own.
That check holds no session, deliberately: `preview-asset` serves every sub-asset
a page pulls, and a session there is a database connection per image.

### The default model is the one the free allowance covers

Every minds-cloud role defaults to `mindshub_air`, for all three roles: planning,
coding and router. **MindsHub's catalog declares it**, in the
`mindshub_model_policy_v1` config that already owns the alias registry, and the
declaration arrives as a `default_for` list on each `/v1/models` row. So moving a
default is a config edit plus an apply, not a release. MindsHub Air's usage draws the included allowance, so a user who has
picked no model can finish a whole turn without the wallet being charged for any
part of it.

The two roles a user never sees are why this is the default rather than a
premium model. Planning is the model in the picker, so a wrong choice there is
visible and fixable. Coding (the completion verifier and the scratchpad) and
router (respond-versus-delegate gating and history summarization) run unseen, so
a paid default there is denied on an empty wallet with nothing on screen to
explain why.

An explicitly stored model is never rewritten by this. Paying for a better model
is a pick in the Settings picker, and a funded wallet resolves to the same
default as an empty one until that pick is made.

### A stored model the wallet cannot pay for is swapped, not rewritten

`/v1/models` marks a model the org cannot currently pay for as `enabled: false`,
and the map is cached as `minds_model_enabled`. When a stored pin is flagged
that way, `_resolved_model` resolves the role to the first affordable model in
the map instead, for all three roles. The alternative is every turn failing on a
denial the user may not be able to see.

The stored row is left exactly as the user set it, which is the load-bearing
half: the moment the wallet can pay again, the next settings load flips the
alias back to `enabled: true` and the role resolves to the original pick with
nothing to re-select.

Two cases share that path and should not be confused. A pin the map flags
`false` is a real MindsHub model that is merely unaffordable right now. A pin
**absent** from a non-empty map is foreign or retired, so it would 404 on every
turn rather than 402, and it is healed the same way with no route back.

`enabled: false` has one more cause: an admin in the organization restricted the model. `/v1/models` names the lock on each disabled row in `disabled_reason` (`model_restricted`, `wallet_empty` or `included_allowance_exhausted`), and `GET /settings/recommended-models` relays it as `modelDisabledReasons`, so the picker shows a restricted model as restricted and offers no credits for it. The map comes from the MindsHub listing only. A custom openai-compatible endpoint's `disabled_reason` is dropped, because a BYO endpoint must not be able to claim an admin restricted a model. Resolution treats a restricted pin like any other `false` one and swaps it, for the members the rule restricts and no one else. A turn that still reaches the gateway on a restricted model fails with the `model_restricted` code: the gateway keeps the 403's `permission_denied` reason and names the rule on its `X-MindsHub-Deny-Detail` header and in the body's `error.deny_detail`, trusted by the same origin rules as the reason.

A restriction belongs to one member, so it is stored per member. MindsHub applies the model rules for each caller's org, workspace and team, so two members of one org get different listings, and `fetch_org_model_catalog` caches one listing per org and member. `minds_model_enabled` is an org row that every member's resolution reads. So in MindsHub mode `recommended_models` leaves the caller's `model_restricted` models out of it: each keeps the flag the org map already held for it, or `true` when it held none. The caller's restricted ids go to their own `minds_model_restricted` row instead, an untagged setting that `SettingService` files at the caller's user scope, and `UserSettings._minds_enabled_map` reads them as `false` over the org map. One member's restriction therefore never swaps another member off a model, and another member's load never lifts it. A member's restrictions reach resolution when that member's own settings load writes them. A listing that publishes no `enabled` flags leaves the stored list alone, the same evidence rule the org map follows. A desktop install has one member, so it keeps restrictions in `minds_model_enabled`.

The listing cache (`_minds_models_cache` in `cowork/services/providers.py`) drops expired entries on every write and holds at most `_MINDS_MODELS_CACHE_MAX` listings, evicting the least recently used, so keeping one entry per member does not grow it for the life of the pod. An entry with no organization in its key, which includes every desktop fetch, is keyed by the credential it was fetched with, held as a pydantic `SecretStr` so it prints as asterisks in any repr or log, so an account switch in the running app is never served the previous account's restricted or locked rows. Nothing is hashed from it.

The desktop closes the loop at the other end: a model the map locks is not
offered in either picker, so a swap only ever applies to a pin that was
affordable when it was made. Allowing the pick meant the turn silently ran a
different model from the one the picker named.

#### Where the answer comes from, in order

`MODEL_ROLE_DEFAULTS` in `cowork/common/settings/app_settings.py` is still a real
answer, not a legacy layer. Resolution is synchronous and does no network call in
the turn path, so the catalog's declaration has to be *persisted* before it can be
read, exactly as the availability map is.

| State | What resolves | When you are in it |
| :---- | :---- | :---- |
| A model is stored for the role | that model | the user picked one, or saved Settings once |
| `minds_role_defaults` names the role | the alias the catalog declares | any install that has loaded Settings since the catalog declared it |
| Nothing persisted | `MODEL_ROLE_DEFAULTS` | a fresh install sending its first message, and any install that has never reached the catalog |

The availability map still overrides the answer in every case where the wallet
cannot pay for it, so a declared default is not a grant: it says where to start,
never what may be called.

`GET /settings/recommended-models` writes the map, from the same fetch that
already refreshes `minds_model_enabled`, and it builds the `recommendedPair` it
serves the picker through `minds_role_start_models`, the same two steps resolution
takes: the declared default replaces the compiled one, then availability adjusts
it. Those two have to agree, because the picker shows the pair as the model each
role starts on and the desktop writes it back as an explicit pin when a save
repoints a role onto MindsHub. A pair built from the compiled table, or from a
declared default the wallet cannot pay for, would pin a model that turns never
run. The pair is rebuilt from whichever map resolution will read: the live one
when the gateway published defaults, the cached one when it did not.

### Provider probes always use a model any key can call

Why a probe sends a model at all: MindsHub bills per model, so a model the wallet
cannot pay for is denied, and that denial is indistinguishable from a bad key.
Probing a paid model tells an account with an empty wallet that its working key is
invalid. `MINDS_PROBE_MODEL` (`mindshub_air`) draws the included allowance
instead of the wallet, so the result reports reachability and key validity, which
is what these endpoints are for.

Two endpoints, and they do not behave identically.

`POST /settings/validate-provider` (onboarding, and the only caller is the
onboarding screen) probes a chat completion on every branch, and takes an optional
`model`:

- `provider: "minds"` always sends `MINDS_PROBE_MODEL` and **ignores** `model`.
- `provider: "openai-compatible"` sends `model` as asked, so validating one
  specific model never reports a pass earned by a different one. Omit it against a
  MindsHub base URL and it falls back to `MINDS_PROBE_MODEL`; omit it against any
  other host and the generic openai-compatible default applies.
- `provider: "anthropic"` sends `model` or `claude-sonnet-4-6`.

`POST /settings/test-providers` (the Settings health dot) probes **per provider
type**, and only the `minds-cloud` type is a chat completion, on
`MINDS_PROBE_MODEL`. The `openai-compatible` type is a `GET {baseUrl}/models`
listing probe, so a MindsHub host configured through that card is health-checked
against a route MindsHub does not deploy everywhere; those routes answer 404 or 401
even for a valid key, which is the reason the `minds-cloud` type does not use one.

An `apiKey` of `""` or `"***"` on that route means "use the stored one", which is
how the Settings UI re-tests a provider it only ever received masked. A stored key
only goes to a host this deployment saved: the provider cards in `providers_json`,
the scalar `openai_base_url`/`minds_url`, and the vendor host an omitted `mindsUrl`
already reaches. The comparison is on origin (`scheme://host[:port]`), not the whole
URL, because the question is which party receives the key; that also means a stored
URL carrying a path still matches a body sending the host alone. A URL the guard
cannot parse names no origin, so it is refused rather than raising. Omitting the URL
stays legal: a `minds-cloud` probe then goes to the vendor host, and an
`openai-compatible` one answers `missing base URL`, which is what it did before this
guard. A key the caller supplied may go anywhere, since nothing stored is at risk.

A refused provider comes back `fail` with its reason and is never pinged, rather
than failing the whole request. Callers send every configured provider in one call,
so refusing the request would blank the other providers' dots over one bad URL.

Each card in the body is a `ProviderProbeCard` (`cowork/schemas/settings.py`), which declares the four fields the ping reads, `type`, `apiKey`, `baseUrl` and `mindsUrl`, as strings. A card whose `type` or URL is not a string fails the whole request with a 422 before any key is resolved. The Settings UI does not hit this: it sends the cards it saved itself, with `type` normalized to a string by `backfillProviders` in cowork's `settingsTransform.js`. Every other field the UI holds on a card, such as `isDefault` or `name`, is ignored, so a renderer that adds one keeps getting its status dots.

When the MindsHub gateway refuses a `minds-cloud` probe with a named reason, the response carries it in `providerStatusReasons`, keyed by provider type: `{"minds-cloud": {"code": "free_air_daily_spend_fuse_exceeded", "resetAt": "<ISO instant or null>"}}`. The code is the gateway's own `X-MindsHub-Reason` word, one of `wallet_empty`, `included_allowance_exhausted`, `free_air_daily_spend_fuse_exceeded`, `rate_limited` and `policy_unavailable`, so the Settings notice can tell a velocity limit or the free-Air fuse from an empty wallet instead of matching on `HTTP 429` in the detail string. When a proxy strips that header, the probe reads the same word from the body `code`, which the gateway sets to match. `resetAt` carries the gate's `X-MindsHub-Reset-At`, which it sends on the allowance and fuse denials. The body `code` counts only when the probe's response came from the configured `minds_url` host. The header counts unless the response provably came from another host. These are the origin rules the turn-failure mapping applies (`gateway_denial` in `cowork/handlers/turn_errors.py`), so a probe aimed at any other host names no reason. A refused provider was never pinged, so it has no entry. `providerStatus` and `providerStatusDetails` keep their shape.

Every MindsHub-bound chat probe caps the completion at `max_tokens: 20`, not 1:
some models refuse a 1-token budget and fail the probe for a perfectly good key
(see `_chat_probe`). The cap is not sent to a non-MindsHub endpoint, because
OpenAI's reasoning models reject `max_tokens` and want `max_completion_tokens`.

The desktop app has a second copy of these validators in its Electron main process
(`cowork/src/main/provider-validation.ts`, called from the `settings:validate` IPC
handler in `cowork/src/main/index.ts`); the endpoints here serve the web build.
Both copies have to change together. One asymmetry worth knowing: the desktop
MindsHub onboarding path signs in through Keycloak rather than validating a pasted
key, so main's `validateMinds` has no live caller today, and it is the
openai-compatible and anthropic validators there that a packaged build actually
runs.

### Organization permission enforcement

Hosted turn admission checks `product.execute` through MindsHub's internal `POST /internal/permissions/authorize/` endpoint. This applies to free and paid models, stored provider credentials, direct responses, remote queue submission and scheduled runs. Schedules resolve the acting identity from the stored owner and check it before creating a conversation. Queue submission rechecks current access even when reusing a previously minted credential.

The request contains the server-resolved user and organization IDs and one permission. It uses the existing `COWORK_TURN_AUTH_INTERNAL_BASE_URL` and `COWORK_TURN_AUTH_INTERNAL_SECRET`; customer headers cannot choose that host or credential. A confirmed denial returns `403` with `permission_denied`. Missing configuration, malformed replies, transport errors and service authentication failures return `503` with `permission_unavailable`. Authorization failures never fall through to another model or delegated execution. Single-user mode keeps its one-person behavior.

Artifact source edits, publishing/access changes, deletion, revision restoration
and repair management require `artifact.manage` as well as existing artifact
ownership. Starting an agent repair also requires `product.execute`. Capability
responses reflect these current grants. Generic project-file writes and deletes
apply the artifact grant to artifact storage paths too. Writes atomically replace
the selected directory entry, preserving the existing file mode where descriptor
chmod is available. This detaches a pre-existing hardlink so editing an ordinary
file cannot modify protected artifact bytes through the same inode. Failed writes
leave the original file intact and remove the temporary file.

Remote turns use the `anton_turn_v2` controller operation and declare their
workspace authority. The controller checks `artifact.manage` again when it
dequeues the turn. Without that grant, it mounts saved conversation files read
only and runs the agent in a temporary copy. All workspace edits in that turn
are temporary, including ordinary files; nothing is copied back or published.
The controller replaces warm workers when their storage authority is unsuitable.
Cowork requires the controller's verified workspace acknowledgement before
accepting worker output, and indexes artifacts only for persistent workspaces.
Deploy the companion scratchpad-controller change before enabling this producer;
older controllers fail these turns with `permission_unavailable`.

Publishing uses a separate `artifact_publish` mint purpose. It requires artifact management without granting model execution. Execution mints retain their own purpose and credential type; the publisher uses a fresh instance ID and never hands its credential to an inference client. Deploy MindsHub's decision/purpose APIs and the publisher's artifact-only authentication path before this server change. Keep these admission checks in place during rollback while restrictive custom roles remain assigned. No new customer or staff permission grants are introduced.

### Datasource grants on cloud turns

`COWORK_TURN_DATASOURCE_ENABLED` (default `false`) lets a hosted turn query the user's verified PostgreSQL and MySQL connections in the organization. When it is on, the producer lists them from MindsHub under the dedicated producer service role (`COWORK_TURN_DATASOURCE_PRODUCER_KEY_ID` and `COWORK_TURN_DATASOURCE_PRODUCER_KEY`), bound to the turn key it has just minted, registers one grant per connection before enqueueing, and puts only connection ids and credential versions on the queue. No capability or password crosses Redis; the pod reaches the gateway on the inference host the turn's `llm` block already names. A listing or registration failure fails the dispatch as `permission_unavailable` instead of silently dropping the datasources.

Turn the flag on only after MindsHub serves the datasource producer and resolver endpoints, the inference deployment serves `/v1/datasources/`, the turn controller passes the `datasource` block on, and the scratchpad image includes the typed helper that reads the gateway from the turn's inference host, with live pods from an older image recycled first. With the flag off nothing in this path runs, so the flag is also the rollback.

### Sharing selected projects

On a local deployment with trusted identity headers and personal projects, an operator can grant read-only file access to one project. Keep `projects` in `COWORK_PERSONAL_RESOURCES`. The owner alone can execute tasks, schedule work, change files or edit instructions. Recipient visibility is confined to `GET /api/v1/projects/` descriptors and the UUID-addressed `GET /api/v1/shared-projects/{project_id}/files`, `/files/{path}` and `/files-raw/{path}` routes. Ordinary project services, execution and mutation lookups remain scoped to the requester's own partition.

Run these as Cowork's user, with the owner's identity pair:

```sh
python -m cowork.ops.projects share --organization-id <deployment-id> --user-id <owner-sub> --project <slug> --recipient <recipient-sub>
python -m cowork.ops.projects share --organization-id <deployment-id> --user-id <owner-sub> --project <slug> --everyone
python -m cowork.ops.projects unshare --organization-id <deployment-id> --user-id <owner-sub> --project-id <project-uuid>
python -m cowork.ops.projects list
```

Repeat `--recipient` for multiple subjects. An explicit audience is required; resharing replaces it. `unshare` fails, changing nothing, when no grant matches the project and owner; `list` also shows grants whose project no longer exists, so they can be revoked by ID. Grants live in `<COWORK_SHARED_DIR>/shared/projects.json`, outside agent-writable project roots, and name immutable project and subject IDs. A rename retains a grant; a deleted and recreated project or account does not inherit a named grant. Everyone means this deployment only. Commands lock and atomically replace the registry, refusing to overwrite malformed or unreadable bytes. Requests fail closed on a broken registry. No database migration is required. Revoke grants before rolling back application versions.

Recipients never read dot paths, conversations, skills, memory, symlinks, hardlinks or nonregular files through these routes. Every path component is opened relative to pinned directories without following links. Listings visit at most 10,000 entries and 32 levels; text is capped at 2 MiB and downloads at 100 MiB. Downloads use attachment disposition and an opaque content type; no shared HTML preview token exists. Each request rereads the grant, so revocation affects subsequent requests. Previously downloaded bytes cannot be recalled. Single-user mode, MindsHub mode, and deployments that leave all projects globally shared keep their existing behavior.

### Sharing one skill with everyone

On a deployment that trusts identity headers and keeps `skills` in `COWORK_PERSONAL_RESOURCES`, a person shares one of their own skills with everyone, read-only, or hides it again.

- `PUT /api/v1/skills/{skill_id}/sharing` takes `{"shared": true}` or `{"shared": false}` and returns the skill. Both directions are idempotent.
- It answers 401 to a caller with no person: the desktop, or the operator view under `COWORK_IDENTITY_ENFORCE=audit`.
- It answers 409 "Sharing one skill at a time isn't available here." where this kind of sharing is unavailable: single-user mode, and skills left off `COWORK_PERSONAL_RESOURCES`. In MindsHub mode the same route sets the skill's audience instead and ignores `shared` ([Sharing a skill in MindsHub mode](#sharing-a-skill-in-mindshub-mode)).
- It answers 403 "This skill is shared with you and can't be changed here." for a managed skill's name or a skill someone else shares with the caller, and 404 when the caller has no skill by that name.
- A share answers 409 with the reason for a packaged built-in, a name the agent already has from the harness's own skills or anton's packaged ones, a managed skill's name, a skill that can't be read in full, a skill holding a link, and a name someone else already shares a skill under.
- Any change answers 503 "Sharing is busy right now. Try again in a moment." while another change holds the list for 5 seconds, and while the filesystem refuses to read the list.

Each skill's `capabilities.canShare` is true for the caller's own skill where sharing is available, and false for a packaged built-in's copy, a managed skill and a skill shared with the caller. The skills list's top-level `canShare` says whether the caller may share here at all; the UI asks "who can use it" on create only when it is true. Both are false on the desktop, where skills are left shared and for the operator view. In MindsHub mode the list's `canShare` is false, and a skill's `capabilities.canShare` is true for its owner and for an organization admin, who may change its audience.

`POST /api/v1/skills/` takes `"shared": true`, and `POST /api/v1/skills/upload` the multipart field `shared`, to share the new skill as it is created. Both default to false. A refused share answers 409 (503 when busy) and writes nothing; when the skill is written but its entry then can't be added, the new skill is removed and the error stands.

Deleting a shared skill stops sharing it. Renaming one answers 403 "Stop sharing this skill before you rename it.", because the list names a skill by its slug. A create, upload or rename into a name the person's list still has for a skill of theirs that is gone removes that entry first, so the new skill is shared only when they choose it. Once the list exists, every create, upload and rename into a new name takes the list's lock for that check: it answers 503 "Sharing is busy right now. Try again in a moment." while another change holds the list or while the filesystem refuses to read it, and writes nothing. A delete still deletes the skill then, and logs a warning naming the list it couldn't update.

Everyone else lists a shared skill after the packaged built-ins, the managed skills and their own, live from its creator's store, with `isBuiltin` false, `shared` true and no capability. A same-named skill earlier in that order hides it, and so does one the agent already has, and a listed skill that no longer exists is skipped; for a name two people share, the first entry wins. Their turns link it from the creator's store into a read-only skill root after the managed skills, with the turn's selection rule (enabled, and scoped to no project or to this project's name) applied to the skill's own metadata. Everyone else's edit, disable, rename or delete of it answers 403 "This skill is shared with you and can't be changed here."; disabling is refused too, because `enabled` lives in the skill's own `SKILL.md`. Everyone else's create, upload or rename into its name answers 409 "Someone else shares a skill named '<slug>'. Choose another name." Its creator sees it `shared` with every capability and edits, disables and deletes it as before, and everyone else sees each change.

The list is `<COWORK_SHARED_DIR>/shared/skills.json`, a JSON list of `{user_id, partition_id, skill}` in the order shared (`cowork.shared_skills`). Cowork writes it itself, under an `flock` on `skills.json.lock` beside it, replacing the file at once with mode 0644. A missing list shares nothing. A malformed one shares nothing too: every read logs a warning naming the file, the server still starts, and the next change that takes the list's lock replaces it, starting from empty: a share or hide, and a create, upload, rename or delete. A list larger than 1 MiB counts as malformed. A list the filesystem refuses to read (EIO, EMFILE, EACCES), or one that isn't a regular file (a FIFO, device, directory or symlink left in its place), is not malformed, so no change replaces it: while it lasts, readers share nothing, the server still starts, and every change logs an error naming the file, answers busy and writes nothing until an operator removes it. `python -m cowork.ops.skills share` and `unshare` change it on a person's behalf ([Operator commands](#operator-commands)).

### Sharing a skill in MindsHub mode

In MindsHub mode every person in an organization reads one skill store, so who each skill is shared with is a row beside it, in the `skill_access` table (`cowork/services/skill_sharing.py` holds the rules, `cowork/services/skill_access.py` the reads and writes). The row holds the skill's audience, its owner and `always_on`, keyed by the organization and the skill's folder name: a rename moves it and a delete removes it. Its org and owner are UUID columns, and its audience lists hold lowercase UUIDs.

| Shared with | Who sees and uses it | Who changes it (edit, rename, delete, enable) |
|-------------|----------------------|-----------------------------------------------|
| Only me | Its owner | Its owner |
| Specific people | Its owner and each person named | Its owner |
| A team | Its owner and every member of the team | Its owner and every member of the team |
| Everyone at ‹org› | Everyone in the organization | Its owner, and anyone with `skills.write_org` |

- Changing who a skill is shared with is its owner's: teams, people, and taking it back from everyone. Sharing it with everyone takes `skills.share_org` too. A team a change adds must be one the caller belongs to.
- An organization admin sees, changes and shares every skill on the skills page. Their turns use only the skills they'd see without the role.
- Built-in and managed skills are shared with everyone, read-only.
- A skill from before sharing gets a row shared with everyone, owned by its recorded creator, the first time the skill store's reader (`SkillService`) reads its organization's rows, for a skills listing or a turn's skills. The always-on check at a turn's start (`has_always_on_rows`) reads the rows without that backfill: a row the backfill writes is never always on, so the check's answer is the same. A settings row (`_skill_access_backfill_v1`) marks the organization done. A folder with no row after that is its recorded creator's alone, or nobody's with none, so only an admin sees it.
- `PUT /api/v1/skills/{skill_id}/sharing` takes any of `org_allowed`, `allowed_team_ids`, `allowed_user_ids` and `always_on`; a field left out keeps what the skill has. Every id must be a UUID (400 naming it), and a person it adds must be in the organization, asked of MindsHub in one lookup (400 naming each one who isn't).
- `GET /api/v1/skills/sharing-capabilities` answers `enabled` (MindsHub mode), `canShareSkillsOrg`, `canWriteSkillsOrg`, `canShareArtifactsOrg` (MindsHub's `skills.share_org`, `skills.write_org` and `artifacts.share_org`, all true for an admin), `permissionsKnown` and `teamsKnown`. While `permissionsKnown` or `teamsKnown` is false, MindsHub couldn't be asked, so a false capability means "couldn't ask", not "may not". It names no organization: the app names the organization from its own list of the caller's organizations.
- `PUT /api/v1/skills/{skill_id}/me` takes `{"enabled": false}` to turn a skill off for the caller alone, and `true` to turn it back on. It works for any skill the caller sees and takes no edit rights. It works in every mode; on the desktop the one person is the nil UUID (`LOCAL_PERSON_ID`).
- The caller's teams and permissions come from MindsHub (`GET /v1/me/teams/` and `GET /v1/entitlements/me/`), cached per caller credential for 60 seconds once MindsHub answers, and 15 seconds when it doesn't. So a team or a capability MindsHub takes away still counts for up to 60 seconds. While MindsHub throttles the caller (a 429), its last answer is served for up to 120 seconds after it gave it. An artifact save that adds a team reads the caller's teams from MindsHub at the save instead ([Who an artifact is shared with in MindsHub mode](#who-an-artifact-is-shared-with-in-mindshub-mode)).
- While MindsHub can't list the caller's teams, they are in no team: they see their own skills, the ones shared with everyone and the ones naming them, never one shared only with a team. A change whose answer rests on what MindsHub couldn't be asked answers 503 with `{"code": "mindshub_unreachable", "message": ...}`, never a 403. A 4xx from MindsHub is a refusal, never "unreachable".

### Always-on skills

Only a skill shared with everyone or with a team keeps `always_on`; a change that leaves it shared with neither clears it. Marking a skill shared with everyone always on, or taking that back, takes `skills.share_org`. Marking a team skill takes edit rights: its owner, a member of one of its teams, or an admin.

An always-on skill reaches the organization or the team it is shared with. A person the skill is shared with personally gets it on demand: a skill shared with people is always on demand. A packaged built-in is never always on.

The routing gate answers some plain questions itself, without the agent, and its answer would not follow an always-on skill. So a turn an always-on skill reaches never meets the gate. The request reads the caller's teams from MindsHub, and then its first database unit reads whether an always-on skill reaches the caller in the conversation's project. When one does, the turn goes to the agent with the route reason `always_on_skills`, with no gate call and no first take. The skill files that read parses are read on the file store's threads (`run_efs_unit`), apart from the database unit. When that read fails, the turn goes to the agent with the reason `router_unavailable`, as when the gate's history can't be read. A caller no always-on skill reaches keeps the gate. While none of the organization's skills is always on, the read parses no skill file.

A cloud turn's request carries every skill the caller sees, the pod's whole recallable set, as `{"skills": {<slug>: {"files": {...}, "always_on": true}}}` (`cowork/schemas/turn_skills.py`), with `always_on` left out for a skill recalled on demand. The always-on skills come first, the ones shared with everyone before the team ones, since the pod puts them in the prompt in that order within its own budget. Every turn sends the block, empty when the caller sees nothing. A browser turn and a channel turn each build it in a unit of their own on the file store's threads (`run_efs_unit`), since it seeds the built-in skills and reads each selected skill's folder. When the request is over its byte cap, the producer drops the on-demand skills first, then project memory, then the always-on skills, team ones first (`cowork/turnqueue/producer.py`).

### The people and teams reads

Both read MindsHub with the Authorization the gateway validated (`caller_bearer`), never the client-set `X-MindsHub-Authorization`. Outside MindsHub mode each answers empty and reachable. `reachable: false` means MindsHub couldn't be asked, which a client must not read as "nobody" or "in no team".

- `GET /api/v1/hub/teams/` answers the caller's teams (MindsHub's `GET /v1/me/teams/`), cached per caller as above, each with `isDefault`.
- `GET /api/v1/hub/people/` with no parameters answers the organization's directory, the caller left out, read fresh every time. MindsHub returns at most 500 members and says `truncated` when it holds more.
- `?ids=a,b` answers the members among those user ids, and `?emails=a,b` the members who hold those addresses, so the app can share with a member past the directory's cap by id. Not both at once (400). Each is asked of MindsHub 50 at a time, and one request may name at most 200 (400 above that).
- Every value is checked first, by the rule MindsHub applies, because MindsHub refuses a whole lookup for one value it doesn't take. An id that isn't a UUID answers 400 `{"code": "invalid_id", "detail", "ids": [...]}`; an address MindsHub wouldn't take (Django's `validate_email`, ported in `cowork/services/mindshub_people.py`) answers 400 `{"code": "invalid_email", "detail", "emails": [...]}`. MindsHub is not asked in either case.
- A 4xx from MindsHub answers 403 or 400 `mindshub_refused`, or 429 `mindshub_throttled`, never `reachable: false`.

### Sharing limits

`cowork/services/sharing_contract.py` holds every bound sharing applies, beside the audience names, the MindsHub capabilities, the MindsHub reads, the code of a refused share, what the agent is told after one, and how an artifact's rule was keyed before it had a stable id. A bound MindsHub owns is mirrored here so a share it would refuse is refused before anything is published. MindsHub's sharing contract module and the app's `lib/sharingContract.js` give the bounds they share the same names, and the upload service gives `LEGACY_RULE_USER_DIR_CHARS` and `REPORT_ID_PATTERN` the same names. MindsHub, the upload service, the app and anton's publisher give the refusal code the same name, and anton's publisher gives the sentence its agent is told the same name; `tests/test_sharing_contract.py` pins every name and value against `tests/fixtures/sharing_contract.yaml`.

| Limit | Value | Name | Owned by |
|-------|-------|------|----------|
| People one skill or artifact is shared with by id | 100 | `MAX_PEOPLE_PER_RULE` | MindsHub (an artifact's rule); this server applies it to skills |
| Teams one skill or artifact is shared with | 100 | `MAX_TEAMS_PER_RULE` | MindsHub (an artifact's rule); this server applies it to skills |
| Addresses one artifact save asks MindsHub to email | 100 | `MAX_NOTIFY_EMAILS` | MindsHub |
| Artifact title the share email names | 200 characters | `MAX_NOTIFY_TITLE_CHARS` | MindsHub |
| Artifact link the share email carries | 2048 characters | `MAX_NOTIFY_URL_CHARS` | MindsHub |
| Ids, and addresses, per MindsHub org-member lookup | 50 each | `MAX_IDS_PER_LOOKUP`, `MAX_EMAILS_PER_LOOKUP` | MindsHub; the app mirrors both |
| Members the directory returns | 500 | `MAX_DIRECTORY_ROWS` | MindsHub |
| The longest address MindsHub takes | 320 characters | `MAX_EMAIL_CHARS` | MindsHub (its address check) |
| The longest id an artifact's rule is stored under | 512 characters | `MAX_ARTIFACT_ID_CHARS` | MindsHub (the `artifact_id` its artifact-access routes take) |
| Hex digits of the owner's folder in a report's old id | 9 | `LEGACY_RULE_USER_DIR_CHARS` | The upload service |
| Ids, and addresses, per `GET /hub/people/` request | 200 each | `MAX_IDS_PER_REQUEST`, `MAX_EMAILS_PER_REQUEST` | This server |
| The `ids` and `emails` query values of `GET /hub/people/` | 8000 and 64800 characters | `MAX_IDS_QUERY_CHARS`, `MAX_EMAILS_QUERY_CHARS` | This server |

### Names across the app, this server and MindsHub

| Concept | On the wire | In this server |
|---------|-------------|----------------|
| Everyone in the organization | `org_allowed` (`orgAllowed` in responses); "Everyone at ‹org›" in the UI | `AudienceField.ORG_ALLOWED` |
| Teams | `allowed_team_ids` (`allowedTeamIds`); with people, "Specific teams and people" in the UI | `AudienceField.ALLOWED_TEAM_IDS` |
| People in the organization | `allowed_user_ids` (`allowedUserIds`) | `AudienceField.ALLOWED_USER_IDS` |
| People outside the organization | `allowed_emails` in a save, `accessEmails` in the access state; anton's publisher calls them `emails` | `AudienceField.ALLOWED_EMAILS` |
| Nothing shared | "Only me" in the UI | an empty audience |
| The capabilities | `canShareSkillsOrg`, `canWriteSkillsOrg`, `canShareArtifactsOrg` | `skills.share_org`, `skills.write_org`, `artifacts.share_org` |
| A share with everyone refused without `artifacts.share_org` | 403 `{"detail": "<the reason>", "code": "share_org_required"}` | `SHARE_ORG_REQUIRED_CODE`, `ShareOrgRequired` (`cowork/services/publish.py`); the agent is told `SHARE_ORG_REQUIRED_INSTRUCTION` |
| MindsHub couldn't be asked | 503 `mindshub_unreachable` | `MindsHubUnreachable` (`cowork/services/mindshub_client.py`) |
| The cloud turn's skills block | `skills`, each entry `files` and `always_on` | `TurnSkillEntry` |

### Who an artifact is shared with in MindsHub mode

In MindsHub mode MindsHub's rule for an artifact is the one record of who it is shared with: everyone in the organization (`org_allowed`), teams, people in the organization by user id, and anyone else by address. The artifact's `.published.json` sits in a folder the pod can write, so it keeps the access mode, the password and the versions, never the audience.

- `GET /api/v1/artifacts/workspace/{project}/{artifact_id}/access` reads the rule from MindsHub's `POST /v1/internal/artifact-access/read/` (`cowork.services.publish.stored_artifact_rule`), under the artifact's identity first. A report published restricted before it had a stable id, and not published since, still has its rule under `<user_dir>/<report_id>`, where `user_dir` is the upload service's folder for the owner (`LEGACY_RULE_USER_DIR_CHARS` in `cowork/services/sharing_contract.py`). While the identity holds no rule, that one is read, so the save check, the dialog and every re-publish see the audience the report already has. When the entry's `report_id` isn't in the upload service's form (`REPORT_ID_PATTERN`), or makes the old id longer than MindsHub takes (`MAX_ARTIFACT_ID_CHARS`), no old id is read: the upload service refuses to publish a report of another form, MindsHub holds no rule under an id that long, and the pod can write the `report_id`. MindsHub moves that audience to the stable id only for the report first published under it, so a member re-saving their own artifact already shared with everyone keeps it shared, and nothing else takes its audience. An identity MindsHub reserved before it recorded first reports takes no old audience, so a member's save, live sync or autopublish of such an artifact shared with everyone is refused with `share_org_required` until an admin shares it or the owner narrows it. A loose file has no identity, and this server sends the upload service no key for it, so its rule is read under the old id alone. Neither read asks a key the entry recorded, a loose file's `artifact_key` included: the pod can write those, and a rule under one may be another artifact's, so its audience would be sent as this one's.
- `PUT .../access` checks a save against that rule: sharing the artifact with everyone (`org_allowed`) takes `artifacts.share_org`, and a team it adds must be one of the caller's, by the caller's teams read from MindsHub at the save rather than the 60-second cache, so a team deleted a moment ago is refused before anything is published (an admin's team they aren't in must be one of the organization's, read the same way). While MindsHub can't list the caller's teams the save is a 503, and while it throttles the caller a 429. The org's default team is never sent: naming it means everyone (`org_allowed`), so a save that adds a team while the caller's teams can't be read is a 503, an admin's included. When a save changes the people, everyone it names is asked about in one org-member lookup. A person it adds who isn't a member is a 400. A person on the rule who has left the organization is dropped, because MindsHub keeps a departed person's id and refuses it on the next write that sends it. The caller's own address among the emails is a 400, and so is a `notify` naming more than 100 addresses or 100 people, the most MindsHub emails from one save. All of this is answered before anything is published.
- A save that would newly share the artifact with everyone without `artifacts.share_org` answers 403 `{"detail", "code": "share_org_required"}` and publishes nothing. `detail` is this server's own sentence when its check refuses the save, and MindsHub's reason when MindsHub refuses the rule's write and the upload service passes that on. A reason over 500 characters keeps only its start and its end, as any upload-service text does. The desktop's `POST /api/v1/publish/` and `/update` answer MindsHub's refusal the same way, and the agent's publish tool reports `share_org_required` and tells the model not to publish wider instead. MindsHub's refusal is logged once, as a warning with `status_code` 403 and `reason` `share_org_required`, and no traceback. A live sync it refuses, such as a desktop entry that still shares with everyone after the rule stopped, adds no record of its own.
- The artifact list (`GET /api/v1/artifacts/`) and a chat's inline cards don't say who a restricted artifact is shared with: its `orgAllowed`, `ownerOnly`, `allowedTeamIds`, and `allowedUserIds` are null, and it carries no `accessEmails`. Only `GET .../access` reads the rule. A public or password card's audience fields are `false` and `[]`. On the desktop, the cards and `GET /api/v1/artifacts/status` read the audience from `.published.json`, `ownerOnly` included.
- A save reads the rule once: the check's read is the rule the publish plans against. Every other restricted publish reads the rule too, so a live sync or an autopublish re-sends what the rule holds and writes no team or person.
- The access mode a live sync or an autopublish keeps comes from `.published.json`, so it is checked against the rule. Only a restricted publish leaves a rule, and a public or password publish deletes it. The pod can write `metadata.json` too, which moves the artifact's identity, so the rule is read under that identity, under each key the entry recorded for its last publish (`artifact_key`, `lambda_artifact_key`), and under the report's `<user_dir>/<report_id>` id. A key MindsHub refuses as an artifact id (blank once stripped, longer than `MAX_ARTIFACT_ID_CHARS`, or holding a NUL or a lone surrogate) holds no rule and isn't asked about. A re-publish whose entry isn't restricted while any of them holds a rule has a mode changed after the owner's last save. A re-publish of a page the entry names no key for can't be checked, because the pod can write the `report_id` that names the old id. Either is refused before anything is uploaded and logged as `artifact_access_mode_changed`, with `reason` `rule_held` or `no_rule_key`. The owner's next save of the artifact's sharing ends it. A key MindsHub binds to another owner, or holds for repair, fails the read (409): the rule there may be this artifact's. A page with no identity of its own, one in a subfolder of its artifact such as one the metadata's `primary` names, or one whose `metadata.json` an agent deleted, publishes with no key, and the keys left to check are ones the pod can write. The upload service refuses such a publish of a report that adopted a stable key (409), so an agent that points that page's entry at another report gets nothing published, while a page that has always published with no key keeps publishing.
- The owner's own save also records the mode it publishes in, and that the artifact is published, in the `artifact_owner_choices` table, where the pod can't write (`cowork/services/artifact_owner_choice.py`). Every other re-publish is checked against the narrowest record under the keys above, and refused before anything is uploaded when it would open the artifact wider: public over password over restricted (`reason` `wider_than_owner_choice`), or published while the record says unpublished (`unpublished_by_owner`). An artifact its owner hasn't saved since the table existed has no record, and keeps the rule check alone.
- Before its upload, the owner's save narrows the record to the narrower of it and the new choice, and records the new choice once the upload is live, so the record is never wider than the owner's last confirmed choice. When MindsHub's upload service refuses the save before it changes who can open the artifact or what it serves, the save puts the record back to what it held before. Those answers are 401, 402, 403 (`share_org_required` included), 409, 410, 413 and 429, judged by status alone, plus anton's refusal of a state schema change, which comes before the request. Any other ending can follow the upload service's writes, so the record stays narrowed: a 400, since a full-stack bundle can be refused after its files are uploaded, a 5xx, a timeout, a connection error, or a local error inside anton's publish, before or after its request. So does any refusal from a publisher that isn't MindsHub's (`is_mindshub_publish_url`), since its order of writes is unknown. Every owner save stamps the row (`modified_at`) before its upload, even when the mode stays, and every write of the row reads it under its lock. The restore runs only while the row still holds its own narrowing's stamp, so it never undoes another save's write, the same mode included, nor a save that started since and may have changed what is live. A restore that can't reach the database leaves the record narrowed and logs a warning (`event` `artifact_owner_choice_restore_failed`), and the save answers with the upload service's refusal.
- What these checks still can't catch:
  - An artifact its owner hasn't saved since the table existed has no record, so its re-publishes keep the rule check alone, and that check can't see a password or unpublished entry an agent rewrites to public, since neither leaves a rule. Such an artifact is covered once its owner saves its sharing.
  - A password changed inside a password entry: the mode stays password, and the record holds the mode alone.
  - An agent that points the metadata id and both entry keys at keys that hold no rule and no narrower record, such as another of the owner's public artifacts.
  - An unpublish: MindsHub mode has no unpublish of its own besides deleting the artifact, so no record says unpublished, and the `unpublished_by_owner` refusal applies only once a route records one (`record_owner_choice` with `published=False`).
- A read that fails fails the request or the publish (503) rather than guessing. A 409 or 403 from MindsHub gets the same answer on every try, so it answers 409.

MindsHub has to serve that read before this server is deployed.

### Publishing with authenticated local users

On a local deployment with `COWORK_TRUST_IDENTITY_HEADERS` and a custom `ANTON_PUBLISH_URL`, the server records each published page's first tester and resolved source path in the database. Only that tester can update or unpublish that page from the original artifact, or delete that artifact while its `.published.json` names the live page; another tester gets 409, “This artifact was published by another owner.” This applies whether projects are shared or personal. Copies and edited `.published.json` files confer no ownership: publishing a different artifact or an unknown page id creates a new link, and unpublishing it or deleting its artifact clears only that artifact's local entry. The original page stays unchanged. Same-owner updates retain their link and password unlock, also after the server renames the project or moves the task that holds the artifact. A publish, unpublish or delete of a live page with no authenticated tester, such as a shared schedule's turn or a call with the shared token alone, is refused: the API answers 401 and the agent's publish tool reports the failure.

After schema migration and before readiness, a one-time backfill imports only provable owners from server-side publish history: a unique tester identity from trusted database rows, an existing source inside that partition's recorded project, and a matching page id and URL in the current publisher inventory. A page id that another artifact folder's top-level `.published.json` also names stays unclaimed, since the history row of its first publisher may have rotated out, and so does a page published from a subfolder of an artifact. Shared-project history contains no tester identity; ambiguous, conflicting, missing or moved sources remain unclaimed. The next publish of an unclaimed entry creates a new link and leaves its old link untouched, and the server logs `orphaned_publish` with `reason=unrecorded_owner`. History retains only the latest 100 successful publishes, so this cannot recover every old link. An unreadable history or unavailable publisher inventory leaves the backfill unfinished without stopping startup: each protected publish, unpublish or delete retries it and answers 503 until it completes. A project rename or task move before then can leave an old page unclaimed. Ownership imports and the completion marker commit together.

Ownership rows survive unpublish, restart and a schema downgrade. Database failure refuses a protected publish or delete before calling the publisher. This protects requests through Cowork; the operator's publisher credential still controls the publisher directly. A shared-project member's agent can still rewrite an artifact's `.published.json`: that can leave the owner unable to manage the page through Cowork until its id is written back, and can change the access the owner's next update sends, but it cannot change or remove the page itself. Hosted publishing and single-user desktop publishing retain their existing authorization.

### Live apps

With anton's `ANTON_LIVE_APP_SLOTS` set, a full-stack artifact runs as a live app in an app slot outside this server, and only the person who launched it opens it, at the slot's own address. anton launches it (`anton.core.artifacts.live_app_slots`) from a turn in this process and calls the turn's host (`cowork.services.live_apps.CoworkLiveAppHost`) around each launch: the host stops the slot's running launch record before the launch, and starts a new one naming the launcher once the slot accepts, so a launch that fails leaves nobody able to open the slot; anton then calls the host's `launch_failed`, and the app that launch stopped is reported as a stop the person hears of (`launch_failed`). Live apps open, share and copy by the person's `sub` and verified email, which the fronting proxy sends as `X-User-Id` and `X-User-Email`. A caller without an email (or with one under 3 or over 254 characters) gets 403 "Live apps need an email address on your account. Ask the operator to add one." from the routes that change something and an empty list from the others, and a turn without one is neither offered live apps nor handed a host.

Launch admission is shared across server workers. The host holds one PostgreSQL session advisory lock, or a file lock beside the SQLite database, while Anton rechecks the slots, resets the selected slot, launches, and records ownership. Approval cards and app probes run outside the lock. A competing launch that cannot acquire admission within 30 seconds asks the person to retry. Cancellation releases admission only after any in-flight database write or lock operation finishes.

When Anton refuses a launch because every slot belongs to someone else, the server remembers that person's wait. A fresh empty-slot status records one durable notice for every waiting person before another launch consumes the capacity. A stopped report alone is insufficient while the slot is restarting. Notices survive reloads and remain advisory: they reserve no slot and create no queue. A successful launch clears only its launcher's previous wait.

- `GET /api/v1/live-apps/<n>/access` answers the edge in front of slot `n`'s address, which sends the visitor's `Cookie` and nothing else. It answers 200 when the viewer sign-in (`COWORK_LIVE_APPS_VIEWER_LOGIN_URL`, its `/oauth2/userinfo`) names, by `sub` and by email, the launcher of the app the slot runs now, by the slot's running record and by the slot's own `GET /status`; 401 for a visitor who isn't signed in; 403 for anyone else; and 503 when the sign-in or the slot can't answer. The owner never comes from the request, and no reply passes on the sign-in's `Set-Cookie`.
- `GET /api/v1/live-apps/<n>/stopped-page` is the page the edge shows at that address instead of a bare 403, 502 or 503: for its launcher, the app by the name its card shows, and when and why it stopped (`12:04 UTC` today, `19 Oct 12:04 UTC` before), with whether another person's app runs there now, or the address it runs at now; a plain refusal for anyone else.
- `POST /api/v1/live-apps/<n>/stopped` takes slot `n`'s report that its app ended: `idle` or `max_time` with the hours, `app_exited`, `terminated` or `reset`, and when it ended, which the record keeps within the launch's own life. It takes no credential, since a slot holds none to send: the report is believed only from slot `n`'s own address, with no forwarding header, and only for the boot id of the slot's running record, or of one the slot's status already stopped for the same reason, whose hours and time the report fills in. A `reset` changes nothing: after a claim the record is already stopped, and otherwise the slot's next boot marks a record it still runs lost.
- `GET /api/v1/live-apps/mine` lists the latest launch of each of the caller's apps, newest first, running or stopped, with the reason as a code and a sentence. A running app whose slot is on a later boot, or that the slot list no longer has, is marked stopped, `lost`: at the slot's new boot time where the slot gives it, else when it is found. A slot that shows the app's boot exited, with the reason it keeps in its status, stops it for that reason; one that still runs it, or exited without a reason yet, keeps it running until its report says why it ended.
- `GET /api/v1/live-apps/availability` returns the authenticated person's `{waiting, notice, scope}`. A notice contains an `id` and `message`; `scope` is an opaque namespace for that person's browser state. Poll this route within one minute even when `/mine` is empty. A waiting person's poll checks real slot readiness when no launch holds admission, and records availability for all waiting people, while returning only the caller's notice. The notice's message names the time the slot was seen free (`14:05 UTC` today, `19 Oct 14:05 UTC` before).
- `POST /api/v1/live-apps/availability/<notice_id>/ack` acknowledges the caller's notice and returns 204. Another person's id, or an older refusal generation, changes nothing. The existing `/mine` response remains an array.
- `POST /api/v1/artifacts/live-app-share` shares one of the caller's full-stack apps with one address (`{"projectId", "slug", "email"}`) or with everyone (`{"projectId", "slug", "everyone": true}`). `GET /api/v1/artifacts/live-app-shares?projectId=&slug=` lists the caller's shares of that app, newest first, and `DELETE /api/v1/artifacts/live-app-share/<shareId>` withdraws one of them (404 "This share no longer exists." for anyone else's); copies already made stay. `GET /api/v1/artifacts/shared-live-apps` lists what others shared with the caller's address or with everyone, each app once, with the owner's address; an app whose folder is gone or can't be read drops out.
- `POST /api/v1/artifacts/live-app-duplicate` copies a shared app into a project of the caller's own once the caller confirms the disclaimer (`"confirmed": true`, JSON true only), and answers the copy's card. The copy runs with the caller's connections. It takes `backend.py`, `requirements.txt`, `state_manifest.json`, `prd.md`, `spec.md` and every regular file under `static/`, read without following links (at the owner's project folder too), and gets a new `metadata.json` from anton's store with the app's name, description, type and declared connections, an empty provenance and a null port. It never takes the log, the publish state, the owner's port or any other file, and a link, a hard-linked file, a missing `backend.py`, a folder it can't read or a file over anton's launch caps refuses the whole copy with 422 naming the path. A copy whose received mark can't be recorded is removed again. Each artifact card then carries `receivedFrom`, the owner's address for a copy, from the server's own record of it. Moving a conversation to another project takes its app's shares, received mark and launch records along with the folder.

Every one of these routes answers 404 unless live apps are on, before it reads the request, so a malformed one does too. The three slot routes pass the bearer and identity checks, since their callers carry neither, and each checks its caller itself. A turn's project block then tells the agent to build full-stack apps with `generate_artifact` and give their address in backticks, and names the person's running apps, each stop they haven't heard of (a turn that fails hands it back to the next), and the copies in the project not launched yet.

## Configuration

Configuration is read from the database (`UserSettings` table) and can be managed through the Settings UI in the desktop app or via `PUT /api/v1/settings/`.

The `favicon` setting stores a browser tab icon as a data URI, separately from the sidebar's `nav_logo`. It defaults to an empty string, which keeps the page's original icon. Saving an empty string restores that default. Like the sidebar logo, this is a personal preference scoped to each member within an organization; outside MindsHub mode the instance shares one setting. `GET /api/v1/settings/` returns the data URI without masking it.

The `show_figure_sources` setting (on by default) is read only by the UI, which marks each figure in an answer with the steps it came from while it is on. The server finds and saves every answer's `response.figure_sources` whatever the setting, so turning it back on shows the marks on earlier answers too. Like `favicon`, it is a personal preference in an organization, and outside MindsHub mode the instance shares one setting.

Environment variables fall into these groups:

**Server-level** (`COWORK_*`) — control the cowork-server process itself. Each one also reaches the desktop sidecar through `<COWORK_HOME>/.env`, which the server reads below the process environment, so a value written there turns a setting on for the desktop too, except in org tenancy, where the server reads no `<COWORK_HOME>/.env`. The two `COWORK_EXPORT_*` PDF settings are the exception: the server reads them from the process environment only.

| Variable | Default | Description |
|----------|---------|-------------|
| `COWORK_LISTEN_PORT` | `26866` | Server port |
| `COWORK_SERVER_HOST` | `127.0.0.1` | Bind address |
| `COWORK_DRAFT_PORT` | `0` (off) | MindsHub mode. Port of the draft-preview listener: a second server in this process that serves only signed draft links, on the address in `COWORK_SERVER_HOST`. Links are issued (by the API) and served (by this listener) only with `COWORK_TENANCY_MODE=org` and all six draft values set; otherwise the listener answers 404 and web previews stay on `srcdoc`. `/health` reports which one as `draft_links`. |
| `COWORK_DRAFT_HOSTS` | unset | Comma-separated host names of the draft host (e.g. `draft.mindshub.co`). The listener answers 404 to any other `Host`. |
| `COWORK_DRAFT_ORIGIN` | unset | Origin draft links point at, e.g. `https://draft.mindshub.co`. |
| `COWORK_DRAFT_LINK_KEYS` | unset | Draft-link signing keys as JSON `{kid: secret}`, from the `mindsdb-secrets` Secret. Two kids can be live during a rotation. A secret shorter than 32 characters is ignored. A pod reads it at start. |
| `COWORK_DRAFT_LINK_KID` | unset | The kid in `COWORK_DRAFT_LINK_KEYS` that signs new draft links. |
| `COWORK_WEB_ORIGIN` | unset | Origin of the cowork web app, e.g. `https://cowork.mindshub.ai`: the only page allowed to frame a draft (CSP `frame-ancestors`). |
| `COWORK_BROWSER_ROUTER_URL` | unset | MindsHub mode only. The browser router's base URL, `https://` with an optional path, for example `https://cowork.mindshub.ai/browser`; the trailing slash is dropped. Unset, the web app offers no browser and makes no browser call. Ignored outside MindsHub mode, where the desktop app runs its own browser. See [The agent's browser](#the-agents-browser). |
| `COWORK_TENANCY_MODE` | `local` | `local` is single-user mode, the desktop sidecar (one person, no organization, no identity headers), or single-tenant mode with `COWORK_TRUST_IDENTITY_HEADERS` ([Server modes and terms](#server-modes-and-terms)). `org` is MindsHub mode, the cloud deployment, and turns on everything in [Who can read what in MindsHub mode](#who-can-read-what-in-mindshub-mode) above. |
| `COWORK_DEPLOYMENT_KIND` | unset | `self-hosted` tells the agent this installation runs in the operator's environment. Requires local, in-process turns and a branding pack. Unset keeps desktop and hosted identity and installation guidance. |
| `COWORK_DEPLOYMENT_BRAND_FILE` | unset | Absolute path to the selected, read-only `pack.json`. With `COWORK_DEPLOYMENT_KIND=self-hosted`, its `name` becomes the agent's product name. Startup accepts the branding pack loader's JSON encodings, including a UTF-8 BOM and UTF-16/32. It refuses missing, malformed or linked files, duplicate JSON keys and invalid names. The name is fixed until restart; member settings cannot change it. |
| `COWORK_SURFACE` | inferred | Set `web` for a self-hosted browser installation. Without an override, MindsHub mode reports `web`, and single-user and single-tenant mode report `desktop`. It sets trace attribution and, without `COWORK_DEPLOYMENT_KIND`, the agent's location line, so `web` makes the agent say it runs in the browser. With `COWORK_DEPLOYMENT_KIND=self-hosted`, the deployment identity's browser line applies instead. It does not select a deployment identity. |
| `COWORK_IDENTITY_ENFORCE` | `enforce` | MindsHub mode and single-tenant mode. `enforce` answers 401 to a request carrying no valid identity headers. `audit` logs it and lets it through, and has to be asked for. |
| `COWORK_ORGANIZATION_SWITCH_ENABLED` | `false` | Shows the canonical-web organization picker. Requires MindsHub mode and identity enforcement. The product enable, not a safety switch: the expected-organization boundary refuses a mismatched tab whatever this says. Long-lived Helm environments set `true`. |
| `COWORK_TRUST_IDENTITY_HEADERS` | `false` | With `COWORK_TENANCY_MODE=local` only, where on is single-tenant mode. Reads `X-User-Id` and `X-Organization-Id` on a request that passed the bearer token, and gives each pair its own partition. The partition id is `uuid5(X-Organization-Id, X-User-Id)` (`cowork.principal.personal_partition_id`); new rows carry it as `org_id` and the user as `created_by`. Only for a deployment behind a proxy that verifies each person, strips client copies of those headers (and of `X-User-Email`, which the server reads as the person's address and live apps trust; the proxy sets it only from the verified token's email), and refuses a person's write (`PUT` or `DELETE /api/v1/settings/<key>`) of an operator-pinned setting: that write lands on the global row every person reads, and nothing here refuses it. Where `COWORK_PERSONAL_RESOURCES` leaves out `memory`, the proxy must also refuse a person's `PUT` or `DELETE` under `/api/v1/memory/`: that write lands on the store every person reads, and nothing here refuses it. `COWORK_IDENTITY_ENFORCE` applies: `enforce` answers 401 without a valid pair, and `audit` serves a request with no pair as the operator view, which lists every person's conversations, running answers, projects, files, schedules and pins, and reads skills, connections, artifacts and memory from the deployment's own stores; every artifact route given a person's `project_id` there (the listing, a delete by id, drafts, workspace and review) answers 404. A pair that is present but broken (one header alone, or an id that is not a UUID) answers 401 under either mode. Paths outside `/api/`, the web UI's page shell, name no person and log nothing. A path with a decoded `.` or `..` segment answers 400. Only a genuine CORS preflight (`Origin` with `Access-Control-Request-Method`) passes the bearer check without the token, so an `OPTIONS` request can't name a person either. With it on, every connection a save writes records its creator, the person's `sub`, in the bookkeeping field `_created_by`: a create stamps it, an update keeps the stored value whatever the request says, and the operator view stamps none. It never reaches the agent's environment or the connections API's fields. Every skill created through the server (create or upload) records its creator's `sub` in its `SKILL.md` metadata as `created_by`, never a value the request or the uploaded file carries; an edit, rename or project-rename rewrite keeps it, and the operator view, the packaged built-ins and the managed skills record none. The server refuses to start with it on in MindsHub mode, with `COWORK_REQUIRE_AUTH` off, with `COWORK_TURN_BACKEND=remote`, whose turns keep memory in each project's own folder rather than each person's or the deployment's tiers, with `COWORK_CODE_SANDBOX` off, and without `COWORK_AUTH_TOKEN` in its environment. With it on, the token acts as any person, so the server never generates it or copies it into `<COWORK_HOME>/.env`, and it removes a `COWORK_AUTH_TOKEN` line an earlier start wrote there. The desktop sidecar reads it from `<COWORK_HOME>/.env` like every setting here, and the desktop has no Landlock, so turning it on there stops the sidecar from starting. |
| `COWORK_PERSONAL_RESOURCES` | all nine | With `COWORK_TRUST_IDENTITY_HEADERS` on, the comma-separated resources each person keeps in their own partition: `conversations`, `projects`, `files`, `skills`, `connections`, `schedules`, `pins`, `memory`, `settings`. A resource left off stays shared across the deployment, as with the setting off. `memory` left off makes the deployment's own memory one store that every person's turns read and learn into, rules, lessons and profile included: its global tier is `COWORK_MEMORY_DIR`, and each project's tier sits under it at `projects/<project id>/`, never in the project's `.anton/memory`. Each turn reads the shared global tier and its current project's tier, never another project's. The server starts, and logs a warning at start that memory is shared (or, with `MEMORY_ENABLED` false, that no turn learns and the shared store is kept). Everyone's Memories listing (`GET /api/v1/memory/`) then holds the shared global tier and every project tier in that store, the caller's own projects included, each under its project id alone: a person sees the learned tier of a project they can't open without its name or files, and gets no capability on it. `projects` requires `conversations`, and `conversations` requires `files`, `pins` and `schedules`; the server refuses to start when the list breaks that, and an unknown name fails settings load. A personal schedule runs as its creator. With `memory` personal, a person's global tier is `<shared>/<partition>/memory/users/<sub>/` and each of their project tiers sits under it at `projects/<project id>/`, never in the project's `.anton/memory`; a turn with no person reads and writes no learned memory. With schedules shared (`memory` alone listed, for one), a scheduled run has no person, so it reads and learns no memory: a shared run that used its creator's memory would show it to everyone. Deleting a project removes its tiers: the deleter's where projects are personal, every person's where they are shared, and the project's tier in the deployment's store, whichever way memory is kept now, since a tier from before a switch stays on disk. Moving `memory` onto or off the list moves and merges nothing: each person's tiers and the shared store are separate folders, and each comes back as it was. With `settings` personal, the operator-pinned keys (the model, its endpoint and credentials, memory, the per-turn budgets, and how the agent acts: `act_first` and `proactive_dashboards`) still resolve from the global row, then the environment, and a person's own row for them is ignored. Whatever the list holds, a person's `PUT` or `DELETE` of one of those keys rewrites that global row for everyone, so the fronting proxy must refuse it. With `projects` personal, every artifact route resolves a path, a preview token, a project name or an artifact id only inside the caller's own projects, and opens each artifacts folder from the outermost of the server's roots without following links, as `COWORK_CODE_SANDBOX` opens each folder it grants, so a person's partition folder, `projects` store or project replaced by a link to someone else's answers 404 or an empty list. So does a project recorded while the person's partition folder was such a link: it names a folder in the other person's partition, and keeps naming it once the link is gone; the server logs an `artifact_folder_refused` warning with the project's id. A preview counts a backend as running only when this server launched it for that project and artifact, the preview proxy forwards a request only to that live backend (503 otherwise), and publish history is per person. A person's open and reveal (`/artifacts/open`, `/artifacts/reveal`) are refused whatever the list holds. Publishing and export work. Export (`POST /api/v1/artifacts/workspace/<project>/<artifact id>/export`) writes a new Word file beside the artifact's primary file, or a new HTML file for a Markdown or text primary, and never overwrites a file. It builds that file only from the artifact's own folder, opens no network connection, and leaves out every image or stylesheet the source names outside that folder. With `projects` shared, deleting a project answers 409 while another person still has conversations or schedules in it, the creator's delete included. With `projects` personal, a person's delete answers 409 too while the operator view has left a conversation or schedule in their project. With `projects` shared and `conversations` personal, a conversation's workspace (`conversations/<id>/` in the project folder) stays its owner's: anyone else's read, write, delete or listing answers 404. Choosing an existing folder as a project (adopting it) is refused on this deployment whatever the list holds. With `skills` personal, nothing fans skill links out of the shared store; each turn seeds the read-only builtins into the person's store and links that store's skills into its project, and the skills others share with the person ([Sharing one skill with everyone](#sharing-one-skill-with-everyone)) into a read-only root after the managed ones. With `skills` shared, everyone uses every skill in the one store, and a person changes, disables, renames or deletes only a skill they created: anyone else's change answers 403 "This skill is shared with you and can't be changed here.", and the skills API marks it `shared` and gives them no capability. The operator view changes any skill there, one with no recorded creator included (made before creators were recorded, by the operator view, or a packaged built-in's copy). Skills name their projects by name, and a project rename rewrites those names in the renamer's skill store, with one exception: with `projects` personal and `skills` shared, a person's rename leaves the shared store alone, because a shared skill scoped to a name applies to every person's project of that name. With `skills` personal and `projects` shared, a rename rewrites only the renamer's own skills. Local OAuth sign-in is refused (403) on this deployment whatever the list holds, the operator view included: the provider's redirect to the callback carries no identity. Connect the service with its credentials instead. With `connections` personal, a turn and an artifact's live backend read only the person's own vault and the connections others share with them (`COWORK_MANAGED_DIR`). With `connections` personal and `projects` shared, trusted-identity sandbox startup refuses the configuration: shared venvs and saved cell state can run one person's code with another person's credentials. Enable personal projects before startup. The live-backend preview and proxy also retain their own 409 refusal for this combination. With `connections` shared, everyone uses every connection, and a person changes or deletes only a connection they created: anyone else's change answers 403 "This connection is shared with you and can't be changed here.", and the connections API marks it `managed` and `shared`; one with no recorded creator stays `managed` only. Where two connections there have names that map to the same DS_ environment prefix (for example `sales-db` and `sales_db`), a person's listing and turn keep their own and hide the other, with a warning in the log. The operator view changes any connection there, one with no recorded creator included (saved before creators were recorded or by the operator view). A turn with no person, such as a scheduled run where schedules are shared, changes none: its agent can't relabel, overwrite or delete a saved connection, and one it saves records no creator. With `connections` personal or shared, a connection the agent saves from chat updates only one the person can change: one whose identity fields (for postgres, the database alone) match a connection shared with them, or someone else's, is saved as a new connection of their own. Ignored while `COWORK_TRUST_IDENTITY_HEADERS` is off. |
| `COWORK_CODE_SANDBOX` | `off` | How the agent's code runs (`cowork.code_sandbox`). `off`: every scratchpad cell, venv check, package install and artifact check starts as a child of the server, with the server's environment and files, as on the desktop. `landlock`: each one starts through anton's launcher, which locks it with Landlock, a Linux kernel feature, and hands it an allowlisted environment without `COWORK_AUTH_TOKEN`, `DATABASE_URI` or `MINDS_API_KEY`. A cell then reads, writes and runs code only in the conversation's project folder and the asker's own `HOME` and `TMPDIR` (`home/` and `tmp/` in `<shared>/<partition>/sandbox/`, each person's own whatever `COWORK_PERSONAL_RESOURCES` leaves shared, or `<COWORK_HOME>/sandbox/` for the operator view); reads and writes `<COWORK_HOME>/shared-files`, which the server creates at start and every person's agent shares, but runs no file there by its path, so nothing one person leaves there starts as a program in another's cell (an interpreter can still run a script it reads from there, such as `python script.py`); and only reads the turn's skill folders (the asker's own, the ones shared with them, and each skill in the managed folder's `skills/`), the conversation's attachment folders, and the system's read-only files. With `skills` and `projects` both left off `COWORK_PERSONAL_RESOURCES`, each project's `skills/` links into the one shared skill store, so every cell reads that whole store, read-only. Everything else is denied: the rest of the person's partition, every other person's, `<COWORK_HOME>`'s own files, the rest of the managed folder, `/home` and the shared `/tmp`. The launcher grants what a path resolves to, so the server opens each folder it grants inside `<COWORK_HOME>`, the shared root, or the projects, files or skills root from the outermost of those roots without following links; a root kept on another disk is set by its own path, not linked into `<COWORK_HOME>`. A link on the way, which agent code from before the sandbox may have left, fails the turn when it leads to the project, `HOME`, `TMPDIR` or `shared-files` (a link at `shared-files` itself fails the start), and leaves a skill or attachment folder out of the grants. A turn also refuses a project reached through a link before writing into it, makes a missing project folder again without following links, refuses an `ANTON_CONTEXT_DIR` outside the project, and makes its own folders in the project (`skills/` and the folders in `.anton/`) without following links. With `off`, those folders are made as on the desktop. The server itself isn't locked, so a link or a hard-linked file in a person's data vault, the memory tiers a turn hands anton (the deployment's shared ones where memory is shared), or the publish store fails that read or write (the turn, the connection's save or the publish), naming the path: agent code from before the sandbox may have left it there to send the server's writes into another person's space. Everyone else reading the connections that person shares goes on without them, and the server logs an error naming the vault and each connection left out. Whatever `COWORK_CODE_SANDBOX` says, the artifact routes `/api/v1/artifacts/serve`, `preview`, `preview-mount` and `preview-asset` answer 404 for a file with a second name, a hard link, as the drafts route does, and export answers 422 for such a primary: a cell can still hard-link a file inside its project, such as `.anton/.env`, into an artifact folder. The project file routes (`files`, `files-raw` and the project `preview-asset`) and the artifact workspace source answer 404 for the project's own `.anton/.env` under a second name, and keep serving other hard-linked files, such as a package store's or a backup tool's. A publish refuses an artifact holding a file with a second name. These refusals stop a second name for `.anton/.env`, not a copy: the agent's own code reads the file and gets its keys as environment variables. With `COWORK_TRUST_IDENTITY_HEADERS` on and `MEMORY_ENABLED` false, a turn hands anton no stored tier, so no tier is checked. The agent's prompt names `shared-files` and says every person's agent reads and changes it. A credential probe hands its candidate credentials to the cell as its `DS_*` variables instead of a file, both the flat names the probe's prompt gives and the namespaced ones anton's connection listing gives, and one with no conversation runs in `probes/` beside the asker's `HOME`. App backends are refused (a backend preview answers 409 "App backends are off on this deployment because agent code runs in a sandbox.", the preview proxy answers 403 and forwards nothing, and the agent is never offered one unless live apps run it outside this server, [Live apps](#live-apps)), and every Code mode route answers 403. Required with `COWORK_TRUST_IDENTITY_HEADERS`. The server refuses to start with `landlock` in MindsHub mode, with `COWORK_TURN_BACKEND=remote`, on an anton without the launcher (`anton.core.backends.sandbox`) or whose policy model refuses the policy this server builds, on a kernel below Landlock ABI 3 (Linux 6.2; Debian 12 needs the bookworm-backports kernel, or Debian 13), with a link at `<COWORK_HOME>`'s `.env`, `.master_key`, `streams`, `tmp` or `shared`, with a link at a person's partition folder in the shared root or at one of its stores (`projects`, `files`, `memory`, `data-vault` and the rest), and when its startup self-test can't confirm that a path outside a grant is denied. With trusted identity headers, personal `connections` require personal `projects`: startup refuses the unsafe combination before probing the kernel. Shared project venvs and saved cell state can execute one person's code with another person's credentials. When both projects and connections are explicitly shared, startup still logs the shared-execution warning. |
| `COWORK_MANAGED_DIR` | unset | The operator's managed folder, which Cowork only reads and every turn gets, whoever asked it. `memory/` holds `rules.md` and `lessons.md` (and an empty `profile.md`): anton's Cortex reads them as a read-only layer ahead of the turn's own memory and never writes them, a turn with no person included. `skills/<slug>/SKILL.md` are skills every person lists as built-in, with no capability; they follow the harness's own skills in anton's read-only skill roots, a person's same-named skill is never linked into a turn, and every change to a managed name (create, upload, rename into, edit, disable, delete, share) answers 403 "This skill is shared with you and can't be changed here." They are the skills the operator adds, which nobody created. A managed skill folder the server's user can't read after the start is left out of the listing and every turn, logged with the `chmod` fix, and keeps its name reserved. `data-vault/` holds the operator's own connections in the vault's own file format: every vault read lists them first, marked `managed` in the connections API, a same-named connection of the reader's own stays hidden, and so does one of the reader's own whose name maps to the same DS_ environment prefix, and every change to one answers 403 "This connection is shared with you and can't be changed here.", a delete before its OAuth revoke. `shared/connections.json` lists the connections people share, a JSON list of `{user_id, partition_id, engine, name}` that `python -m cowork.ops.connections share` and `unshare` write and the server reads live. Where connections are personal, everyone else reads each listed connection after their own, live from its creator's vault, marked `managed` and `shared` in the connections API. A same-named connection earlier in that order (the operator's, then the reader's own) hides it, and so does one whose name maps to the same DS_ environment prefix (for example `sales-db` and `sales_db`), which is logged as a warning; a listed connection that no longer exists is skipped. Everyone else's change to it (delete, access mode, picked files, token, label, or a save of that engine and name) answers 403 with the same sentence, before any OAuth revoke. Its creator sees it `shared` but not `managed` and changes, reconnects and deletes it as before, and everyone else sees each change, a refreshed token included. Once its creator deletes it, its entry shares nothing, and a new connection of theirs under that name answers 403 "This name is still shared for a connection of yours that is gone. Ask your operator to stop sharing it before you use the name again." until the operator unshares it. Blank means unset. The server refuses to start with it set in MindsHub mode or with `COWORK_TURN_BACKEND=remote`, whose workers receive none of it, when the folder or any of `memory/`, `skills/` and `data-vault/` is missing (it names each one), when the server's user can't read and search `memory/` (every turn would fail reading its rules), when it can't read and search `skills/` or read a managed skill folder or its `SKILL.md`, when it can't read and search `data-vault/` or open a managed connection file read-only (it logs `managed_connection_unreadable`, with the stored filename in `managed_connection_file` for a file failure; see [Server diagnostics](docs/structured-logging.md)), when `shared/connections.json` is malformed (it names the file), and when the installed anton's `Cortex` takes no `managed_hc`. The connection check opens files without reading their contents, changes no ownership or permissions, and adds no warning when all connections are readable. Repair permissions on the host, then restart Cowork to repeat the check. A missing `shared/` folder or file shares nothing, and a registry broken after the start shares nothing and logs an error until it is fixed. The desktop sidecar reads it from `<COWORK_HOME>/.env` like every setting here. |
| `COWORK_SHARED_DIR` | `~/.cowork` | **MindsHub mode and single-tenant mode only.** Root of the org-keyed tree: `<shared>/<org_id>/{skills,memory,projects,files,data-vault}` (and `publish` for a personal partition), where a personal partition's `<org_id>` is its partition id. Where skills are personal, it also holds `shared/skills.json`, the skills people share, which Cowork writes ([Sharing one skill with everyone](#sharing-one-skill-with-everyone)); its server user needs write access to `shared/`. In MindsHub mode, point it at the durable mount: on the default the data is ephemeral (boot warning). The server resolves its links once at startup, to tell which partition a project's folder sits in without a file store call per request, so a link that changes while the server runs takes effect at the next restart. |
| `COWORK_PROJECTS_DIR` | `~/.cowork/projects` | Project storage root (outside MindsHub mode). Set it by its own path: artifact routes open every folder below the outermost of the server's roots without following links, so a folder between `COWORK_HOME` and this root replaced by a link leaves its projects' artifacts out, and the server logs an `artifact_folder_refused` warning. |
| `COWORK_FILES_DIR` | `~/.cowork/files` | Uploaded files root (outside MindsHub mode) |
| `COWORK_SKILLS_DIR` | `~/.cowork/skills` | Skills store root (outside MindsHub mode) |
| `COWORK_MEMORY_DIR` | `~/.cowork/memory` | Memory store root (outside MindsHub mode) |
| `COWORK_VAULT_DIR` | `~/.cowork/data-vault` | Connector credential vault |
| `ANTON_LIVE_APP_SLOTS` | unset | anton's live-apps switch, the app slots' control addresses, comma-separated, read through anton (`live_apps_enabled`). With it set, the agent builds full-stack apps and anton launches each into a slot outside this server ([Live apps](#live-apps)). The server refuses to start, whenever it is set or anton reads live apps as on, with `COWORK_CODE_SANDBOX` off, `COWORK_TRUST_IDENTITY_HEADERS` off, `projects` left off `COWORK_PERSONAL_RESOURCES`, an anton without `anton.core.artifacts.live_app_slots`, whose `live_app_slots` lacks a name this server uses (the switch, the slot addresses, `SlotStatus`, `LiveAppLauncher`, `LiveAppHost`, `LiveAppLaunchError`, the launch list and its caps, `MAX_REPLY_BYTES`), whose `LiveAppHost` lacks `launch_lock`, `slots_full` or `slots_available`, or whose `ChatSessionConfig` takes no `live_app_host`, an entry that doesn't parse (a blank one included), `COWORK_LIVE_APPS_VIEWER_LOGIN_URL` that isn't an `http://` or `https://` address with a host, no query or fragment and no space around it, or `COWORK_LIVE_APPS_LAUNCH_TOKEN` shorter than 32 printable ASCII characters or with a space around it. |
| `COWORK_LIVE_APPS_VIEWER_LOGIN_URL` | unset | With live apps on, the viewer sign-in (oauth2-proxy) the access check asks who a visitor is, for example `http://viewer-login:4180`: its `/oauth2/userinfo`, with the visitor's cookie, never through a proxy variable, trusting `SSL_CERT_FILE` or `SSL_CERT_DIR` for an `https://` one on a private CA. |
| `COWORK_LIVE_APPS_LAUNCH_TOKEN` | unset | With live apps on, the token anton sends each slot with a launch, and this server with each status read; a slot holds only its SHA-256. With `COWORK_CODE_SANDBOX=landlock` the agent's code never gets it, because anton's launcher passes no `COWORK_*` name. |
| `COWORK_PUBLISH_TOKEN` | unset | Outside MindsHub mode, the bearer the server sends the publish service (`ANTON_PUBLISH_URL`) for every share, unpublish and version change, in place of the provider's key, whatever the provider. MindsHub mode ignores it: there each publish carries a per-user key the server mints. Without it, the server sends `openai_compatible_api_key` when `openai_base_url` is a MindsHub API host (`api*.mindshub.ai`), and `minds_api_key` otherwise. Set it for a publisher you run yourself: on a MindsHub endpoint the agent's code holds the model key as its `OPENAI_API_KEY`, so a publisher that takes that key takes calls from the agent's code too. With `COWORK_CODE_SANDBOX=landlock` the agent's code never gets this variable, because anton's launcher passes no `COWORK_*` name. With `off`, a cell inherits the server's whole environment, this variable included. Blank means unset. |
| `COWORK_OPENAI_COMPATIBLE_API` | `chat_completions` | The API an `openai_compatible` provider's planning, coding and router roles call. `responses` moves all three to `{base}/responses` through anton's openai flavor, where OpenAI and Azure accept function tools together with a reasoning effort. It needs an anton that reports `RESPONSES_TRANSPORT_READY`; with an older anton all three roles stay on chat completions, and Cowork logs one warning that says so. On the Responses path the agent loop runs without web tools, because OpenAI's hosted `web_search` reads the web from the provider's side, outside the deployment's egress controls. A Python cell's `web_search()` is separate and unchanged. The router's gate, first take and history summaries share this selection. Gemini is unaffected. Read once at start. |
| `COWORK_EXPORT_PDF_ENGINE` | unset | `render` turns on PDF export through an optional HTML-to-PDF render service at `COWORK_EXPORT_RENDER_URL`; unset, no artifact offers PDF and no other library makes one. Read from the process environment only, never from `<COWORK_HOME>/.env`, which `POST /settings/raw` can write on the desktop. Any other value stops the server's start. |
| `COWORK_EXPORT_RENDER_URL` | unset | The render service's `http://` or `https://` base address, with no query or fragment; the server posts each PDF export to `<URL>/v1/pdf`, with no proxy and no redirect, and waits at most 60 s in all. An `https://` service needs a certificate from a public certificate authority: the server checks it against the `certifi` bundle and ignores `SSL_CERT_FILE` and `SSL_CERT_DIR`, like the proxy variables. Read from the process environment only, like `COWORK_EXPORT_PDF_ENGINE`. With the engine `render`, a missing or non-http(s) URL stops the server's start. |

A failed router check delegates to the agent and logs the event `router_fallback` at WARNING, with `outcome` `delegated_agentic`, the `reason` (such as `router_timeout` or `router_unavailable`), and the `provider` and `conversation_id` when available, without a prompt or provider response body. It also covers failures before the check starts, such as a missing model, unavailable history or a failed read of the turn's always-on skills. Intentional delegation logs `router_route` at INFO. Every check that runs also logs `router_gate_decided` at WARNING with its `outcome`, `reason` and `duration_ms`. The gate keeps its two-second first-event, five-second idle and ten-second total budgets; selecting Responses does not make a slow model meet them. A provider failure also logs `router_call_failed` with the exception class as `error_type` and the HTTP status as `status_code` when available, without the provider's text. These are fields of the structured log line ([Server diagnostics](docs/structured-logging.md)), not text in its message.

With `COWORK_DEPLOYMENT_KIND=self-hosted`, outside MindsHub mode and with local turns, each server process also starts one background router probe after loading its configuration. It sends a synthetic greeting with the gate's function tool through the effective router model, API and effort, with a 30-second response deadline and separate two-second cleanup waits (up to eight seconds in all). It executes no tool and creates no Cowork conversation. The call can consume provider tokens; the normal transport's retries and provider retention settings still apply.

Look for the events `router_startup_probe_passed`, at INFO, and `router_startup_probe_unverified`, at WARNING with its `reason`, `provider`, `error_type` and `status_code` when available, without provider bodies, credentials or exception text. A refused request, timeout, empty response or incomplete stream never prevents startup. A pass confirms only this synthetic request, not the gate's latency budget or model quality, and the result may arrive after the first user request. Restart after changing the router configuration to recheck it. Single-user mode and MindsHub mode do not run this probe, and health checks do not call a model.

**Database pool**: these read their bare names, with no `DATABASE_` prefix. A SQLite engine is built without them and waits SQLAlchemy's 30 s for a connection outside a database unit; its database units run one at a time, and `POOL_TIMEOUT` still bounds a unit's waits.

| Variable | Default | Description |
|----------|---------|-------------|
| `POOL_SIZE` | `20` | Connections the pool keeps open. |
| `MAX_OVERFLOW` | `20` | Connections the pool may open beyond `POOL_SIZE` under load. Database units take one of `POOL_SIZE` plus `MAX_OVERFLOW` slots before they ask the pool. A negative value means no limit, for the pool and for units. |
| `EFS_UNIT_SLOTS` | `16` | Database unit slots the units that also work on the shared file store (`run_efs_unit`) may hold at once, out of `POOL_SIZE` plus `MAX_OVERFLOW`. Such a unit holds its connection while the file store answers, so units stuck on a slow store never hold every connection and database units keep the rest. Keep it below `POOL_SIZE` plus `MAX_OVERFLOW`. SQLite gives file units no unit slot, so it ignores this. It must be at least `1`. |
| `POOL_TIMEOUT` | `5` | Seconds a database unit may wait in all, for a slot and then for a connection, before it is refused: a request with `503` and a `Retry-After` of the same number of seconds, a streamed answer with `server_busy`. On Postgres it is also the pool's own wait, so every route that reads the database answers `503` with `Retry-After` once it passes, `/api/v1/health/` included. `/api/v1/health/live` and `/api/v1/health/ready` read no database and keep answering. It also bounds the wait for a shared file store slot (`run_efs`, see "Work on the shared file store"): a request that gets none in time answers `503` with the same `Retry-After`, and a download whose headers are already out waits instead. It must be at least `1`; a lower value stops the server at startup. |
| `STATEMENT_TIMEOUT_MS` | `0` | PostgreSQL ends any statement on the app's own connections that runs longer than this many milliseconds, and the failure logs `sqlstate=57014`. `0` leaves it off. Migrations never carry it: the `db-migrate` init container opens its own connections, and the server's startup migration turns both timeouts off for its own transaction. SQLite ignores it. |
| `IDLE_IN_TRANSACTION_TIMEOUT_MS` | `0` | PostgreSQL closes an app connection that sits idle inside an open transaction for longer than this many milliseconds, and the next statement on it fails with `sqlstate=25P03`. `0` leaves it off. SQLite ignores it. |
| `SLOW_STATEMENT_LOG_MS` | `0` | Logs a `db_statement_slow` warning for each PostgreSQL statement that takes at least this many milliseconds, with its `duration_ms`, its verb and first table as `operation`, and a `statement_fingerprint` of its SQL text. The line never carries the SQL or its parameters. `0` leaves it off. The hosted chart sets `1000`, so the two timeouts above can be chosen from what hosted statements take. |

**Shared file store slots**: these read their bare names too, and `POOL_TIMEOUT` bounds the wait for a slot.

| Variable | Default | Description |
|----------|---------|-------------|
| `EFS_SLOTS` | `100` | Threads `run_efs` may hold at once per process (see "Work on the shared file store"). A call stuck on the mount keeps its thread, so this is also the most threads a stalled mount can take. It must be at least `1`. |
| `EFS_SHARE` | `10` | Of those, the threads one person may hold at once, or one organization's work that runs for no person. Nothing caps an organization: with the defaults, ten busy people fill the slots. It must be at least `1`. |

**Harness-level** (`ANTON_*`) — configure a specific agent harness. These are read by the harness adapter, not by cowork-server core. They use the harness prefix because the upstream agent library (anton) defines them:

| Variable | Harness | Description |
|----------|---------|-------------|
| `ANTON_PUBLISH_URL` | Anton | Artifact publish endpoint. Pointed anywhere but MindsHub's service, it takes HTML and Markdown only: sharing a full-stack app is refused before anything is uploaded, because that upload carries the app's connection credentials |
| `ANTON_SKILLS_ROOT_DIR` | Anton | Skill file storage |
| `ANTON_GLOBAL_MEMORY_ROOT_DIR` | Anton | Global memory files |
| `ANTON_WRITE_APPROVAL` | Anton | Off by default. On, anton holds an agent cell that changes connected data, or deletes, renames or moves an existing file, until the person who asked approves it. anton reads it through its own settings, so set it in the server's process environment. The server refuses to start with it on and remote turns (`COWORK_TURN_BACKEND=remote`), questions off (`COWORK_ASK_USER_ENABLED=false`), an anton without the `write_approval` setting, or a value anton's own settings read as off, such as one only in a `.env` file anton doesn't read. It refuses remote turns and questions off also when only anton's own settings read it as on, from a `.env` file anton reads that this server doesn't, such as `~/.cowork/.env` with `COWORK_HOME` set elsewhere |

**By field name** (outside MindsHub mode only): every `UserSettings` field also reads the environment variable named after it, and a stored row wins over the variable. The router's reasoning effort is set this way, beside `PLANNING_REASONING_EFFORT` and `CODING_REASONING_EFFORT`:

| Variable | Default | Description |
|----------|---------|-------------|
| `MEMORY_ENABLED` | `true` | Whether the agent reads and learns memory. The key is operator-pinned, so on a deployment that trusts identity headers a stored global row wins over the variable for every person. There `false` turns learned memory off, whether memory is kept per person or shared: every turn gets two empty learned tiers and the operator's managed rules, runs with memory mode off and no background memory passes, and writes nothing; the Memories listing is empty, and a change or delete of a slot under `/api/v1/memory/` answers 409; and stored memory stays on disk, unread, until memory goes back on. Elsewhere (single-user mode, MindsHub mode) `false` only turns anton's memory mode off, and stored memory is still read. |
| `ROUTER_MODEL` | the provider's default | The model that writes history summaries. Only for your own inference provider (Anthropic, OpenAI, Gemini or an OpenAI-compatible endpoint): on MindsHub, summaries already run on MindsHub's default model, and Settings no longer shows this choice once turns are routed by Jev or a decision model of your own. It applies to the router's provider (`ROUTER_PROVIDER`, by default the planning provider). The route gate runs on the provider's fast default whatever this says, except on `openai_compatible`, which has no default, so there it is the gate's model too. A model picked in Settings is stored and wins over the variable. |
| `ROUTER_REASONING_EFFORT` | unset | The router model's reasoning effort. The router's provider serves the route gate, first take and history summaries, so the effort reaches all three. On `openai_compatible` the gate runs `ROUTER_MODEL`. On other providers the gate runs the provider's default router model, so the effort is sent only while `ROUTER_MODEL` is that default. Unset or empty sends no effort, which a model that doesn't reason needs, because it refuses any effort. Choose an effort the model accepts on the selected API. Some models require Responses for function tools and reject `none`; Cowork sends the configured effort unchanged and never substitutes one after a refusal. The composer's per-task effort never applies to the router. |

In Docker/Lightsail deployments, the container also receives `ANTON_MINDS_API_KEY`, `ANTON_OPENAI_API_KEY`, etc. — these are consumed by the Anton agent library directly (not by cowork-server settings), and are injected by the provisioning lambda via cloud-init user-data.

### Operator commands

On a deployment that trusts identity headers, an operator with a shell where cowork-server runs acts on one person's space with `python -m cowork.ops.<module> <action>`. Every action takes `--organization-id`, the deployment's id the proxy sends as `X-Organization-Id`, and `--user-id`, the person's `sub` the proxy sends as `X-User-Id`, except the memory actions where memory is shared, which act on the deployment's one store and refuse both, and `connections list`, which takes neither to enumerate all stored spaces. cowork-server works out the space from them with `cowork.principal.personal_partition_id`, as it does for the person's requests. Results print on stdout as plain text, and errors and log lines on stderr; a command that can't do what it was asked exits with status 1. Every connection `share` and `unshare` takes `--registry-stdout`: it then writes nothing, prints the new registry on stdout (exactly the bytes it would have written) and its message on stderr, so a caller can check the registry and write it itself. A skill `share` and `unshare` take no such flag and no `--managed-dir`: they change the shared-skill list as the person's own Skills page does, so they run inside Cowork's container as the user Cowork runs as, and refuse root, whose `shared/` folder or lock file would stop Cowork's own writes.

| Command | What it does |
|---------|--------------|
| `python -m cowork.ops.memory print ...` | Where memory is kept per person (`memory` in `COWORK_PERSONAL_RESOURCES`), prints what the person's agent learned, as their Memories page lists it: the global tier, then each project tier, with the folder each is in, and refuses a call without both `--organization-id` and `--user-id`. Where memory is shared, takes neither (either one is refused) and prints the deployment's one store: the global tier, then each project tier, titled `Project <project id>`. It ends with `Nothing learned.` when every tier is empty. While `MEMORY_ENABLED` is false it prints only that memory is off and stored memory is kept, unread, until memory goes back on. |
| `python -m cowork.ops.memory clear ...` | Where memory is kept per person, removes everything the person's agent learned, in every project of theirs (`cowork.services.memory.reset_person_memory`). Where memory is shared, takes neither person argument (either one is refused, so a clear meant for one person never clears everyone's) and removes the shared store's global slots and every project tier in it (`cowork.services.memory.clear_deployment_memory`), and no person's tier or managed file. A link where the store's `projects/` folder should be stops print and clear with an error before anything is read or removed. It can't be undone. While `MEMORY_ENABLED` is false it refuses and removes nothing, so stored memory stays as it is. The person arguments are checked first in every case. |
| `python -m cowork.ops.skills share ... --skill <slug>` | Shares the person's skill `<slug>` with everyone, read-only, on their behalf: adds it to `<COWORK_SHARED_DIR>/shared/skills.json`. Nothing is copied: everyone else reads the skill live from the person's store, and the person still edits, disables and deletes it, but can't rename it until it is hidden. They can hide it again from their Skills page. It runs the checks and prints the refusal texts of a share from the Skills page ([Sharing one skill with everyone](#sharing-one-skill-with-everyone)), plus "User `<sub>` has no skill named `<slug>`." for a skill the person doesn't have and "Skill `<slug>` of user `<sub>` is already shared." for one already shared. |
| `python -m cowork.ops.skills unshare ... --skill <slug>` | Removes the person's entry for `<slug>` from `<COWORK_SHARED_DIR>/shared/skills.json`, whether or not the skill still exists, so it also clears the entry of an account the identity provider no longer holds. The person's skill stays theirs, in their store. Refused when the person shares no skill by that name. |
| `python -m cowork.ops.skills snapshot ... [--project <name>]` | Lists the skills the person's agent sees in a project (`general` unless named), the skills others share with them included, with where each comes from (`own`, `host`, `managed`, `shared` or `anton`), and prints one `sha256`: the sha256 of the lines `<sha256 of the file>  <slug>/<path in the skill>`, one per file of each skill, sorted by slug and path, as `sha256sum` prints them. Every file the agent can read counts, helper scripts and reference files included; a skill's `stats.json`, hidden files and links don't. A shared skill's lines hash the files in its creator's store, so the digest changes when its creator edits it. |
| `python -m cowork.ops.connections list [--organization-id <id> --user-id <sub>]` | Read-only JSON inventory of stored connection metadata on a POSIX trusted-identity local deployment. Without the pair, includes every stored partition, even deleted accounts; with it, selects that partition plus managed and base stores, and opens no other partition's records. Reports engine, name, creator when recorded, partition, origin, visibility and override notes. No credentials, hosts or fields are printed. Inactive stores and stale shares remain visible. Linked, non-regular, malformed or oversized records it opens refuse the whole command without printing record contents. |
| `python -m cowork.ops.connections share ... --name <name> [--engine <engine>] --managed-dir <folder>` | Shares the person's saved connection named `<name>` with everyone, read-only: adds it to `<folder>/shared/connections.json`, where `<folder>` is the managed folder, writable by the caller. Nothing is copied: everyone else reads the connection live from the person's vault, and the person still changes it. Names are unique per engine only: one Google account connected to Gmail and to Calendar saves two connections under one name, and `--engine` picks one. Refused where `COWORK_PERSONAL_RESOURCES` leaves connections shared, when the person has no connection by that name, or more than one and no `--engine` (the message names `--engine`), when it is already shared, when someone else already shares one of that engine and name, when the managed `data-vault/` holds one, and when the registry is malformed, which the command never rewrites. |
| `python -m cowork.ops.connections unshare ... --name <name> [--engine <engine>] --managed-dir <folder>` | Removes the person's connection named `<name>` from `<folder>/shared/connections.json`. The person's connection stays theirs, in their vault. Refused when the person shares none by that name, and when they share it for more than one engine and no `--engine` is given. |

### Connection inventory and cutover

The inventory describes stored connections, not one person's picker. `visibility` is `private`, `shared`, `operator`, `inactive` or `stale`. `operator` marks a base-vault record while connections are personal and `COWORK_IDENTITY_ENFORCE` is `audit`: no person reads it, but a request with no identity pair is the operator view, which reads and changes it. `inactive` means nothing reads the store: the base vault under `enforce`, and personal vaults while connections are shared. `shared_as` lists the registry's engine/name pairs that reach each stored connection. Names such as `sales.db` and `sales_db` reach the same vault file, so an older share remains active after such an update; `registry-name-alias` marks this case. The `notes` identify managed-prefix overrides, same-vault and shared-layer prefix collisions, recipient-owned overrides and duplicate registry entries, using the registered names for shared-layer precedence. Managed connections take priority over personal and shared entries. Two names that normalize to the same `DS_*` prefix can conflict even when their spellings differ.

Adding `connections` to `COWORK_PERSONAL_RESOURCES` switches readers to personal vaults. It leaves old base-vault records stored but unread by testers; under `audit`, the operator view still reads them. Preserve explicitly chosen team connections in the managed vault, or have their owners recreate and share them. Do not copy private connections to every person. Enable `projects` too and preserve needed shared-project files before changing that namespace. The new startup refusal deliberately prevents private connections alongside legacy shared executable state; removing that guard does not make such a configuration safe.

The inventory reads up to 1 MiB per record or registry and follows no partition, vault or record links. It never creates a missing store. Run it between connector changes for a stable operator review; it is not a transactional snapshot. A missing creator stamp is `null`, while the stored partition still identifies the space. One person's listing reads only that partition's records. For other people's shares it reads only the vault's file names, and a vault holding a link or a file with two names shares nothing, as at runtime. So a bad record elsewhere doesn't fail it, and the full listing is the check for every store. The command does not infer current identity-provider membership.

## Docs

- [docs/DESIGN.md](docs/DESIGN.md) — Architectural overview and design decisions
- [docs/MIGRATION.md](docs/MIGRATION.md) — Migration guide from the legacy server
- [docs/MIGRATION_PROGRESS.md](docs/MIGRATION_PROGRESS.md) — Migration status tracker

## License

See [LICENSE](LICENSE).

## Diagnostics

See [structured logging](docs/structured-logging.md) for defaults, troubleshooting
overrides, retained evidence streams and rollout requirements.
Shared-memory startup warnings state whether turns learn into the shared store
or leave it unread, without logging its private path.
