Metadata-Version: 2.5
Name: shidoshi
Version: 0.15.0
Summary: A local, AI-native notebook for iterative work with LLMs and coding agents
Project-URL: Homepage, https://github.com/tdchaitanya/shidoshi
Project-URL: Repository, https://github.com/tdchaitanya/shidoshi
Project-URL: Issues, https://github.com/tdchaitanya/shidoshi/issues
Author-email: Chaitanya <tdchaitanya@gmail.com>
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: agents,ai-assistant,anthropic,ipynb,llm,notebook,openai,openrouter
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Libraries
Requires-Python: >=3.13
Requires-Dist: anthropic<1,>=0.121
Requires-Dist: claude-agent-sdk==0.1.81
Requires-Dist: cryptography>=46
Requires-Dist: deepagents<0.8,>=0.7.5
Requires-Dist: fastapi>=0.115
Requires-Dist: filelock>=3.20
Requires-Dist: ipykernel>=6.29
Requires-Dist: jsonschema>=4.18
Requires-Dist: jupyter-client>=8.6
Requires-Dist: keyring<26,>=25.7
Requires-Dist: langchain-anthropic<2,>=1.5.4
Requires-Dist: langchain-openai>=1.5.0
Requires-Dist: langchain-openrouter>=0.2.7
Requires-Dist: langgraph-checkpoint-sqlite<4,>=3.1.1
Requires-Dist: mcp<2,>=1.21
Requires-Dist: mistune<4,>=3.2.1
Requires-Dist: nbdime<5,>=4.0.4
Requires-Dist: nbformat>=5.10
Requires-Dist: openai>=2.41.1
Requires-Dist: pillow>=12.2.0
Requires-Dist: platformdirs>=4.10.0
Requires-Dist: psutil<8,>=7.2.2
Requires-Dist: pydantic>=2.12.5
Requires-Dist: send2trash>=1.8
Requires-Dist: typing-extensions>=4.0
Requires-Dist: uv>=0.12
Requires-Dist: uvicorn[standard]>=0.32
Requires-Dist: zstandard>=0.23
Description-Content-Type: text/markdown

# shidoshi

[![PyPI](https://img.shields.io/pypi/v/shidoshi.svg)](https://pypi.org/project/shidoshi/)
[![CI](https://github.com/tdchaitanya/shidoshi/actions/workflows/test.yml/badge.svg)](https://github.com/tdchaitanya/shidoshi/actions/workflows/test.yml)
[![License](https://img.shields.io/badge/license-Apache--2.0-blue.svg)](LICENSE)

shidoshi is a local notebook for iterative work with LLMs and coding agents.
You write Python, bash, notes and prompts in one notebook, and the notebook
itself, in order, is the conversation: your code, its output and your notes are
the context a prompt sees. Nothing runs or changes on its own. You run each cell,
and you decide what the model gets to see.

Notebooks are ordinary `.ipynb` files, so anything you write in shidoshi opens
in Jupyter, VS Code or GitHub.

## Install and run

On macOS and Linux:

```bash
curl -LsSf https://raw.githubusercontent.com/tdchaitanya/shidoshi/main/install.sh | sh
```

On Windows (PowerShell):

```powershell
powershell -ExecutionPolicy ByPass -c "irm https://raw.githubusercontent.com/tdchaitanya/shidoshi/main/install.ps1 | iex"
```

The script installs [uv](https://docs.astral.sh/uv/) with Astral's installer
if you don't have it, then runs `uv tool install shidoshi` and adds uv's tool
directory to your PATH. It never uses sudo or your system Python. Run it again
to upgrade. Set `SHIDOSHI_VERSION=0.14` to install a specific version, or
`UV_NO_MODIFY_PATH=1` to leave your shell profile alone.
[install.sh](install.sh) is short if you'd like to read it first.

If you already have uv, or would rather install it yourself (`winget install
astral-sh.uv` on Windows), install shidoshi as a uv tool directly. It gets its
own isolated environment and a `shidoshi` command on your PATH, so one install
works from any folder and doesn't touch your projects' environments:

```bash
uv tool install shidoshi    # later: uv tool upgrade shidoshi
uv tool update-shell        # if `shidoshi` isn't found in a new terminal
```

Then:

```bash
shidoshi                    # open the current directory
shidoshi ~/work/project     # open a folder
shidoshi analysis.ipynb     # open a notebook directly
```

shidoshi needs Python 3.13 or newer, and uv downloads it if your machine
doesn't have it. `uvx shidoshi` runs it once without installing. pipx works too
(`pipx install --python python3.13 shidoshi`), and so does `pip install
shidoshi` into an environment you manage yourself.

`shidoshi` starts a local server, prints its URL and opens it in your browser.
The URL carries a one-time access code, so use the full printed link when
opening it in another browser. By default, the server listens on loopback. Use `--ip 0.0.0.0` to accept
connections from other machines; authentication remains required.

To keep an access token across restarts, set `SHIDOSHI_TOKEN` or set
`SHIDOSHI_TOKEN_FILE` to a readable UTF-8 file containing the token (for example,
a Docker secret). Setting both is an error. Tokens must contain 16–1024
printable characters without whitespace, using at most 1024 UTF-8 bytes.
A token file may end in one newline, which is removed. With a configured
token, the server prints and opens a bare URL; enter your token in the login
form. The configured token is never printed. Without either variable, the
server generates a new access token on every start.

Useful flags: `--port` (default 8420, and the next free port if it is taken),
`--no-browser`, and `--ip` (default `127.0.0.1`). For TLS termination, see
[trusted reverse proxies](docs/reverse-proxy.md) and `--trusted-proxy`. Run
`shidoshi --help` for the rest.

## Self-hosting with Docker

One container serves one user. Build from a checkout (Docker builds the frontend
and packages the wheel in the same build):

```sh
docker build -t shidoshi:local .
docker volume create shidoshi-workspace
docker volume create shidoshi-data
docker run -d --name shidoshi -p 8420:8420 \
  -v shidoshi-workspace:/workspace -v shidoshi-data:/data shidoshi:local
docker logs shidoshi
```

Open `http://<server>:8420/#access=<generated-token>` using the token from the
logs. Replace the printed `localhost` with your server's address. The generated
token changes on startup and invalidates existing sessions. To preserve browser
logins across replacement, configure a stable token as below and keep the same
data volume and browser origin. Treat the access link and
logs as credentials. `/healthz` is an unauthenticated readiness endpoint.

For a token you control, use the Compose example with a secret file:

```sh
mkdir -p secrets
python3 -c 'import secrets; print(secrets.token_urlsafe(32))' > secrets/shidoshi_token
chmod 600 secrets/shidoshi_token
docker compose up --build -d
docker compose logs shidoshi
```

The default container user is UID/GID **1000:1000**. The secret must be readable
by UID 1000 inside the container: when your host UID differs, set ownership with
`sudo chown 1000:1000 secrets/shidoshi_token`. Open `http://<server>:8420/` and
enter the file's token. Compose mounts it at `/run/secrets/shidoshi_token` and
sets `SHIDOSHI_TOKEN_FILE`; configured tokens never appear in server logs.
`SHIDOSHI_TOKEN` is also supported; do not set both variables.

Named volumes are initialized with the image's ownership. For bind mounts,
create and provision the folders first for UID/GID 1000:1000. `/workspace` must
be writable; `/data` must allow that user to create the private
`/data/shidoshi` directory. If that directory already exists, it must be owned
by UID 1000 with mode `0700`; startup refuses an unsuitable state root with an
error identifying the UID and path. The image does not chown host files or run
as root. See [history deployment](docs/history-deployment.md) for private history
roots, coordinated backups and single-writer requirements.

Keep both volumes when replacing the container. `/workspace` holds notebooks,
assets and their adjacent history; `/data/shidoshi` holds environments, preferences,
auth sessions, caches and configured saved-key storage. Saving API keys without
an OS keychain additionally needs the encrypted file-store configuration described
in [credential storage](docs/encrypted-credentials.md). `docker compose down` keeps
named volumes; `docker compose down -v` deletes them. Back up both volumes while
the server is stopped. Do not share them between running instances.

The process runs under `tini`, uses the wheel's bundled uv for environment
management, and listens on container port 8420 without port fallback or a browser.
Python and bash cells execute with the container user's permissions. The HTTP
file API confines remote access to the workspace and explicitly allowed roots;
code and agents can reach anything the container user can, including the network.
Mount only the data you intend to expose; never mount the Docker socket.

This slice supplies the local build and plain-HTTP port mapping. Use an encrypted
transport for off-box access: Caddy/TLS deployment and trusted proxy handling are
tracked in [#223](https://github.com/tdchaitanya/shidoshi/issues/223) and
[#224](https://github.com/tdchaitanya/shidoshi/issues/224). Multi-architecture GHCR
publishing is [#221](https://github.com/tdchaitanya/shidoshi/issues/221); a published
image is not required for the commands above. Container agent CLI installation
and browser sign-in are tracked in
[#232](https://github.com/tdchaitanya/shidoshi/issues/232).

To reproduce image acceptance tests, install the project development dependencies
and `npm --prefix web-ui ci`, install Chromium with
`web-ui/node_modules/.bin/playwright install --with-deps chromium`, then run:

```sh
SHIDOSHI_CONTAINER_IMAGE=shidoshi:local uv run pytest tests/container -v
```

These opt-in tests use disposable Docker volumes and real Chromium through a
published port. They cover generated and configured tokens, Python/bash execution,
repeated kernel restarts, zombie detection, valid notebook saves, container
replacement with a persistent session, and an inaccessible state volume.

## The notebook

A notebook has four kinds of cell:

- Python cells run in a Jupyter kernel, in the Python environment you pick
  from the notebook's environment panel. Nothing starts until you run a cell.
- Bash cells run in a persistent shell in the notebook's folder.
- Notes are Markdown, with math, tables, checklists and images.
- Prompt cells are where you talk to a model.

Shift+Enter runs the current cell and selects the next; at the end it creates
a blank Python cell ready for typing. Cmd/Ctrl+Enter runs in place. Notes render
Markdown and prompts run Ask; neither shortcut runs adjacent cells.
Python `input()` and `getpass.getpass()` accept replies inline, with password
fields masked. Stop interrupts a pending input request.

A prompt cell can run in two ways. Ask sends one request: the cells above it
become the conversation history, and the answer lands in the cell's output. The
model can search the web, and you will see what it searched for. Agent hands the
prompt to an agent that can act in the notebook. That is Codex or Claude Code
running locally, if you have their CLIs installed, or the built-in agent.
Codex and Claude Code show an approval card in the notebook for anything your
permission settings don't already allow. The built-in agent can read your
variables, run Python and bash, and propose new cells, and it runs code without
asking first.

You control the context per cell. Pin a cell to keep it in every prompt's
context. Exclude it to keep it out. Notes can be marked as asides, which prompts
skip unless pinned.

If Ask or the built-in agent exceeds the model's context limit, it can retry
with older unpinned cells omitted. A notice tells you what was omitted; the
notebook stays intact. The built-in agent also summarizes older conversation
memory. See [context limits](docs/context-limits.md) for pin protection,
recovery limits, and how Claude Code and Codex differ.

Drop PNG, JPEG, GIF or WebP files between cells to add them as image notes. The
files are saved next to the notebook and linked with ordinary relative Markdown,
so keep them with the `.ipynb` when you move it. Ask and Agent send images from
eligible cells above the prompt to models that accept images.

## Python environments

Your notebook code should run in an environment of its own, not in the one
shidoshi is installed in. With `uv tool install`, shidoshi's environment
belongs to uv: packages you add to it can be removed on the next upgrade, and
they can break shidoshi's own dependencies. The environment panel labels that
kernel "Shidoshi's Python" and warns when a notebook uses it. In that kernel
`%pip`, `%uv`, `%conda`, `%mamba` and `%micromamba` install nothing and point
you to an environment of your own. Shell commands such as `!pip` can't be
intercepted, so don't install with them there either.

The panel can create a separate environment for you, using uv, with no terminal
needed. Each one is a small uv project under `~/.shidoshi/envs`. `%pip install`
in a notebook installs into that environment, and the Packages window adds,
removes and upgrades packages with a preview first. You can also link a uv
project you already have, or pick any Jupyter kernel registered on your
machine.

Mark one environment "Use for new notebooks" and new notebooks start with it
suggested. While a notebook is on Shidoshi's Python, the panel offers to
create a default environment in one click. A notebook that already names a kernel keeps it, and
nothing switches without you choosing it.

## Models and keys

Ask uses OpenAI (through the Responses API), OpenRouter, or Anthropic (through
the Messages API). The built-in agent supports OpenAI and OpenRouter; for Claude
models there, use the Claude Agent harness. Choose the provider and model from
the LLM provider panel in any notebook; each notebook remembers its own choice.

Keys come from `OPENAI_API_KEY` / `OPENROUTER_API_KEY` / `ANTHROPIC_API_KEY` if they are set.
Otherwise enter one in the panel and keep it for the session, or save it to your
OS keychain. Keys are never written to notebooks or browser storage.
`OPENAI_BASE_URL` points the OpenAI provider at a proxy or compatible endpoint,
which must implement `/responses`.

Codex and Claude Code use their own CLI login and settings.

[web-ui/README.md](web-ui/README.md) covers kernels, credentials, the Debug
toggle, reconnecting and branching in detail.

## Coming from the Jupyter magics

Earlier versions of shidoshi were a set of Jupyter magics (`%ask`, `%%agent`,
`%%pin`, `%shidoshi config`) with a TOML config file. Those are gone. The web
notebook replaces them, and it can open your existing notebooks.

- `%load_ext shidoshi` no longer loads anything. Old `%ask` cells open as plain
  code cells. Their saved answers stay in place, but they are no longer prompts.
- If you ran `shidoshi install-kernel`, remove that kernel with
  `jupyter kernelspec remove shidoshi`.
- `config.toml` files are ignored. Choose the provider and model in the web UI.
- `shidoshi web` is now just `shidoshi`.

## Development

To launch from a source checkout with a fresh UI, use:

```bash
./scripts/shidoshi.sh
./scripts/shidoshi.sh /path/to/notebook.ipynb --port 8422
```

The launcher requires Node/npm and uv, installs frontend dependencies from the
lockfile, rebuilds the UI, and then runs `uv run shidoshi` from this checkout.
It forwards all Shidoshi arguments and preserves the directory you launched
from. A failed install or build stops the launch. It builds the code currently
checked out; run `git pull --ff-only` yourself when you want to update it.

For separate setup and testing:

```bash
uv sync
npm --prefix web-ui ci && npm --prefix web-ui run build   # the UI the server serves
uv run pytest tests/unit tests/btp tests/test_import_rule.py
```

In a source checkout, Shidoshi's Python refuses `%pip` too, as it does for
users. To install into the checkout's `.venv` from a notebook anyway, start the
server with `SHIDOSHI_ALLOW_INTERNAL_PIP=1 uv run shidoshi`. Only the value `1`
turns the guard off, and it applies to kernels started after that.

Integration tests in `tests/integration/` call real models. They need
`OPENAI_API_KEY` / `OPENROUTER_API_KEY` (or `OPENAI_BASE_URL` for a proxy):

```bash
uv run pytest tests/integration/ -v -m integration
```

To audit prompts, capture the exact requests sent to providers. Capture is off
unless you pass the flag:

- `shidoshi --capture-requests DIR` routes Ask, the built-in agent, Codex and
  Claude Code through a local logging proxy. It records every provider request
  as sent, including the harness-owned prompts and tools Debug can't show.
- Files go to `DIR/<session>/<notebook>/<run>/`, one JSON file per request,
  with a `run.json` per run and a `DIR/index.jsonl` manifest. Responses are
  recorded as metadata only (status, usage, timing). Add `--capture-responses`
  to keep full response bodies too.
- Auth headers are stripped, bodies are redacted, and files are owner-only
  (0700/0600). A DIR inside the workspace is refused unless you pass
  `--capture-in-workspace`, and nothing is written into `.ipynb`.

See [docs/request-capture.md](docs/request-capture.md) for what each path
captures and the known gaps.

See [web-ui/README.md](web-ui/README.md) for running the frontend against a
live-reloading dev server and for the browser test suites.

For PyPI releases, `scripts/release.sh` and the publish workflow rebuild the
frontend and run `scripts/verify_frontend.py`. Both the wheel and source archive
must contain exactly the fresh frontend before publishing can proceed.

The test workflow uses Linux x64 runners with the `shidoshi-ci` label
on `luge-per-middlegame.exe.xyz`: `gondola-tarte-1` and `gondola-tarte-2`.
Each runs as a separate non-sudo user with its own workspace and caches under
`/opt/shidoshi-runners/runner-1` or `runner-2`. Both systemd services start on
boot; inspect them on the VM with `sudo systemctl status 'actions.runner.*'`.
Chromium system dependencies are installed on the VM; rerun Playwright's
`install-deps chromium` as an administrator when an upgrade requires new ones.
Adding runners to this same VM also requires assigning distinct browser test
ports in `.github/workflows/test.yml`.

Two additional runners, `neon-1` and `neon-2`, run in separate unprivileged
Docker containers on the Ubuntu homelab. Each has a two-CPU / 4 GiB RAM limit,
separate workspace/cache volumes and its own network namespace, so browser
ports can overlap safely. Jobs cannot access homelab files or its Docker
socket. Their setup, tests and operations live in
[homelab/services/github-runners](https://github.com/tdchaitanya/homelab/tree/main/services/github-runners).
Run the manual `runner-smoke` workflow to validate both exact runner labels
through dependency installation, builds, package verification and browser
tests. These older runners increase capacity but are slower per job; the
test workflow allows 90 minutes for either job.

CI stores pytest data in `$RUNNER_TEMP/pytest`, which the runner clears between
jobs, and prunes uv's dependency cache after each suite, including failed runs.
Before installing dependencies, CI checks for at least 4 GiB and 500,000 free
inodes on the temporary filesystem, leaving headroom on each runner host. The old
retained pytest runs consumed 6.8 GiB and over 1.1 million inodes across the VM.
If the check fails, inspect `df -h`, `df -i`, and retained test data before retrying.

## License

Apache License 2.0. See [LICENSE](LICENSE).

Server child processes (Python kernels, bash cells, agent CLIs, and environment
management tools) do not inherit `SHIDOSHI_TOKEN`, `SHIDOSHI_TOKEN_FILE`,
`SHIDOSHI_SECRET_KEY`, or `SHIDOSHI_SECRET_KEY_FILE`. Other environment settings,
including provider credentials, continue to work. This is defense in depth:
notebook code and agent tools run as the application user and can still read
secret files accessible to that user. Make `_FILE` secrets readable only during
startup where your deployment supports that, or accept this limit.

Claude's SDK currently merges the parent environment into its CLI launch even
when an explicit environment is supplied. Shidoshi therefore sanitizes both its
CLI and version-probe launches through a private transport adapter. The SDK is
pinned to the reviewed version; an SDK upgrade requires reviewing those launch
methods and updating the adapter's source fingerprints and spawn tests. An
unrecognized launch implementation fails before starting Claude.
