Metadata-Version: 2.4
Name: forest-cli
Version: 1.2.0
Summary: git-like data management for arbitrary data trees: workspaces, checkouts, remotes, and sync
Author-email: Troy Sincomb <troysincomb@gmail.com>
License: MIT
Project-URL: Homepage, https://github.com/tmsincomb/forest
Project-URL: Repository, https://github.com/tmsincomb/forest
Project-URL: Issues, https://github.com/tmsincomb/forest/issues
Project-URL: Changelog, https://github.com/tmsincomb/forest/blob/main/CHANGELOG.md
Keywords: data-management,sync,cli
Classifier: Development Status :: 3 - Alpha
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: click>=8.0
Requires-Dist: pydantic>=2.0
Requires-Dist: pyyaml>=6.0
Requires-Dist: rich>=14.2
Requires-Dist: rich-click>=1.9.8
Provides-Extra: dev
Requires-Dist: deptry>=0.23; extra == "dev"
Requires-Dist: mypy>=1.14; extra == "dev"
Requires-Dist: pre-commit>=4.0; extra == "dev"
Requires-Dist: pytest>=7.0; extra == "dev"
Requires-Dist: pytest-cov>=6.0; extra == "dev"
Requires-Dist: pytest-randomly>=3.15; extra == "dev"
Requires-Dist: pytest-xdist>=3.6; extra == "dev"
Requires-Dist: ruff>=0.4; extra == "dev"
Requires-Dist: types-PyYAML; extra == "dev"
Dynamic: license-file

# 🌲 forest <a href="https://tmsincomb.github.io/forest/"><img align="right" width="42%" src="https://tmsincomb.github.io/forest/assets/hero.svg" alt="Animated isometric forest: data trees on a workspace platform, packets syncing up a git branch to a remote cloud"></a>

**git-like data management for arbitrary data trees.**

<!-- GitHub Actions workflow badges 404 outside the repo while it is private;
     re-add CI/Docs/Release badges if the repo goes public. -->
[![Site](https://img.shields.io/badge/site-tmsincomb.github.io%2Fforest-2d6a4f?logo=materialformkdocs&logoColor=white)](https://tmsincomb.github.io/forest/)
[![PyPI](https://img.shields.io/pypi/v/forest-cli?logo=pypi&logoColor=white)](https://pypi.org/project/forest-cli/)
[![Python 3.10+](https://img.shields.io/badge/python-3.10%2B-3776AB?logo=python&logoColor=white)](https://www.python.org/downloads/)
[![License: MIT](https://img.shields.io/badge/license-MIT-2ea44f)](https://github.com/tmsincomb/forest/blob/main/LICENSE)
[![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)
[![mypy: strict](https://img.shields.io/badge/mypy-strict-blue)](https://mypy-lang.org/)

Forest is the data-side parallel to git's version control. Git tracks code in
`.git/`; forest tracks large data — local layout plus remote sync — in
`.forest/`. It borrows git's mental model and verbs (`checkout`, `status`,
`push`, `pull`, `remote`, a HEAD-style pointer) so your git intuition carries
over, but the two domains never overlap and neither requires the other.

Forest is **domain-agnostic and self-contained**: it manages *any* data trees,
knows nothing about what the data means, and depends on no other project. It
moves files and tracks their sync state; it does not validate or interpret their
contents.

## Documentation

Full docs live at **[tmsincomb.github.io/forest](https://tmsincomb.github.io/forest/)**.
Start with the **[hands-on tutorial](https://tmsincomb.github.io/forest/tutorial/)**:
one runnable path from `forest init` to push, status and pull, using a local
directory as the remote, so no cloud credentials are needed. Every command and
option is listed in the [CLI reference](https://tmsincomb.github.io/forest/reference/cli/).

## Model

Fixed-depth, no arbitrary nesting:

```
workspace → checkout → stage → unit → files
```

- **Workspace** — a per-repo `.forest/` control area (registry + active pointer).
- **Checkout** — a named data view/focus registered in the workspace (like a git
  branch you stay rooted in). Switching is an O(1) pointer rewrite; data never
  moves.
- **Stage** — a named data category inside a checkout, with a remote layout.
  Command selectors call it `BRANCH`.
- **Unit** — one addressable item within a stage (a subdirectory, a directory,
  or a file, per the stage's `sync_by`). In the default subdirectory mode, loose
  files at the stage root are not units; push, pull and status skip them.

## Install

```bash
pip install forest-cli
```

Requires [`rclone`](https://rclone.org/) on `$PATH` for transfers. For a
development install, clone the repo and `pip install -e ".[dev]"`.

Install the current stable rclone release (CI pins **1.75.1**, verified on
2026-09-25). Native copy reports first appeared in 1.71; versions lacking those
flags fail with an upgrade message, never an offline synced verdict.

## Quick start

No config files are hand-written. Onboarding is a few commands:

```bash
forest init                      # create the nameless .forest/ workspace container
forest remote add origin s3://my-bucket/prefix
forest remote use origin          # select the source for new checkouts
forest checkout demo             # create + activate, remembering origin
forest add raw ./data/raw        # register stage 'raw' and bind it to a local path
forest push                      # sync every bound stage to the checkout's saved remote
```

- `forest init` creates **only** `.forest/config.yaml` (`version: 1`,
  `checkouts: {}`) and a managed `.gitignore`. No root config, no checkout, no
  active pointer.
- `forest checkout <name>` switches to the checkout, creating, registering, and
  activating it first (with `.forest/checkouts/<name>/forest.yaml`) when the
  name is not registered. `forest checkout create <name>` is the explicit form.
- Remotes belong to the workspace and can be added before the first checkout.
  `remote use` changes only the choice for new checkouts. Existing trees keep
  their saved remote, including when adding stages. With one
  workspace remote, a new checkout remembers it automatically in shared metadata.
- `forest add STAGE PATH` registers a new stage and binds it to a local path in
  one step (use `forest bind` to rebind an existing stage).

## Metadata layout

Everything forest owns lives under `.forest/`; your data does not.

```
.forest/
  config.yaml                     # shared: version, checkouts{}, remotes{}
  local.yaml                      # user-local: remote choice for new checkouts (gitignored)
  HEAD                            # active checkout name (gitignored)
  checkouts/
    demo/
      forest.yaml                 # shared: stages, manifest; legacy remotes if present
      local.yaml                  # user-local: active_remote, stage_paths (gitignored)
      sync_state.json             # user-local push/pull state (gitignored)
```

Shared metadata (`config.yaml`, each `forest.yaml`) is committed so a fresh
clone of a bound tree bootstraps with `bind` + `pull`. User-local files
(`HEAD`, both `local.yaml` files, `sync_state.json`, lock files) are gitignored.

## Commands

| Command | Purpose |
|---|---|
| `forest init` | Create the workspace container, or report setup status if it exists. |
| `forest checkout create/list/current/remove <name>` | Manage checkouts; bare `forest checkout <name>` switches, creating first if needed. `remove --yes` skips the prompt for scripts. |
| `forest add STAGE PATH [REMOTE:PATH] [--direction push\|pull\|both]` | Register a new stage and bind it to a local path; units are subdirectories unless the stage's `sync_by` in `forest.yaml` is `directory` or `file`. `REMOTE:PATH` saves `PATH` as the stage's `remote_path` on the tree's own remote (default `<checkout>/<stage>`). `--direction` (default `both`) limits which transfers the stage allows. |
| `forest bind [STAGE PATH]` / `forest unbind STAGE` | Manage local stage↔path bindings. |
| `forest remote add/remove/list/use/show` | Manage workspace remotes; `use` selects the remote for new checkouts. |
| `forest push / pull / status / diff [BRANCH \| BRANCH/UNIT ...]` | Sync and inspect against the current tree's saved remote. Bare commands cover every bound stage (unbound stages warn and skip); a named `BRANCH` must be bound. `push`, `pull` and `status` also take `--id ID` to select units through the manifest. |
| `forest flow` | Emit a Mermaid data-flow diagram of the active checkout. |
| `forest prompt init bash\|zsh` | Print an eval-able snippet that shows the active checkout in your shell prompt. |

Run any command with `-C <path>` to operate on another repo without `cd`.

Forest syncs **all** files in a data unit, skipping OS junk (`.DS_Store`,
AppleDouble `._*`, `*.tmp`) and partial `*.forest-tmp` files. It applies no
content-based include/exclude rules.
Push and pull use native rclone comparison against real destinations. Changed
and missing files transfer; unchanged files are skipped. Counts show files and
bytes actually transferred, not the size of the selected unit. Layout-preserving
transfers batch selected files in one native copy per root mapping, even when
root names differ; true filename renames use copyto. Pull uses
rclone's temporary-file rename, with `.forest-tmp` names and no `--inplace`.

Select work with positional names: `forest push outputs` for every unit in
that branch, `forest pull outputs/run1 outputs/run2` for units of one branch.
`--stage`, `--all` and `add --sync-by` are removed without aliases; set
`sync_by` in the checkout's `forest.yaml` instead. Unknown branches, invalid
paths and missing push targets fail before any selected target transfers.

Forest reuses saved local hashes when the resolved path, device/inode, size,
mtime and ctime are unchanged. Changed evidence causes a fresh read, including
edits that restore the old mtime. This relies on filesystem stat evidence;
`status --checksum` performs an explicit fresh content check. Legacy state
warms the cache on its next successful operation. Remote hashes are read live,
not cached. Verification still precedes recording, and missing backend hashes
limit verification to sizes. `--force` uses native checksum comparison when a
transfer is needed, including edits whose size and mtime were preserved.

## Live status

`forest status` compares bound stages with the current tree's saved remote using
[rclone copy's dry-run report](https://rclone.org/commands/rclone_copy/#logger-flags).
It downloads and uploads nothing, changes no file metadata, and never saves sync
history. It uses the real paths, not temporary transfer destinations.

Each row names a file: `synced`, `different`, `local-only`, or `remote-only`.
Saved history can further identify `local-changed`, `local-missing`, and
`remote-missing`. With `--checksum`, an unchanged local hash can also establish
`remote-changed`. `different` does not guess which side changed.

Normal comparison uses rclone's size/time checks and may hash files with changed
times. It does not hash the whole tree routinely. `--checksum` requests available
backend hashes and checks local contents against saved hashes; it can be slow.
`Synced` means a match under that comparison, not guaranteed byte equality on
backends without suitable hashes. A failed comparison exits with an error and
prints no synced rows.

Use `status BRANCH/UNIT` or `status --id ID` to select units.
`--incomplete` hides synced rows, for this tree and saved remote only.
New remote files appear within configured stage prefixes, even when the bound
local directory is missing. A prefix `sequencing/run0001` does not discover its
sibling `run0002`. Bare status skips unbound stages with a warning; bind them
before comparing. Read-only status inspects both push-only and pull-only stages.

The old `ls` and `list` commands are removed. Status always includes file paths,
so their `--tree` and `--long` views are removed too. Use `checkout list` for
checkout names; status does not scan other trees. Transfer behavior is unchanged.

## Terminal output

On an interactive terminal forest renders rich output — colored status
tables, panels, and live progress bars during `push`/`pull`
(rclone JSON stats streamed to a bar; needs rclone ≥ 1.49, older versions
transfer fine without a bar). When stdout is piped or redirected, output
falls back to the plain, machine-parseable format, so scripts never see
ANSI codes or box drawing.

Overrides: `--color`/`--no-color` per invocation, `FOREST_OUTPUT=rich|plain|auto`
in the environment, and the [`NO_COLOR`](https://no-color.org)/`FORCE_COLOR`
conventions. Precedence: flag > `FOREST_OUTPUT` > `NO_COLOR` > `FORCE_COLOR` >
TTY detection. `--verbose` shows rclone's raw transfer log instead of a
progress bar; `--quiet` suppresses everything but errors.

## Shell prompt

Show the active checkout in your prompt (bash and zsh), the way git prompts
show the current branch:

```bash
# ~/.bashrc
eval "$(forest prompt init bash)"

# ~/.zshrc
eval "$(forest prompt init zsh)"
```

```text
~/lab/G004 $                # outside a workspace: unchanged
🌲 G004 ~/lab/G004 $        # inside a workspace with an active checkout
```
The snippet is pure shell — it reads `.forest/HEAD` directly and never invokes
the forest CLI while rendering, so your prompt stays fast. Set
`FOREST_PROMPT_ICON` to replace the 🌲.

## Config reference

Workspace `.forest/config.yaml` (shared, committed):

```yaml
version: 1
checkouts:
  demo:
    remote: origin            # saved source for this tree, not a workspace default
remotes:
  origin:
    url: s3://my-bucket/prefix
    region: us-east-2          # optional; also endpoint, profile, key_file, known_hosts
```

Checkout `forest.yaml` (shared, committed):

```yaml
project: demo
stages:
  raw:
    remote_path: demo/raw      # optional; defaults to <checkout>/<stage>
    sync_by: subdirectory      # subdirectory | directory | file
    direction: both            # push | pull | both; bare push/pull skip
                               # wrong-direction stages, explicit selection errors
```

Checkout `local.yaml` (per-machine, gitignored):

```yaml
active_remote: origin
stage_paths:
  raw: ../data/raw             # relative resolves from the workspace root
```

`remote use NAME` writes the workspace's local choice for future checkouts.
It never changes an existing tree's source. A new tree saves its remote name
under `checkouts` in workspace config, so cloning the shared metadata preserves
that source. Stage mappings belong to the tree and use its saved remote.

Legacy trees keep a recorded checkout-local `active_remote`, or their sole
checkout-local remote when no explicit choice was recorded. Forest never guesses
from the latest workspace choice. Legacy definitions stay in their original
files and keep their addresses. `remote list/show` still display the selected
tree's effective sources; without a selected tree they show workspace sources.
`remote use` selects only workspace definitions for future trees.

A tree with no saved source cannot add stages or transfer data. Restore its
verified source metadata, or select a workspace remote and create a new tree.
Missing definitions and conflicting saved references also fail clearly. There
is no rebinding command in this change.

`remote remove` refuses workspace sources referenced by any tree, even when its
local config is missing. Overlapping legacy names must be resolved explicitly.
Removing an unused definition never removes data or sync history.

## Environment variables

All optional, all off by default — forest is silent and sends nothing anywhere
unless configured. Copy `.env.example` for a commented template. Logs are
always secret-scrubbed.

| Variable | Default | Effect |
|---|---|---|
| `FOREST_LOG_FILE` | unset | Append structured logs (JSON lines) to this file. |
| `FOREST_LOG_FORMAT` | `json` | `json` or `text`; set without `FOREST_LOG_FILE` to log to stderr. |
| `FOREST_LOG_LEVEL` | `INFO` | Standard logging level name. |
| `FOREST_TRANSFER_RETRIES` | `2` | Extra attempts for transient rclone failures; `0` disables. |
| `FOREST_RETRY_BASE_DELAY` | `0.5` | Initial retry backoff in seconds; doubles per attempt. |
| `FOREST_OUTPUT` | `auto` | Terminal rendering: `rich`, `plain`, or `auto` (rich on a TTY, plain when piped). |
| `NO_COLOR` / `FORCE_COLOR` | unset | Standard color conventions; `FOREST_OUTPUT` and `--color/--no-color` take precedence. |

## Notes

- **Single active machine (v1).** `HEAD`/`local.yaml`/`sync_state.json` are
  git-invisible but may be synced by a file-syncing tool; forest assumes one
  active machine and uses atomic writes plus a per-checkout `flock` for
  intra-machine write races.
- **Real filenames.** Forest stores data under real paths, not a
  content-addressed blob store.
