Metadata-Version: 2.5
Name: criceto
Version: 0.1.0
Summary: Incremental document downloader for Scalable Capital — mailbox documents and transaction statements, deduplicated by SHA-256.
Project-URL: Homepage, https://github.com/moritzblume/criceto
Author: Moritz Blume
License-Expression: AGPL-3.0-or-later
License-File: LICENSE
Keywords: documents,downloader,pdf,playwright,scalable-capital
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: End Users/Desktop
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Office/Business :: Financial
Requires-Python: >=3.11
Requires-Dist: platformdirs>=4.0
Requires-Dist: playwright>=1.49
Description-Content-Type: text/markdown

# criceto

[![License: AGPL v3](https://img.shields.io/badge/License-AGPL%20v3-blue.svg)](LICENSE)
![Python](https://img.shields.io/badge/python-3.11%2B-blue)
![Playwright](https://img.shields.io/badge/playwright-chromium-green)

**Incremental document downloader for [Scalable Capital](https://de.scalable.capital/).**

`criceto` downloads your mailbox documents and transaction statements
(Wertpapierabrechnungen) from the Scalable Capital web app. Every run
walks the document lists from the newest date down to an **archive
boundary** stored in `criceto.json`, fetches everything newer than it,
and deduplicates by **SHA-256** — so a document is ever downloaded
twice, but never stored twice, and renaming files never causes gaps.

> Inspired by [SC-Downloader](https://github.com/MichaelN0815/SC-Downloader)
> by MichaelN0815 — thanks for showing that UI-driven automation of
> Scalable Capital works! *criceto* takes a different approach to
> incremental downloads and is fully cross-platform.

## Why *criceto*?

*criceto* is Italian for **hamster** — and the hamster is the archetypal
collector among hoarders. Unlike the squirrel, which scatters thousands
of caches (and forgets half of them), a hamster is a *larder hoarder*:
one central, carefully sorted hoard with a mental map of every chamber.

That is exactly how this tool works. Everything lands in **one archive
directory**, sorted by source, indexed in a single state file. And unlike
any real rodent, it never forgets where anything is — the SHA-256 index
guarantees that no document is ever stored twice and no day is ever left
with a gap.

---

## How it works

1. A Chromium window opens. On the first run you log in manually
   (including two-factor authentication). The session is persisted
   locally, so later runs usually start right away.
2. Each source (mailbox, transactions) is walked **newest first**. Every
   document dated **after** the archive boundary is fetched into a
   temporary folder and hashed.
3. Only documents whose SHA-256 is genuinely new are moved into the
   target directory. The first document dated **on or before** the
   boundary ends the walk — everything older is guaranteed to be
   archived already.
4. The archive state lives in one file:

```
criceto-documents/
├── scalable-mailbox/
│   ├── 2026-08-31_Kontoauszug....pdf
│   └── (…)
├── scalable-transactions/
│   └── 2026-08-25-Wertpapierabrechnung-IE0003XJA0J9.pdf
└── criceto.json        <- the archive index
```

```json
{
  "version": 2,
  "sources": {
    "scalable-mailbox": {
      "archived_through": "2026-08-30",
      "shas": { "<sha256>": "<filename>", "…": "…" }
    }
  }
}
```

### The archive boundary (`archived_through`)

The index records one date per source with a simple guarantee: **every
document dated on or before this day is archived.** Documents in the
past never change, so nothing below the boundary is ever looked at
again — the scan starts at the newest date and stops at the boundary.

Two rules keep the guarantee honest:

- The boundary is **never today** — today can always gain more
  documents, so it is re-checked on every run.
- The boundary only moves after a run processed the whole region above
  it **without errors**. On any failure it stays where it is; the next
  run re-checks that region (documents already archived are detected by
  hash and skipped).

Because every run always walks the list down to the boundary, local
gaps (e.g. after a crash) are impossible by construction: the boundary
only ever claims dates the walk actually verified.

### The tmp folder

Everything newer than the boundary is downloaded into
`<target>/tmp/<source>/` first, hashed, and only moved into the target
directory when the content is new. That makes interruptions
self-healing: a partially processed day is simply re-fetched on the
next run, and already archived files are detected by their identical
content — no duplicates, no gaps.

### Why hashes instead of filenames?

Filename-based dedup breaks whenever the naming scheme changes (a real
problem other downloaders have run into). Hashing the actual PDF bytes
means:

- Renaming files is safe.
- Changed filename schemes in the app don't matter.
- Content changes are detected even when a document keeps its name.

## Installation

Requires [Python 3.11+](https://www.python.org) and
[uv](https://docs.astral.sh/uv/) (works on Linux, macOS and Windows).

```bash
# from a clone of this repository
uv sync
uv run playwright install chromium

# first run: a browser window opens, log in (including 2FA)
uv run criceto --target ~/ScalableDocs
```

Afterwards, just run `uv run criceto` (or `uvx criceto` once
published) whenever you want to fetch new documents. Only documents
newer than the archive boundary are fetched, and only genuinely new
files are written to disk.

## Command-line options

| Option | Description |
| --- | --- |
| `--target DIR` | Download root (default: `./criceto-documents`) |
| `--source {all,transactions,mailbox}` | Restrict to sources whose key ends with this kind (default: all) |
| `--all` | Forget the archive state and re-download everything |
| `--headless` | Run without a visible browser (only after a session exists) |
| `--profile DIR` | Custom browser session directory |
| `--diagnose` | Check all UI selectors and take screenshots |
| `--version` | Print version |

## Architecture

```
providers/   GUI layer — one adapter per company portal.
             ScalableMailboxSource, ScalableTransactionsSource:
             walk the website, fetch documents, deliver bytes.
controller   GUI-neutral: hash, tmp folder, move-if-new,
             archived_through bookkeeping. Unit-testable.
manifest.py  SeenIndex — the persisted archive state (criceto.json).
```

A new company portal is a new provider package implementing two small
interfaces — controller, index and CLI stay untouched.

And Scalable Capital is just the beginning: the provider layer is
deliberately generic, and **more providers are planned**. If your bank,
insurer or telco has a web portal you'd love to archive from, that's
all it takes.

## Resilience to website changes: fail fast, never guess

Scalable Capital is a single-page app without a public API, so
*criceto* drives the real web UI. When their website changes, a
downloader can fail in two ways: loudly (annoying) or silently wrong
(dangerous). A silently wrong download would be the worst outcome here —
it could stop the incremental logic at the wrong point and leave
permanent gaps in your document collection.

criceto therefore uses **one canonical selector per UI element** and
fails immediately when the website no longer matches:

- `data-testid` / `data-*` attributes are used where Scalable provides
  them (mailbox rows, download buttons) — they are set deliberately by
  the frontend team and are the most stable anchor available.
- Transaction rows have no test id, so they are matched by accessible
  role (`div[role='button']` test ids like `SETTLED-transaction-*`) plus
  visible content (a German date in a per-day group header), checked in
  code.

If a selector stops resolving, criceto aborts with a clear
`WebsiteChangedError` (exit code 2) and an error screenshot — never with
a silently wrong download. Extra consistency checks:

- No transaction rows found → fail.
- List not sorted newest-first → abort (the archive boundary logic
  would be unsafe).

All selectors live in a single file,
[`src/criceto/selectors.py`](src/criceto/selectors.py). If Scalable
Capital changes something, that is the only file to update — and
`criceto --diagnose` (run while logged in) shows exactly which selector
is missing (✓ / · MISSING).

The downloader also avoids fragile UI areas on purpose: it does not
touch filter dropdowns, matches on visible content (dates, labels)
rather than styling, and waits on real elements instead of fixed sleeps.

## Security notes

- Your credentials are entered only in the real Scalable Capital login
  page, in a browser session stored locally on your machine.
- The session is stored unencrypted in the profile directory — treat it
  like a logged-in browser.
- The archive index (`criceto.json`) contains no personal data, only
  dates and hashes.

## Disclaimer

This is an unofficial community tool, not affiliated with or endorsed by
Scalable Capital. It automates your own logged-in session and downloads
only your own documents. Use at your own risk; a UI change on Scalable
Capital's side can break it at any time.

Please also be aware that automated access may be in conflict with
Scalable Capital's terms of service. Whether and how that applies to your
usage is between you and them — criceto can't answer that question for
you. If this is a concern, ask before using the tool or skip it entirely.

## Contributing

Issues and pull requests are welcome! If downloads suddenly fail after a
Scalable Capital update, please run `criceto --diagnose` after logging
in and attach the output plus the `diagnose-*.png` screenshots to your
issue.

**A note on contributions:** to keep the project flexible for the long
term — for example, to offer optional commercial add-ons alongside the
free community edition — a short
[CLA](CLA.md) will be required for contributions. The tool itself stays
free and open source under AGPL-3.0-or-later; the CLA simply allows
alternative licensing of contributed code in future offerings. Details
will be published here before contributions are accepted.

## License

[AGPL-3.0-or-later](LICENSE) — see [LICENSE](LICENSE).
