Metadata-Version: 2.4
Name: aleth-ingest
Version: 0.2.0
Summary: Open-source file ingestion companion: walk a folder of text files and store them verbatim in Aleth memory over the public wire, skipping unchanged files and superseding changed ones. Depends only on aleth-client.
Author: Aleth Authors
License: Apache-2.0
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: aleth-client<2,>=1.5
Provides-Extra: pdf
Requires-Dist: PyMuPDF<2,>=1.24; extra == "pdf"
Provides-Extra: office
Requires-Dist: python-docx<2,>=1.1; extra == "office"
Requires-Dist: openpyxl<4,>=3.1; extra == "office"
Requires-Dist: odfpy<2,>=1.4; extra == "office"
Provides-Extra: ebook
Requires-Dist: EbookLib<1,>=0.18; extra == "ebook"
Provides-Extra: email
Requires-Dist: extract-msg<1,>=0.48; extra == "email"
Provides-Extra: documents
Requires-Dist: PyMuPDF<2,>=1.24; extra == "documents"
Requires-Dist: python-docx<2,>=1.1; extra == "documents"
Requires-Dist: openpyxl<4,>=3.1; extra == "documents"
Requires-Dist: odfpy<2,>=1.4; extra == "documents"
Requires-Dist: EbookLib<1,>=0.18; extra == "documents"
Requires-Dist: extract-msg<1,>=0.48; extra == "documents"
Provides-Extra: docling
Requires-Dist: docling<3,>=2.120; extra == "docling"
Dynamic: license-file

# aleth-ingest

The file ingestion companion for Aleth memory. Point it at a folder of text
files and it stores them verbatim in your memory, then keeps them in sync:
unchanged files are skipped, changed files supersede their old version, removed
files are retracted. Your memory does not bloat with duplicates.

It only ever calls the public memory verbs (`remember`, `supersede`,
`retract`, `forget`). It has no special access to the engine, so what it can
do, you can do.

## What it does

- **Walks a directory** of text files (`.md`, `.txt`, `.rst` by default; add
  your own globs).
- **Reads documents, not just text.** PDF, Word (`.docx`), OpenDocument
  (`.odt`), ePub, Excel (`.xlsx`), Outlook messages (`.msg`) and HTML, with
  `--all-formats` or an explicit `--include '*.pdf'`. Each format's library is
  an optional install; run `aleth-ingest formats` to see what your machine
  can read now and the one line that fixes anything it cannot.
- **Stores content verbatim.** Small files become one memory; larger files are
  split on paragraph and heading boundaries, never rewritten. Because the
  content is stored exactly, you can delete the original file and rely on the
  copy in memory.
- **Skips unchanged files** using a size + modified-time fast path, so a re-run
  over a big folder is cheap and does not re-read files that did not move.
- **Supersedes changed files, by name.** The new version is stored first,
  then each old memory is retired *behind the memory that replaced it*. Ask
  your memory what a file used to say and it can answer, because the link
  from the old text to the new one is recorded rather than implied. Nothing
  is destroyed unless you ask for it.
- **Leaves unchanged parts alone.** Edit one section of a long document and
  only that section's memories are replaced. The rest keep the identity they
  already had, so a small edit does not churn the whole file.
- **Retracts what nothing replaced.** A deleted file, or a section dropped
  when a file shrank, has no successor to name, so it is retracted instead:
  marked no longer current, with its history kept.
- **Is safe with overlapping folders.** Track a project and a folder inside
  it, the same folder twice, or two folders that each hold a copy of a file:
  a memory is retracted or erased only once nothing you track still holds that
  text. Your browser history and page text count too, so one article you read
  on two sites, or a page you also keep as a file, is not erased by forgetting
  only one of them. `plan` marks such a file HELD. If a tracked folder cannot
  be read (an unplugged drive, say), the run leaves alone anything that folder
  might hold, says so (`plan` marks it WITHHELD), and tries again next time.
- **Tracks everything in a local manifest** (`<root>/.aleth-ingest/manifest.json`),
  so runs are idempotent and an interrupted run resumes correctly.
- **Imports your browser history, locally.** Chrome, Edge, Brave and Firefox
  keep it in a database on your own disk; this reads a copy of that database,
  shows you the domains it would store, and stores nothing until you say so.
  No extension, no upload, no server asked about any site you visited. See
  **Browser history** below.

## Install

```
pip install aleth-ingest
```

Requires a running memory engine reachable on the same machine (the companion
finds it through its discovery file, the same way the memory client does).

## Use

```
aleth-ingest plan    ./notes     # dry run: show what would change (no engine needed)
aleth-ingest run     ./notes     # ingest: apply the plan
aleth-ingest status  ./notes     # manifest summary against what is on disk
aleth-ingest history ./notes     # what replaced what, per file
aleth-ingest untrack ./old       # stop counting a folder you no longer track

aleth-ingest browser-preview     # what your browser history would store (no engine)
aleth-ingest browser-import --confirm-preview   # store it
```

`history` prints one line per replacement, old memory to new:

```
  pricing.md
    2026-08-22T22:04:00+00:00
      da2b0a62-24b5-48c2-bef7-830e160580b7  ->  ac331252-fd06-4137-a04d-4158342d5551
```

Both ids are real memories you can look up. The old one is still there and
still readable; ask for it and your memory will tell you it is no longer
current and which memory took its place.

There is no "undo this replacement" command, because a memory that could be
un-replaced would not be much of a record. The way back is the way forward:
put the earlier text back in the file and run again. The old wording returns
as the current memory, and today's is retired behind it.

Common options:

```
--include GLOB     add an include glob (repeatable; default *.md,*.txt,*.rst)
--exclude GLOB     add an exclude glob (repeatable)
--all-formats      include every readable format, not just text

--manifest PATH    use a specific manifest file
--max-files N      cap the walk (default 1000); a truncated walk never removes
--max-supersessions N
                   cap replacements in ONE run (run; default 1000). Past the
                   cap a changed memory is retracted rather than replaced.
                   It still stops being current, it just does not name a
                   successor, and the run tells you the cap bound. It is
                   there so a sweeping mechanical edit across a whole folder
                   is something you see rather than something that happens.
--origin TAG       origin recorded on each memory (run; default "user")
--on-removed MODE  what happens to a memory when its file leaves the tree
                   (run): retract (default) stops it being current and keeps
                   the history; forget erases it permanently; keep leaves it
                   CURRENT. --hard-remove is a DEPRECATED alias for
                   "--on-removed forget", removed at the next major version.
--discovery-dir D  where the engine's discovery file lives (run)
--timeout SECONDS  wire timeout (run; default 30)
--scopes PATH      the list of tracked folders and records (default
                   ~/.aleth-ingest/scopes.json); every command that can
                   retract or erase takes it
```

`run` exits non-zero if any write fails, or if it held a change back because
another tracked folder could not be read.

Each folder you run joins a short list of tracked folders, which is how a run
knows what the others still hold. A tracked folder inside or above the one you
run is found even before it is on the list; any other folder you tracked with
an earlier version joins the list the next time you run it.
`aleth-ingest untrack <folder>` takes a folder off the list, for one that is
gone for good. It does not touch any memories.

## Browser history

The pages you read are worth remembering, and the usual way to collect them is
a browser extension that sends them somewhere. This does the opposite: your
browser already keeps its history in a database on your disk, and this reads a
copy of that file.

```
aleth-ingest browser-preview                    # look first. Stores nothing
aleth-ingest browser-import --confirm-preview   # then store what you saw

aleth-ingest browser-tracking                   # what is tracked, what is not
aleth-ingest browser-track   <domain>           # always store this one
aleth-ingest browser-block   <domain>           # stop storing new pages for it
aleth-ingest browser-category finance track     # switch a whole category
aleth-ingest browser-forget  <domain>           # erase everything you have of it
```

`browser-preview` prints one row per domain (pages, visits, first and last
time you were there) and a count of what it refused. Nothing is stored.
`browser-import` prints exactly the same table again and then **refuses**
unless you pass `--confirm-preview`, so nothing can reach your memory without
you having seen the list of places it came from.

Each page is stored as one memory holding the address, the title, how many
times you visited and when you first and last did. Run it again next week and
unchanged pages are skipped; a page whose title or visit count moved supersedes
its previous memory by name, the same way a changed file does.

```
--browser NAME          only this browser (repeatable: chrome, edge, brave, firefox)
--exclude-domain D      never read anything under this domain (repeatable)
--exclude-file PATH     a file of such domains, one per line
--max-urls N            cap one run (default 5000, most recently visited first)
--rows N                how many domain rows to print (default 40)
--manifest PATH         where the local state lives
                        (default ~/.aleth-ingest/browser-history.json)
--policy PATH           where your decisions live
                        (default ~/.aleth-ingest/browser-tracking.json)
--confirm-preview       (import) you have read the table and want it stored
--origin TAG            (import) origin recorded on each memory (default "user")
```

### Deciding what gets tracked

The categories below are blocked by **default**, not by decree. If you want an
assistant that can actually help with your money, it has to be allowed to see
your money. So you can switch a whole category on, and then still block the
one place inside it you would rather it never saw.

```
aleth-ingest browser-category finance track    # the whole category
aleth-ingest browser-block    mybank.example   # ... except this one
aleth-ingest browser-track    news.example     # or open a single domain
aleth-ingest browser-tracking                  # show me where things stand
```

Your decisions live in `~/.aleth-ingest/browser-tracking.json`, so you make
them once. The narrower one always wins: a domain you named beats whatever its
category says, in either direction. Anything you have never mentioned keeps the
shipped default, so installing this changes nothing until you use it. An
`--exclude-domain` you pass to `browser-import` is written into the same file,
so you do not have to remember it next time.

When a category is switched on, the preview says so in its own line, under the
counts, and names the policy file it read.

### Stopping and forgetting are not the same thing

**`browser-block` stops NEW records for a domain and KEEPS the existing ones.**
Telling a companion to stop following something is not telling it to delete
what it already knows, and it will never make that leap on its own initiative.

**`browser-forget` is the explicit erasure, and it is the only path that
deletes.** It permanently erases every memory this companion holds for that
domain, drops them from the local record, and blocks the domain so the next run
does not simply bring them back. It reports how many memories it erased, never
which pages they were. A memory whose exact text a tracked file or another
site's page also holds is kept and counted, because it is that source's memory
too; erase it there as well to remove it.

```
aleth-ingest browser-block  somewhere.example    # stop. Keep what you have
aleth-ingest browser-forget somewhere.example    # erase what you have
```

### Seeing all of it at once

The verbs above can change anything. None of them can SHOW you where you stand.
For that there is one page:

```
aleth-ingest browser-page
```

It opens a page listing every site your memory actually holds: how many pages
of each, when you last visited, whether it is tracked or blocked and why, with
the category switches at the top and an erase button on every row that tells you
exactly how much it will destroy before it destroys it. Erasing asks you to type
the site's name, because it is the one thing on that page that cannot be undone.

**The page never leaves your machine and never reaches the network.** It is
served on the loopback interface only, there is no flag to change that, the link
carries a one-time key that dies when you stop the command, and the page itself
loads no script, no font and no image from anywhere. A page that governs a
local-only memory has no business making a request of its own, and if you are
watching your network to check the claims in this file, this page must not be
the first thing you see.

The switches write the same file the verbs write, so the page and your terminal
can never disagree, and it re-reads that file on every request rather than
caching it. Nothing on the page needs the engine to be running except erasing,
which is the only action that has to reach it. You can always say "stop", even
when there is nothing running to be told.

### The privacy contract

**Everything here happens on your machine.** No page, domain or count leaves
it. There is no service to ask whether a site is sensitive, which is the
difference that matters: a tool that asks a server "is bank.example blocked?"
has just told that server you bank there.

**Your browser is never modified.** The history database is copied to a
temporary file, read from the copy, and the copy is deleted. That is also why
it works while the browser is running.

**These are never stored, and the preview shows only a count for each:**

- Finance, health, adult, sign-in and mail sites, by a built-in list of
  well-known hosts *and* by name: any host with `login`, `signin`, `sso`,
  `auth`, `account` or `passw` in it is treated as a sign-in page, and any
  host with `bank`, `banco`, `banca`, `banque` or `wallet` in it is treated as
  a money page. Password managers and national e-ID portals are sign-in pages.
- Anything on your own machine or network: `localhost`, private addresses, any
  name without a dot, `.local` and `.internal`.
- Anything that is not a web page: `file:`, `about:`, `chrome:`, `edge:`,
  `javascript:`, `data:`.
- Tracking parameters (`utm_*`, `fbclid`, `gclid` and their family) and any
  parameter whose name looks like a credential (`token`, `key`, `session`,
  `sig`, `password`, `code`, `state` and neighbours). That last rule is
  deliberately over-eager: it will drop a harmless `keywords=` along with a
  real `access_token=`, because a coarser address is cheap and a stored
  session token is not.

The category lists are short and readable on purpose: open
`aleth_ingest/browser_history.py` and check them. They are examples; the name
rules above are what catch the bank, the insurer or the intranet that no list
ever heard of. Anything they miss, `--exclude-domain` is yours.

**A page that leaves your browser's history is not retracted.** Browsers
expire history on their own schedule, and that is not you saying you never
read it. Memories of pages stay until you remove them yourself.

## Page text

A list of addresses is an index, not a memory. "What did that article about sleep
cycles actually say" is not answerable from a title, so this can fetch the pages you
already read and keep the readable text. Then a recall by **meaning** lands on
the words, with the source address attached.

**The reader runs on your machine.** The page is fetched by this program with Python's
own HTTP client and the text is extracted in the same process. There is no cloud
service, no language model and no third-party API anywhere in the path, so there is no
server that could be told which pages you read. The only network traffic is to the
page's own site: exactly the request your browser already made when you read it.

**And it is a separate decision from tracking, per domain.** Letting this read your
browser history is letting it read a file that is already on your disk. Fetching is
different: it opens a connection to somebody else's server, from your address, at a
moment you did not choose. So page text has its own opt-in and **nothing is fetched
for a domain you have not named**.

```
aleth-ingest page-text-opt-in  arstechnica.com   # yes, fetch pages from here
aleth-ingest page-text-preview                   # which pages? Fetches nothing
aleth-ingest page-text-fetch --confirm-preview   # fetch them, store the text
aleth-ingest page-text-status                    # what is opted in, what is stored
aleth-ingest page-text-opt-out arstechnica.com   # stop. Keep what you have
aleth-ingest page-text-forget  arstechnica.com   # erase the text you have
```

The urls come from the reading record `browser-import` wrote, so page text can only
ever fetch pages you actually visited. It does not follow links and it does not
discover anything: there is no seed list, no crawl and no way to point it at a site
you have never been to.

`page-text-preview` prints one row per domain and a count of what it refused, and
touches no network at all. `page-text-fetch` prints the same table again and then
**refuses** unless you pass `--confirm-preview`.

```
--domain D         only this domain (repeatable). Narrows; it grants nothing
--limit N          cap one run (default 25 pages, most recently visited first)
--delay SECONDS    between two pages of the SAME domain (default 2.0)
--max-bytes N      refuse a page larger than this (default 2000000)
--min-chars N      refuse a page with less readable text than this (default 400)
--timeout SECONDS  per page (default 20)
--wire-timeout S   for the memory daemon (default 30)
--dry-run          fetch, report, store nothing
--consent PATH     your opt-in file (default ~/.aleth-ingest/page-text.json)
--manifest PATH    page-text state (default ~/.aleth-ingest/page-text-manifest.json)
--history-manifest P  the reading record (default ~/.aleth-ingest/browser-history.json)
--policy PATH      your tracking decisions (default ~/.aleth-ingest/browser-tracking.json)
```

### What it will not fetch

**The block always wins over the opt-in.** A page is refused, before any connection is
opened, if it falls in a blocked category (finance, health, adult, sign-in, mail), if
you blocked its domain by name, if the domain is not opted in, if the address is not a
public web page, or if you have no stored record of having visited it. Blocking a
domain for history blocks it here too (you never have to say it twice), and the
counts are reported per reason, never per site, exactly as the history preview does.

**`robots.txt` is honoured, and there is no flag to turn that off.** A local reader
that ignores robots is a scraper with better manners, and that is not what this is. A
site that refuses to show its rules (401 or 403 on `robots.txt`) is treated as a no.

It fetches HTML and plain text only. A redirect is followed within the same site and
**refused if it leaves it**: storing another host's words under your address would
make the source line a lie. A page larger than `--max-bytes` is **refused rather than
truncated**: half an article stored as if it were the article is a memory nothing marks
as incomplete. A page whose readable text is shorter than `--min-chars` is skipped:
a cookie wall and a navigation menu match every query and answer none.

### What gets stored, and how it stays attached to the page

The text exactly as extracted, split into chunks on its own structure. **Every chunk is
linked to the memory of your visit to that page**, the one `browser-import` stored,
whose first line is the address. Every chunk, not just the first: a recall lands on
whichever chunk matched your question, and a chunk that cannot say where it came from
is an unattributed quotation. Ask your memory a question in your own words, get the
paragraph back, and the address is one step away from it.

That link is why **a page you have not imported is not fetched**. If there is no stored
record of your visit, there is nothing to attach the text to, and text stored loose can
never be traced back afterwards. So it is refused before anything is downloaded, and
the preview says so. Run `browser-import` first.

Nothing is summarised, classified or rewritten. Re-run it next month and an unchanged
page is skipped; a page that changed has its new text stored first and its old text
superseded behind it, so what the page used to say stays readable as history. A page
that has gone offline is **not** erased: a 404 is not you saying you never read it.

To erase what is stored for a site, `aleth-ingest page-text-forget <domain>` really
deletes the text and stops fetching that domain. It is a **separate store from the
visit records**, so erasing both is two commands and the output of each says so:

```
aleth-ingest page-text-forget somewhere.example   # erase the TEXT
aleth-ingest browser-forget   somewhere.example   # erase the VISIT records
```

## How re-runs stay clean

Each stored memory is the file's text, verbatim. Where it came from (path,
content hash, size, modified time, and which chunk it is) lives in the local
manifest, the companion's authoritative file-to-memory mapping. On the
next run it compares each file against the manifest by hash and only touches
what changed. Even without the manifest (say it was deleted), a re-ingest is
safe: the engine drops exact-duplicate content on its own.

That last property is also why an edit stays local. Because identical text
stores as the same memory, re-reading a document you barely touched hands back
the same memories for every unchanged part, so the companion recognises them
as still current and leaves them alone, and only the part you actually edited
is replaced.

## What is out of scope

Images, audio and video are not handled here. A scanned page needs OCR,
which needs software installed outside Python and gives you a confident guess
rather than the document's own words. It does not watch folders or run as a
daemon either: it ingests once when you run it, and again when you run it next.

For a document format, "verbatim" means the text the format's own reader gives
back, in reading order, with nothing added and nothing summarised. Layout,
images and styling are not stored. Plain text files are stored exactly as they
are on disk.

If a page has no text layer, the file is reported and skipped rather than
stored empty, so a later run can pick it up once you can read it. Nothing is
recorded for it in the meantime.

## License

Apache-2.0.
