Metadata-Version: 2.4
Name: tracecite
Version: 0.0.6
Summary: Source-linked table normalisation and evidence retrieval for executable documentation
Author: ARPST-UniMelb
License-Expression: MIT
Project-URL: Homepage, https://github.com/yasirroni/tracecite
Project-URL: Documentation, https://yasirroni.github.io/tracecite/
Project-URL: Issues, https://github.com/yasirroni/tracecite/issues
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: MacOS
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: Text Processing :: Indexing
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: evidence
Requires-Dist: huggingface-hub<2,>=1.23; extra == "evidence"
Requires-Dist: pymupdf<2,>=1.28; extra == "evidence"
Requires-Dist: sentence-transformers<6,>=5.6; extra == "evidence"
Requires-Dist: sqlite-vec<0.2,>=0.1.9; extra == "evidence"
Provides-Extra: test
Requires-Dist: build>=1.2; extra == "test"
Requires-Dist: pytest<9,>=8.4; extra == "test"
Dynamic: license-file

# TraceCite

TraceCite lets people and AI agents search their documents, cite the exact page or cell, and check that a quote matches its source.

TraceCite indexes Markdown, PDF, `.xlsx`, and `.xlsm` files. It stores retained text, source hashes, full-text and vector search records, and locators (PDF pages, Markdown lines, and worksheet ranges), and verifies quotations and report citations against them.

[tracecite-docs](https://pypi.org/project/tracecite-docs/) builds Quarto sites from executable Python and Julia pages, normalises their tables into source-linked row text for search, and indexes the retained pages with TraceCite. The [TraceCite documentation](https://yasirroni.github.io/tracecite/) covers both packages.

## Architecture

```text
declared PDF, Markdown, or workbook sources
    -> retained source extraction
    -> locator-aware chunks
    -> SQLite FTS5 and sqlite-vec
    -> ranked JSON results with source identity and evidence locators
```

## Installation

TraceCite needs Python 3.11 or newer. Evidence synchronisation and hybrid search need the `evidence` extra:

```sh
pip install "tracecite[evidence]"
```

The `evidence` extra installs PyTorch through `sentence-transformers`. On Linux, pip installs the CUDA build by default, a download of several gigabytes. On a machine without a GPU, install the CPU build first:

```sh
pip install torch --index-url https://download.pytorch.org/whl/cpu
```

Workbook parsing reads OOXML packages with only the Python standard library, without Excel, Julia, Quarto, or third-party workbook readers.

## Usage

The `tracecite` CLI syncs, searches, and verifies source evidence, and installs the bundled agent skills.

### Index and search source evidence

List sources in a manifest by explicit path or by include and exclude globs over `.pdf`, `.md`, `.xlsx`, and `.xlsm` files.

```toml
schema_version = 1

[[source]]
path = "2023-iasr-ev-workbook.xlsx"
```

Sync the sources into a database, search it lexically and semantically, then check its integrity:

```sh
tracecite sync \
  --root sources \
  --manifest tracecite-sources.toml \
  --database .tracecite/evidence.sqlite \
  --model-cache-dir .tracecite/model-cache

tracecite search \
  "weekday versus weekend electric vehicle charging behaviour" \
  --database .tracecite/evidence.sqlite \
  --model-cache-dir .tracecite/model-cache

tracecite doctor --database .tracecite/evidence.sqlite
```

#### Sync

- `sync` needs at least one `[[source]]` or `[[include]]` rule; without one, it stops without creating or changing the database.
- It prints one line per added, reparsed, renamed, missing, or unselected path; `--json` prints the full report.
- A declared `[[source]]` missing under the root is listed under `missing-sources`. A selected file that sync cannot read or parse, or that is a symlink, is listed under `skipped` with the reason; a skipped file indexed earlier keeps its previous rows. Either case ends with `status: incomplete` and exit 1, and the other sources still sync. A sync of one path checks only that path.
- A second `sync` or `prune --apply` of the same database waits for the first to finish. The wait uses an empty `<database>.lock` file beside the database.
- PDF pages without text, such as blank or scanned pages, are listed under `pages-without-text` when they are parsed, and `doctor` lists them on every run. Scanned pages need [Tesseract](https://github.com/tesseract-ocr/tesseract) for OCR; without it, sync stores those pages without text. After installing it, run `tracecite sync --full` to extract their text.
- `sync` is incremental and non-pruning. Sources that are gone or no longer selected stay searchable and citable until you preview and apply `tracecite prune`.

#### Prune

`prune` needs the source root (`--root` or a profile). It targets indexed sources the rules no longer select or whose file is gone from the root; the preview lists the latter under `missing_paths`. `--all` targets every indexed source and needs no root. `prune --apply` also compacts the database file.

#### Doctor

`doctor` warns about PDF pages without text, and with `--root <dir>` about sources that changed or disappeared since the last sync. Only database, vector, and asset integrity issues change its exit status; the last line counts the warnings.

#### Search

`search --source-type` (`pdf`, `markdown`, or `workbook`; repeatable) filters lexical and vector candidates before ranking and the `--limit` cut, not after:

```sh
tracecite search "coal retirement" \
  --database .tracecite/evidence.sqlite \
  --source-type pdf
```

- An empty query exits with status 2.
- `--lexical-only` uses full-text search alone and skips loading the embedding model.
- Chinese, Japanese, and Korean (CJK) text has no spaces, so lexical search also matches each CJK run of the query as a substring of the indexed text. This reads every passage, so it is slower than ordinary full-text search on a large database.
- The embedding model is English-only, so vector search leaves out the CJK text of a query. A query with only CJK text uses lexical search only.
- Byte-identical files at different paths give one hit, under the shortest path (then the first in sort order), with the others in `duplicate_paths`.

Each result includes retrieval provenance and a source-specific locator. This abbreviated workbook result shows the stable fields; the worksheet and ranges vary with the workbook and chunk settings:

```json
{
  "source_path": "inputs-and-assumptions.xlsx",
  "source_type": "workbook",
  "source_sha256": "<64-character SHA-256>",
  "locator": {
    "kind": "excel-range",
    "sheet": "<worksheet name>",
    "range": "<bounding A1 range>",
    "range_kind": "bounding",
    "exact_ranges": ["<exact A1 range>"]
  },
  "provenance": ["lexical", "vector"]
}
```

The path and SHA-256 identify the workbook version, and the worksheet and `exact_ranges` identify the passage's cells. `range` is only the bounding rectangle and can include other cells. Rank only marks candidate evidence, so inspect the returned cells before citing a result.

A portable workbook citation is the path, SHA-256, worksheet, and A1 ranges. Opening the exact range needs a workbook viewer or a separate provider-specific link.

#### Workbook boundaries

- The parser supports `.xlsx` and `.xlsm` OOXML packages, not legacy binary `.xls`.
- It keeps stored text, numeric, Boolean, error, formula, and cached formula values. Cached values reflect the workbook's stored state; formulae are not recalculated.
- It reads stored package content only and does not run VBA or refresh external links, Power Query, pivot caches, or data connections.
- It ignores style-only empty cells and does not reconstruct display formatting, so styled dates and times can stay numeric serials.
- It chunks workbooks by row and does not yet infer formal Excel tables or carry complex multi-row headers into every row.

### Retrieve retained pages and visual evidence

`tracecite page` prints retained PDF page text from the SQLite evidence database, physical page 1 by default. Selectors combine pages and closed or open ranges; overlaps are merged and pages come back in ascending physical order.

```sh
tracecite page reports/example.pdf --database evidence.sqlite
tracecite page reports/example.pdf 5,12-15,20 --database evidence.sqlite
tracecite page reports/example.pdf 97- --database evidence.sqlite
tracecite page reports/example.pdf -20 --database evidence.sqlite
tracecite page reports/example.pdf all --database evidence.sqlite
```

Only `all` implicitly selects the whole source, and it must stand alone. Multi-page text output marks each physical page, and a page with no text, such as a scanned page synced without OCR, adds a note on stderr. `--format json` returns an ordered array of retained text, extraction metadata, a citation link, and validated paths to the indexed page render and figure crops. Search results carry the same render and crop fields for page-local PDF matches.

`extract-pages` takes the same selectors and writes a derivative PDF and provenance manifest from the indexed source PDF. The output directory must already exist outside the source root; stale sources, symlinked output directories, overlap, and overwrites are refused.

```sh
mkdir -p ../tracecite-exports
tracecite extract-pages reports/example.pdf 5,12-15,20 \
  --database evidence.sqlite \
  --output-dir ../tracecite-exports
```

Answer page-local questions from database text or indexed page assets; select `all` only when you need the whole source.

### Verify quotations

`tracecite verify quote` checks a quotation against the retained text of one physical page, or a contiguous range such as `4-5` for a quotation that crosses a page break:

```sh
tracecite verify quote reports/example.pdf 4-5 "Exact quotation" --database evidence.sqlite
```

- `exact`: the quotation occurs verbatim.
- `normalised`: matches after collapsing whitespace, straightening curly quotes and apostrophes, and ignoring case.
- `not-found`: anything else, including one changed word.
- `page-not-indexed` or `source-not-indexed`: the page range or the source is not in the database; these exit 2, other failures exit 1.

Verification reads only the indexed text, so run `tracecite sync` after a file changes to check against the new version.

### Install the agent skills

TraceCite bundles three AI agent skills: `using-tracecite`, `writing-evidence-backed-reports`, and `routing-documentation-source-links`. Copy them into the project's `.agents/skills/` directory:

```sh
tracecite skills install
```

`--target` picks another directory, such as Claude Code's personal skills:

```sh
tracecite skills install --target ~/.claude/skills
```

Rerunning it replaces the copies with the installed version's skills.

## Contributing

### Environment

```sh
uv sync --extra evidence --extra test
```

`uv sync` also installs `tracecite-docs` from the `docs` dependency group to build the documentation.

`requirements.txt` pins the evidence dependencies to the newest versions `pyproject.toml` allows; install it for a reproducible environment. Refresh it with [`pru`](https://github.com/yasirroni/pru); installing `-e ".[evidence]"` in the same command keeps upgrades within `pyproject.toml` bounds:

```sh
uv pip install -r requirements.txt
pru -r requirements.txt upgrade_requirements --cmd 'uv pip install --upgrade -e ".[evidence]"'
```

### Build the documentation

The build needs [Quarto](https://quarto.org/), which bundles Pandoc, and Julia 1.10 or newer for the Julia pages. Set up the Julia environment once:

```sh
julia --project=. -e 'using Pkg; Pkg.instantiate()'
```

```sh
uv run tracecite-docs build docs
uv run tracecite-docs build docs_quarto_py --only python
uv run tracecite-docs build docs_quarto_jl --only julia
```

<!-- TODO:
Explain the expected `[WARNING] Could not fetch resource`
-->

Without Julia, the builder names each Julia source it skips and renders the Python overlay, leaving Julia sources untouched. `--only python` or `--only julia` builds one language.

`docs/examples/report-adoption/aemo-isp-comparison/` is a complete author -> check -> index -> search -> doctor -> publish-only example using two AEMO Integrated System Plan reports.

`docs/examples/report-adoption/aemo-isp-comparison/docs/tracecite.toml` demonstrates the documentation evidence contract. Schema version 1 defines exactly `authored_root`, `retained_root`, `staged_root`, `source_links`, `index_output`, `publication_exclude`, and optional `host_render_command` under `[docs]`. Contract paths are repository-relative and stay inside the repository.

Open the built site:

```sh
open docs/build/index.html
```

### Lint and format

On each commit, [pre-commit](https://pre-commit.com/) runs [Ruff](https://docs.astral.sh/ruff/) on changed Python files and [ty](https://github.com/astral-sh/ty) on the whole project. Install it once and enable the hook:

```sh
uv tool install pre-commit
pre-commit install
```

`pre-commit run --all-files` checks every file. [Pyright](https://github.com/microsoft/pyright), the checker behind Pylance, runs only on request with `pre-commit run --hook-stage manual pyright --all-files`.

### Test

Python tests and build:

```sh
python -m pytest
uv build
```

The Julia package is tested by `docs/examples/julia/dirty_dataframe.jl`, which the documentation build runs with its hidden checks:

```sh
tracecite-docs build docs
```
