Metadata-Version: 2.4
Name: sid-sdk
Version: 0.2.0
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Programming Language :: Rust
Classifier: Typing :: Typed
License-File: LICENSE
License-File: THIRD_PARTY_NOTICES.md
Summary: Context-management primitives for rendering search results and tracking document spans shown to language models
Keywords: context-management,retrieval,search,snippets,llm
Home-Page: https://github.com/sidhq/sid-python
Author: SID
License-Expression: MIT
Requires-Python: >=3.12
Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
Project-URL: Documentation, https://github.com/sidhq/sid-python#readme
Project-URL: Homepage, https://github.com/sidhq/sid-python
Project-URL: Issues, https://github.com/sidhq/sid-python/issues
Project-URL: Repository, https://github.com/sidhq/sid-python

# SID SDK

`sid-sdk` turns search results into compact, model-facing document views. It
assigns stable short IDs, selects relevant snippets, tracks the exact character
ranges already shown, masks repeated text, and renders SID's `<doc>` format.

Snippet selection is implemented in Rust with
[Alyze](https://github.com/turbopuffer/alyze) Unicode UAX #29 tokenization and
separate unigram and bigram BM25 scores. Returned ranges are Python character
offsets into the original text.

## Installation

```console
pip install sid-sdk
```

Prebuilt ABI3 wheels require no Rust compiler on supported macOS, Linux, and
Windows platforms. Building from the source distribution requires Python 3.12+
and Rust 1.88 or newer.

## Basic use

```python
from sid import DocumentCache

cache = DocumentCache(language="english")
cache.add_document(
    "database-id",
    {"title": "Example", "content": "The complete document text ..."},
)
view = cache.apply_snippet(
    "database-id",
    snippet_field="content",
    query="complete document",
)
print(view.render_xml())
cache.update_seen(view)
```

Named analyzers are available for Danish, Dutch, English, Finnish, French,
German, Hungarian, Italian, Norwegian, Portuguese, Russian, Spanish, and
Swedish. They lowercase tokens and remove Alyze's language-specific stopwords.
Use `language="generic"` for lowercase Unicode tokenization without stopword
removal. Chinese, Japanese, and Korean text is handled through `generic` UAX
#29 segmentation; there are no dedicated language entries. `apply_snippet`
accepts a one-call `language=` override, and forks inherit the cache default.

If every query token is a stopword, all windows have the same score and the
selector deliberately returns the beginning of the document.

## Character ranges

`DocumentCache(range_mode="lenient")` is the default. A partially overlapping
range is reduced to its exact intersection with the document, while an empty,
inverted, malformed, or wholly disjoint range raises `InvalidCharacterRange`.
Use `range_mode="strict"` to require `0 <= start < end <= len(content)` exactly.
Ranges are half-open Python character offsets and are never expanded to word
boundaries, reversed, or replaced with the complete document.

Call `cache.resolve_char_range(data_id, snippet_field, (start, end))` before
passing a model-provided range onward. `get_single_span_document_view` applies
the same configured policy, and rendered IDs always describe the exact slice
displayed.

## Rendering and forks

XML document bodies and attribute values escape `&`, `<`, and `>`; quotes are
left unchanged. Falsy attributes are omitted from XML. Markdown tables stringify
their values normally, so empty strings make empty cells while `0` and `False`
remain visible. Snippet-less views require explicit `display_fields`.

Forks share one document store, ID mapping, and ID stream. A document added by
any family member is immediately readable by the others, and adding the same
`data_id` again is an idempotent lookup. Each fork keeps its own seen ledger,
initialized from the parent's current ledger, so agents can track what they
have shown independently.

Concurrent document additions are safe across a fork family, and separate
forks may run on separate threads. A single fork is intended for one agent;
`apply_snippet` and `update_seen` are separate ledger operations and are not an
atomic read-modify-write pair when called concurrently on the same fork.

## Development

```console
uv sync --group dev
uv run maturin develop --locked
uv run pytest -rxX --strict-markers
cargo test --locked
```

## License

MIT. See [LICENSE](LICENSE) and
[THIRD_PARTY_NOTICES.md](THIRD_PARTY_NOTICES.md).

