Metadata-Version: 2.4
Name: Grimmerie
Version: 0.1.8
Summary: Functions for Prototyping, QOL and Sanity checking
Author: Joe Petrecca
License-Expression: MIT
Requires-Python: >=3.9
Description-Content-Type: text/markdown
Requires-Dist: transformers>=4.38
Requires-Dist: adapters>=1.0
Requires-Dist: numpy>=1.23
Requires-Dist: sentencepiece
Requires-Dist: scikit-learn>=1.2
Requires-Dist: pandas>=1.5
Provides-Extra: nlp
Requires-Dist: spacy>=3.0; extra == "nlp"
Provides-Extra: specter
Requires-Dist: torch>=2.0; extra == "specter"

# Grimmerie

Grimmerie is a small collection of high-level Python utilities for rapid NLP
prototyping, vectorization, clustering review, and data inspection.

The package is built around a simple idea:

```python
result = spell(data)
```

Each spell hides common setup while keeping useful control over its inputs and
outputs.

## Current Spells

- `specterize`: Generate SPECTER2 document embeddings.
- `tfidfize`: Generate configurable TF-IDF representations.
- `tfidf_features`: Inspect the learned TF-IDF vocabulary.
- `tfidf_document_frequency`: Count how many documents contain each feature.
- `cluster_viz`: Write an interactive HTML cluster inspector.

## Installation

```bash
pip install grimmerie
```

Grimmerie supports Python 3.9 and newer.

### Optional NLP Features

The base package includes TF-IDF and embedding dependencies. Install the NLP
extra only if you want spaCy lemmatization:

```bash
pip install "grimmerie[nlp]"
python -m spacy download en_core_web_sm
```

The first call to `specterize` downloads the SPECTER2 model, tokenizer, and
adapter from Hugging Face. These files are cached by the underlying libraries.

## Quick Start

```python
import pandas as pd

from grimmerie import cluster_viz, specterize, tfidfize

df = pd.DataFrame({
    "title": [
        "Pre-training of Deep Bidirectional Transformers",
        "Attention Is All You Need",
        "Document-level Representation Learning",
    ],
    "abstract": [
        "A language representation model based on bidirectional training.",
        "A sequence transduction architecture based on attention.",
        "A method for representing scientific documents.",
    ],
})

text = df["title"] + " " + df["abstract"]

tfidf = tfidfize(text, return_type="array")
embeddings = specterize(text, return_type="numpy")
```

Both vectorizers preserve input order: row `i` in the output corresponds to
row `i` in the input.

## Input Handling

The vectorization spells accept:

- A single string
- A dictionary
- A list
- A pandas Series
- Other iterables
- Scalar values, which are converted to strings

A dictionary is converted into one text item by joining its values:

```python
record = {"title": "BERT", "abstract": "A language model"}
embedding = specterize(record, return_type="numpy")
```

Lists containing dictionaries are also supported:

```python
records = [
    {"title": "BERT", "abstract": "A language model"},
    {"title": "SPECTER", "abstract": "A document model"},
]

embeddings = specterize(records, return_type="numpy")
```

Input normalization is intentionally convenient rather than schema-aware. If
you need a specific field order or formatting, combine your fields into a
string column before calling a spell.

## `specterize`

```python
specterize(
    input_data,
    return_type="list",
    max_length=512,
)
```

`specterize` uses the SPECTER2 base model and adapter:

- Base model: `allenai/specter2_base`
- Adapter: `allenai/specter2`

The input is tokenized, truncated to `max_length`, and passed through the
model. The first token representation is returned as the document embedding.
The standard output has 768 features per input item.

### Return Types

| `return_type` | Result |
| --- | --- |
| `"list"` | Python `list[list[float]]` |
| `"numpy"` | NumPy array with shape `(n, 768)` |
| `"tensor"` | PyTorch tensor with shape `(n, 768)` |

Example:

```python
texts = [
    {"title": "BERT", "abstract": "We introduce a new model."},
    {"title": "Attention", "abstract": "Transformers use attention."},
]

embeddings = specterize(texts, return_type="numpy")
print(embeddings.shape)
# (2, 768)
```

### SPECTER2 Notes

- The first call may download model data.
- PyTorch must be installed separately or through `grimmerie[specter]`.
- Models are cached after loading.
- Model and tokenizer objects are cached in the module process.
- Text longer than `max_length` is truncated.
- Inference runs under `torch.no_grad()`.
- The current implementation runs inference on the default PyTorch device.

## `tfidfize`

```python
tfidfize(
    input_data,
    *,
    lemmatize=False,
    spacy_model="en_core_web_sm",
    batch_size=2000,
    n_process=1,
    progress_interval=None,
    min_df=1,
    max_df=1.0,
    stop_words="english",
    ngram_range=(1, 1),
    lowercase=True,
    max_features=None,
    norm="l2",
    use_idf=True,
    smooth_idf=True,
    sublinear_tf=False,
    return_type="sparse",
    return_vectorizer=False,
    vectorizer=None,
)
```

By default, `tfidfize` fits scikit-learn's `TfidfVectorizer` on the supplied
text and returns a sparse matrix.

### Return Types

| `return_type` | Result |
| --- | --- |
| `"sparse"` | SciPy sparse matrix |
| `"array"` | Dense NumPy array |
| `"list"` | Python list of lists |
| `"frame"` | pandas DataFrame with vocabulary columns |

Use sparse output for large vocabularies. Dense output is convenient for small
experiments but can use substantially more memory.

```python
X = tfidfize(
    df["title"] + " " + df["abstract"],
    return_type="array",
)
```

### Reusing a Vectorizer

Use `return_vectorizer=True` to keep the fitted vectorizer:

```python
X_train, vectorizer = tfidfize(
    train_text,
    return_type="sparse",
    return_vectorizer=True,
)

X_test = tfidfize(
    test_text,
    vectorizer=vectorizer,
    return_type="sparse",
)
```

When `vectorizer` is supplied, Grimmerie calls `transform` rather than fitting
a new vocabulary.

### Lemmatization

Set `lemmatize=True` to preprocess text with spaCy before TF-IDF:

```python
X = tfidfize(
    text,
    lemmatize=True,
    spacy_model="en_core_web_sm",
    return_type="array",
)
```

The lemmatization pipeline keeps alphabetic, non-stopword lemmas in lowercase.
Use `batch_size`, `n_process`, and `progress_interval` to control spaCy
processing.

### Important TF-IDF Parameters

| Parameter | Purpose |
| --- | --- |
| `min_df` | Ignore terms appearing in fewer documents than this threshold. |
| `max_df` | Ignore terms appearing in more documents than this threshold. |
| `stop_words` | Stopword configuration passed to scikit-learn. |
| `ngram_range` | Minimum and maximum n-gram sizes. |
| `max_features` | Optional vocabulary size limit. |
| `norm` | Row normalization, usually `"l1"`, `"l2"`, or `None`. |
| `use_idf` | Enable inverse document-frequency weighting. |
| `smooth_idf` | Smooth inverse document-frequency values. |
| `sublinear_tf` | Use logarithmic term-frequency scaling. |
| `lowercase` | Lowercase text during vectorization. |

## TF-IDF Helpers

### `tfidf_features`

Return the learned feature names from a fitted `TfidfVectorizer`:

```python
from grimmerie import tfidf_features

X, vectorizer = tfidfize(
    text,
    return_vectorizer=True,
)

features = tfidf_features(vectorizer)
print(features[:10])
```

The feature order matches the columns of the TF-IDF matrix.

### `tfidf_document_frequency`

Count the number of documents containing each feature:

```python
from grimmerie import tfidf_document_frequency

document_frequency = tfidf_document_frequency(X)
```

The returned NumPy array has one value per feature and follows the same
feature order as the matrix columns.

## `cluster_viz`

`cluster_viz` writes a standalone interactive HTML cluster inspector from a
pandas DataFrame. It does not modify the input DataFrame.

```python
cluster_viz(
    dataframe,
    cluster_id_column,
    *,
    entry_columns=None,
    truth_value_column=None,
    truth_true_value=True,
    truth_false_value=False,
    truth_unknown_value=None,
    output_path,
    title=None,
    title_column=None,
)
```

`output_path` and at least one `entry_columns` value are required.

### Basic Visualization

Use `cluster_viz` without a truth column when you only want to inspect cluster
membership and row data:

```python
from grimmerie import cluster_viz

cluster_viz(
    df,
    cluster_id_column="cluster",
    entry_columns=["title", "orcid", "url"],
    output_path="clusters.html",
    title="Paper Clusters",
)
```

Open `clusters.html` in a browser. Each bubble represents a cluster and its
size represents the number of visible entries. Selecting a bubble opens its
entries in the side panel.

A cluster ID of `-1` is displayed as `Noise`.

### Truth-Aware Visualization

The truth column is a ternary status column. It can use any values you choose
for true, false, and unknown:

```python
cluster_viz(
    df,
    cluster_id_column="cluster",
    entry_columns=["title", "orcid", "url"],
    truth_value_column="truth",
    truth_true_value=82829383,
    truth_false_value="santa",
    truth_unknown_value="",
    output_path="clusters.html",
    title="Author Resolution Review",
)
```

Values matching the configured true and false values are treated as known.
Values that match neither are treated as unknown. Unknown entries remain
available for inspection but are excluded from metric calculations.

When a truth column is configured, the header provides global filters:

- `All`: Show every entry.
- `True`: Show only true entries.
- `False`: Show only false entries.
- `Unknown`: Show only unknown entries.

The filters update visible entries, bubble sizes, entry counts, cluster counts,
and noise counts. The filter does not change the metric calculations.

Every displayed entry has a copy button that copies the first configured entry
column. The side panel also includes actions for copying all selected entries
or copying the selected entries as TSV.

### Truth-Aware Metrics

Metric cards appear only when a truth column is supplied. The cards display the
global mean, median, and a compact distribution for:

- Precision
- Recall
- F1
- Split score
- True-reference B3 precision
- True-reference B3 recall
- True-reference B3 F1

Metrics use the following scope:

1. True entries define the target population.
2. Only predicted clusters containing at least one true entry are evaluated.
3. False entries inside those clusters affect precision and B3 precision.
4. False-only clusters are ignored.
5. Unknown entries are excluded from metric denominators.

The split score measures the share of true entries retained in the largest
evaluated predicted cluster. The B3 metrics are evaluated for true entries,
using the known entries in their predicted clusters for precision and the full
true population for recall.

### Visualization Titles

Use `title` for an explicit name:

```python
cluster_viz(
    df,
    cluster_id_column="cluster",
    entry_columns=["title"],
    output_path="clusters.html",
    title="Block 17",
)
```

Alternatively, use the first non-null value from a DataFrame column:

```python
cluster_viz(
    df,
    cluster_id_column="cluster",
    entry_columns=["title"],
    title_column="block_name",
    output_path="clusters.html",
)
```

`title` takes precedence over `title_column`. If neither is provided, the
visualization is named `Cluster Inspector`.

### Browser Requirements

The generated HTML loads D3.js and web fonts from public CDNs. The file is
otherwise self-contained, but the browser needs network access to load those
resources. Copy buttons use the browser Clipboard API, which may be restricted
when opening a file directly in some browser configurations.

## Saving Outputs

### Dense NumPy Output

```python
import numpy as np

X = tfidfize(text, return_type="array")
np.save("tfidf.npy", X)
```

### Sparse TF-IDF Output

```python
from scipy import sparse

X = tfidfize(text, return_type="sparse")
sparse.save_npz("tfidf.npz", X)
```

### HTML Visualization

The visualization is written directly to the requested path:

```python
cluster_viz(
    df,
    cluster_id_column="cluster",
    entry_columns=["title"],
    output_path="reports/clusters.html",
)
```

The parent directory must already exist.

## Common Workflows

### Compare TF-IDF and SPECTER2 Representations

```python
text = df["title"] + " " + df["abstract"]

tfidf_matrix = tfidfize(text, return_type="array")
specter_embeddings = specterize(text, return_type="numpy")

# Row i in both outputs corresponds to df.iloc[i].
```

### Review Clustering Results

```python
from grimmerie import cluster_viz

df["cluster"] = predicted_cluster_labels

cluster_viz(
    df,
    cluster_id_column="cluster",
    entry_columns=["title", "orcid", "url"],
    truth_value_column="truth",
    truth_true_value="confirmed",
    truth_false_value="rejected",
    truth_unknown_value="unknown",
    output_path="cluster_review.html",
    title="Resolution Review",
)
```

## Troubleshooting

### `spaCy is required for lemmatization`

Install the optional dependency:

```bash
pip install "grimmerie[nlp]"
```

Then download the configured model:

```bash
python -m spacy download en_core_web_sm
```

### Model Downloads Are Slow

The first SPECTER2 call downloads model files from Hugging Face. Later calls
reuse the local cache. A network connection is required for the first call
unless the model has already been cached.

### `empty vocabulary; perhaps the documents only contain stop words`

This is a scikit-learn error. Check that the input contains usable text and
that your stopword, `min_df`, `max_df`, and preprocessing settings do not
remove every token.

### Missing Cluster Columns

`cluster_viz` validates every configured column and raises an error listing
missing columns. Check `cluster_id_column`, `entry_columns`,
`truth_value_column`, and `title_column` against `dataframe.columns`.

### Missing Cluster IDs

Cluster IDs cannot be missing. Fill or remove missing IDs before calling
`cluster_viz`.

### Large Outputs

Prefer sparse TF-IDF output for large vocabularies. Dense matrices can require
substantial memory. The HTML cluster inspector embeds all selected entry data
in the generated file, so very large DataFrames can produce large HTML files.

## Design Principles

- One-call workflows for common experiments.
- Strong defaults for rapid prototyping.
- Consistent row alignment across vectorization spells.
- Explicit output formats where downstream code needs control.
- Interactive inspection when aggregate metrics are not enough.

## Limitations

Grimmerie is designed for experimentation and review rather than fully
managed production pipelines. You may want lower-level control when you need:

- Exact model device placement and batching controls.
- Fully reproducible model and dependency locking.
- Custom SPECTER2 pooling behavior.
- Very large-scale visualization.
- Custom browser asset hosting instead of CDN resources.

## Public API

The package exports the following names:

```python
from grimmerie import (
    cluster_viz,
    specterize,
    tfidf_document_frequency,
    tfidf_features,
    tfidfize,
)
```
