Metadata-Version: 2.4
Name: iconoscope
Version: 0.2.2
Summary: python package for exploring image collections using visual similarity based on image embeddings extracted from models
Author: rlskoeser
Author-email: rlskoeser <rebecca.s.koeser@princeton.edu>
License-Expression: Apache-2.0
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Classifier: Topic :: Scientific/Engineering :: Image Processing
Classifier: Topic :: Scientific/Engineering :: Visualization
Requires-Dist: h5py>=3.16.0
Requires-Dist: numpy>=2.0
Requires-Dist: pillow>=10.0
Requires-Dist: polars>=1.41.2
Requires-Dist: iconoscope[embed,analysis] ; extra == 'all'
Requires-Dist: lap>=0.4 ; extra == 'analysis'
Requires-Dist: scikit-learn>=1.2 ; extra == 'analysis'
Requires-Dist: scipy>=1.14 ; extra == 'analysis'
Requires-Dist: tqdm>=4.68.2 ; extra == 'analysis'
Requires-Dist: umap-learn>=0.5 ; extra == 'analysis'
Requires-Dist: accelerate>=1.14.0 ; extra == 'embed'
Requires-Dist: torch>=2.12.0 ; extra == 'embed'
Requires-Dist: torchvision>=0.27.0 ; extra == 'embed'
Requires-Dist: tqdm>=4.68.2 ; extra == 'embed'
Requires-Dist: transformers>=5.12.1 ; extra == 'embed'
Requires-Python: >=3.12
Provides-Extra: all
Provides-Extra: analysis
Provides-Extra: embed
Description-Content-Type: text/markdown

# iconoscope

`iconoscope` is a python package for creating an image dataset with extracting features using vision or multimodal models including DINOv2/3 and CLIP. An image dataset with embeddings can be used to generate a static mosaic where visually similar images are placed near each other, or for exploration of similar images in a notebook.

The name **iconoscope** comes from the Greek words for _image_ (εἰκών) and _to see_ (σκοπεῖν). The iconoscope
"was the first practical video camera tube to be used in early television cameras" ([Wikipedia](https://en.wikipedia.org/wiki/Iconoscope)). Fun fact: the photosensitive system within the iconoscope was referred to as a "mosaic"!

![iconoscope diagram](images/iconoscope_diagram.png) ![brightly colored iconoscope displaying a grid of images, with sparks and networks coming out the sides.](images/gemini_logo.png) ![black and white logo based on iconoscope diagram.](images/bw_logo.png)

_From left to right: iconoscope diagram from Wikipedia; Logo image generated by Gemini (prompt: `icon for a software library to visualize images based on embeddings; inspired by iconoscope video tube`); my attempt at a black and white logo based on the diagram and letters._

## Install

Install with `pip`, [uv](https://docs.astral.sh/uv/), or similar.

```bash
pip install "iconoscope[all]"
```

The base package only reads and writes image datasets (`create`, `info`,
and loading stored data in Python), with minimal dependencies (`h5py`, `numpy`,
`pillow`, `polars`). It can be used in WASM/Pyodide environments such as marimo
notebooks running in the browser. Other commands need optional dependencies:

| extra | commands | installs |
|---|---|---|
| `embed` | `embed` | torch, torchvision, transformers, accelerate |
| `analysis` | `umap`, `cluster`, `mosaic` | umap-learn, scikit-learn, scipy, lap |
| `all` | everything | both of the above |

For example, `pip install "iconoscope[analysis]"` to generate mosaics from a
dataset that already has embeddings.

## Usage

iconoscope functionality is currently command-line only. The typical workflow
is:

- `create` an image dataset based on a directory of images (may be nested); saves an HDF5 file
- `embed` those images with one or more models and save the features
- Generate a `mosaic` from model embeddings (automatically generates `umap` coordinates)
- You can optionally generate `umap` coordinates or `cluster` assignments.

Run `iconoscope info` to get a report of the data included in a dataset.

Run `iconoscope <command> --help` to see available options for each command.

### create

Create an image inventory by discovering and validating local images. This step
does not load an embedding model.

```console
iconoscope create path/to/images/ my_collection.h5
```

iconoscope will search for image files anywhere within the specified image
directory. Supported image extensions (case-insensitive) are
`.jpg`, `.jpeg`, `.png`, `.tiff`, `.webp`, and `.avif`. Use `--limit N` to
include at most N images (an arbitrary subset).

### embed

Extract features with a pre-trained image model and save them to an existing
dataset:

```console
iconoscope embed my_collection.h5
iconoscope embed my_collection.h5 --model siglip --center-crop 60 --grayscale
```

Available models (`-m/--model`):

| name | Hugging Face model |
| -------- | ------------------------------------------------------------------------------------------------------------- |
| `dinov2` | [`facebook/dinov2-base`](https://huggingface.co/facebook/dinov2-base) (default) |
| `dinov3` | [`facebook/dinov3-vitb16-pretrain-lvd1689m`](https://huggingface.co/facebook/dinov3-vitb16-pretrain-lvd1689m) |
| `resnet` | [`microsoft/resnet-50`](https://huggingface.co/microsoft/resnet-50) |
| `siglip` | [`google/siglip-base-patch16-224`](https://huggingface.co/google/siglip-base-patch16-224) |
| `clip` | [`openai/clip-vit-base-patch32`](https://huggingface.co/openai/clip-vit-base-patch32) |

Preprocessing options:

- `--center-crop CROP`: square center crop retaining CROP percent (0–100\] of
  the shortest edge; replaces the model's default resize/crop
- `--grayscale`: convert images to grayscale before embedding

Each model + preprocessing combination is stored in the dataset under a separate
_configuration_, labeled `model/key` (e.g. `dinov2/defaults`,
`siglip/center_crop=0.6_grayscale`), so multiple configurations can coexist in
one dataset. If features for a configuration already exist, `embed` does
nothing unless `-u/--update` is given; updating replaces previously saved features and
removes any UMAP or cluster results associated with the previous features.

If an image becomes unreadable after the inventory is created, embedding
skips it. When adding the first configuration, the skipped image is removed
from the dataset's shared image inventory so paths and feature rows remain
aligned. Once any features have been saved, images cannot be removed because
that would invalidate existing rows; embedding reports an error instead.

### Selecting stored features

Commands that use stored features (`mosaic`, `cluster`, `umap`) select a
configuration either with the same `--model` / preprocessing options used for
`embed`, or with `-c/--config MODEL/KEY` (as shown by `info`); the two cannot
be combined. If neither is given and the dataset has only one configuration,
that one is used; otherwise the default model (`dinov2`) with default
preprocessing is used.

### umap

Calculate and save 2D UMAP coordinates for stored features. With no model or
configuration specified, UMAP is calculated for all stored configurations;
`--config` can be repeated. Existing coordinates are kept unless `-u/--update`
is given.

```console
iconoscope umap my_collection.h5
iconoscope umap my_collection.h5 -c dinov2/defaults -c clip/defaults --update
```

UMAP is not run with a fixed seed, so results vary between runs; saving them
lets mosaics and notebooks reuse the same layout.

### mosaic

Generate a mosaic image where visually similar images appear near each other:

```console
iconoscope mosaic my_collection.h5
iconoscope mosaic my_collection.h5 -m clip --size 3000x1500 -o clip_mosaic.png
```

If UMAP coordinates for the selected embeddings are generated and saved if
not yet available. The mosaic grid is determined by the output size and number
of images, and is optimized for the most frequent image aspect ratio in the dataset.
Images are slotted into the grid based on UMAP coordinates. Images are scaled and
center-cropped to fill their grid cell.

Options:

- `-o/--output`: output path (default: dataset name with `.jpg`, e.g. `my_collection.jpg`)
- `-s/--size`: output size, a single number for a square or `WIDTHxHEIGHT` (default: 2000)
- `--jpeg-quality`: 1–100, for JPEG output only
- `-l/--limit N`: use at most N images

If the canvas is too small for the number of images, the error message
suggests a `--limit` or `--size` that would work.

#### Example output

Here is an example of an image generated by `iconoscope mosaic`; this mosaic generated from 1000 images from [Princeton Geniza Project](https://geniza.princeton.edu/) fragments, based on feature embeddings from SigLIP with a center crop of 60%.

![mosaic of 1000 Princeton Geniza Project fragment images](images/pgp_1k_mosaic.jpg)

### cluster

Cluster stored features with k-means and save the labels to the dataset:

```console
iconoscope cluster my_collection.h5 8
```

Prints the number of images per cluster. Results for different numbers of
clusters are stored separately.

### info

Report dataset contents: schema, number of images and source directory, most
common image size and aspect ratio, and for each stored configuration the
feature dimensions, UMAP coordinates, and cluster results. Example:

```console
iconoscope info my_collection.h5
Details for my_collection.h5 :
  schema: iconoscope 0.2
  4,302 image paths from /path/to/images
	Most frequent image size:  1024x768
	Most frequent aspect ratio: 1.33
  dinov2/defaults extracted features (4302x768)
	umap coordinates (4302x2)
	4,302 cluster labels (kmeans, k=8)
```

## Notebooks

The [`notebooks/`](notebooks/) directory has [marimo](https://marimo.io)
notebooks for exploring datasets: UMAP → grid animation, nearest neighbor
comparison across models, and an experimental PGP image explorer. See
[notebooks/README.md](notebooks/README.md) for details and usage.

## HDF5 file layout

iconoscope stores image paths, extracted features, and derived data (UMAP
coordinates, cluster labels) in a single HDF5 file (schema version 0.2),
organized per model and preprocessing configuration so multiple
configurations can coexist in the same dataset. All datasets are
gzip-compressed.

```
my_collection.h5                (attrs: schema="iconoscope", schema_version=0.2)
└─ image/                       (attrs: image_dir)
   ├─ paths                     # image file paths relative to image_dir, shape (N,)
   ├─ sizes                     # image width and height in pixels, shape (N, 2)
   └─ models/                   # [optional, added by `embed`]
      └─ dinov2/                # one group per model
         └─ defaults/           # one group per preprocessing configuration (key)
            │                   #   (attrs: model, model_id, and any set options,
            │                   #    e.g. center_crop, grayscale)
            ├─ features         # embedding vectors, shape (N, embed_dim)
            ├─ umap             # 2D UMAP coordinates, shape (N, 2) [optional, added by `umap`/`mosaic`]
            └─ clusters/        # [optional, added by `cluster`]
               └─ kmeans_8      # cluster label per image, shape (N,)
                                #   (attrs: method, n_clusters)
```

The configuration key is `defaults` when no custom preprocessing options are specified and model defaults are used.
Otherwise, the key is based on the combined set of options, e.g. `center_crop=0.6_grayscale`
(`center_crop` is stored as a fraction). Cluster datasets are named
`{method}_{n_clusters}`; `kmeans` is currently the only supported method.

`image_dir` is stored as an absolute path when the dataset is created, and
image paths are stored relative to it; iconoscope combines them when loading,
so a dataset can be used from any working directory. If the image directory
moves, the `image_dir` attribute will need to be updated. All datasets in `image/` and under
each configuration share the same row order: row _i_ is the same image
everywhere. Image order is fixed once any features are stored.

iconoscope validates the file type, `schema` and `schema_version` attributes,
and required `image/` contents when opening a dataset; files from other
schema versions are rejected.

## Acknowledgments

Version 0.1 of this package was a vibe-coded prototype which started as a python port of [Andrej Karpathy's CNN embedding visualizer](https://cs.stanford.edu/people/karpathy/cnnembed/) created with ClaudeCode.

Version 0.2 is a complete rewrite. Development has been supported by AI coding tools, primarily OpenCode and Anthropic models; see commit history for specifics.

## License

Apache 2.0
