Metadata-Version: 2.4
Name: vesuvius
Version: 0.2.2
Project-URL: Homepage, https://github.com/ScrollPrize/villa
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Topic :: Software Development :: Libraries
Classifier: Topic :: Scientific/Engineering
Requires-Python: <3.13,>=3.9
Description-Content-Type: text/markdown
Requires-Dist: numpy>=2.0.2
Requires-Dist: pillow>=11.3.0
Requires-Dist: pynrrd>=1.1.3
Requires-Dist: requests>=2.32.5
Requires-Dist: pyyaml>=6.0.2
Requires-Dist: fsspec>=2025.9.0
Requires-Dist: numcodecs>=0.12.1
Requires-Dist: zarr<3,>=2
Requires-Dist: s3fs>=2025.9.0
Provides-Extra: volume-only
Requires-Dist: numpy>=2.0.2; extra == "volume-only"
Requires-Dist: pillow>=11.3.0; extra == "volume-only"
Requires-Dist: pynrrd>=1.1.3; extra == "volume-only"
Requires-Dist: requests>=2.32.5; extra == "volume-only"
Requires-Dist: pyyaml>=6.0.2; extra == "volume-only"
Requires-Dist: fsspec>=2025.9.0; extra == "volume-only"
Requires-Dist: numcodecs>=0.12.1; extra == "volume-only"
Requires-Dist: zarr<3,>=2; extra == "volume-only"
Requires-Dist: s3fs>=2025.9.0; extra == "volume-only"
Provides-Extra: models
Requires-Dist: torch>=2.6; extra == "models"
Requires-Dist: pytorch-lightning>=2.5.5; extra == "models"
Requires-Dist: pytorch-optimizer>=3.8.0; extra == "models"
Requires-Dist: monai>=1.5.1; extra == "models"
Requires-Dist: huggingface-hub>=0.35.0; extra == "models"
Requires-Dist: timm>=1.0.20; extra == "models"
Requires-Dist: dynamic-network-architectures>=0.3.1; extra == "models"
Requires-Dist: nnunetv2; extra == "models"
Requires-Dist: numba>=0.60.0; extra == "models"
Requires-Dist: batchgenerators>=0.25.1; extra == "models"
Requires-Dist: batchgeneratorsv2; extra == "models"
Requires-Dist: einops>=0.8.1; extra == "models"
Requires-Dist: fft-conv-pytorch>=1.2.0; extra == "models"
Requires-Dist: fvcore>=0.1.5.post20221221; extra == "models"
Requires-Dist: scikit-image>=0.24.0; extra == "models"
Requires-Dist: scipy>=1.13.1; extra == "models"
Requires-Dist: pandas>=2.3.2; extra == "models"
Requires-Dist: tensorstore>=0.1.69; extra == "models"
Requires-Dist: tqdm>=4.67.1; extra == "models"
Requires-Dist: typed-argument-parser>=1.11.0; extra == "models"
Requires-Dist: wandb[media]>=0.22.0; extra == "models"
Requires-Dist: psutil>=7.1.0; extra == "models"
Provides-Extra: render
Requires-Dist: open3d<0.19,>=0.18; extra == "render"
Requires-Dist: trimesh>=4.8.2; extra == "render"
Requires-Dist: opencv-python-headless>=4.11.0.86; extra == "render"
Requires-Dist: opencv-contrib-python>=4.11.0.86; extra == "render"
Requires-Dist: libigl>=2.6.1; extra == "render"
Requires-Dist: alphashape>=1.3.1; extra == "render"
Requires-Dist: connected-components-3d>=3.24.0; extra == "render"
Requires-Dist: simpleitk>=2.5.2; extra == "render"
Requires-Dist: tifffile>=2024.8.30; extra == "render"
Provides-Extra: gui
Requires-Dist: napari>=0.5.6; extra == "gui"
Requires-Dist: magicgui>=0.10.1; extra == "gui"
Requires-Dist: magic-class>=0.7.17; extra == "gui"
Requires-Dist: pyqt6; extra == "gui"
Provides-Extra: notebooks
Requires-Dist: jupyterlab>=4; extra == "notebooks"
Requires-Dist: ipykernel>=6; extra == "notebooks"
Requires-Dist: matplotlib; extra == "notebooks"
Provides-Extra: tests
Requires-Dist: pytest>=8.4.2; extra == "tests"
Provides-Extra: all
Requires-Dist: torch>=2.6; extra == "all"
Requires-Dist: pytorch-lightning>=2.5.5; extra == "all"
Requires-Dist: pytorch-optimizer>=3.8.0; extra == "all"
Requires-Dist: monai>=1.5.1; extra == "all"
Requires-Dist: huggingface-hub>=0.35.0; extra == "all"
Requires-Dist: timm>=1.0.20; extra == "all"
Requires-Dist: numba>=0.60.0; extra == "all"
Requires-Dist: dynamic-network-architectures>=0.3.1; extra == "all"
Requires-Dist: nnunetv2; extra == "all"
Requires-Dist: batchgenerators>=0.25.1; extra == "all"
Requires-Dist: batchgeneratorsv2; extra == "all"
Requires-Dist: einops>=0.8.1; extra == "all"
Requires-Dist: fft-conv-pytorch>=1.2.0; extra == "all"
Requires-Dist: fvcore>=0.1.5.post20221221; extra == "all"
Requires-Dist: scikit-image>=0.24.0; extra == "all"
Requires-Dist: scipy>=1.13.1; extra == "all"
Requires-Dist: pandas>=2.3.2; extra == "all"
Requires-Dist: tensorstore>=0.1.69; extra == "all"
Requires-Dist: tqdm>=4.67.1; extra == "all"
Requires-Dist: typed-argument-parser>=1.11.0; extra == "all"
Requires-Dist: wandb[media]>=0.22.0; extra == "all"
Requires-Dist: psutil>=7.1.0; extra == "all"
Requires-Dist: open3d<0.19,>=0.18; extra == "all"
Requires-Dist: trimesh>=4.8.2; extra == "all"
Requires-Dist: opencv-python-headless>=4.11.0.86; extra == "all"
Requires-Dist: opencv-contrib-python>=4.11.0.86; extra == "all"
Requires-Dist: libigl>=2.6.1; extra == "all"
Requires-Dist: alphashape>=1.3.1; extra == "all"
Requires-Dist: connected-components-3d>=3.24.0; extra == "all"
Requires-Dist: simpleitk>=2.5.2; extra == "all"
Requires-Dist: tifffile>=2024.8.30; extra == "all"
Requires-Dist: napari>=0.5.6; extra == "all"
Requires-Dist: magicgui>=0.10.1; extra == "all"
Requires-Dist: magic-class>=0.7.17; extra == "all"
Requires-Dist: pyqt6; extra == "all"
Requires-Dist: pytest>=8.4.2; extra == "all"

# vesuvius
From [Vesuvius Challenge](https://scrollprize.org), a Python library for : 
- Accessing CT data of Herculaneum Scrolls
- Training 2d or 3d semantic segmentation models, with support for multi-task and multi-class
- Training 2d or 3d models on regression tasks
- Inferring with models trained with the trainers provided in the package or with pretrained nnUNetv2 models
  - Inference can be performed on remote data (http, s3) stored as Zarr arrays
- Rendering .obj segments with local or remote data
- Voxelizing large .obj segmentations for use as 3d labels
- Preprocessing labels of fiber-like structures
- Interactive labeling and model training through a Napari based trainer
- Proofreading large arrays of image/label pairs and saving approved chunks
- Computing structure tensors on large Zarr arrays, and deriving eigenvalues and eigenvectors from them

`vesuvius` also comes prepackaged with: 
- extensive data augmentation
- Dataset orchestrator with streaming adapters for image (tif/png/jpg), zarr, and napari-backed sources

_this package is in active development_


### Entrypoints: 

| Name                          | Script                             | Description                                                                                                                                                                                               |
| ----------------------------- | ---------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `vesuvius.predict`            | `models.run.inference`             | outputs logits from a pretrained nnUNet-v2 model or one trained within the Vesuvius training framework; currently only works on Zarr data and can be fully distributed with `--num_parts` and `--part_id` |
| `vesuvius.blend_logits`       | `models.run.blending`              | blends the logits from `vesuvius.predict` using Gaussian blending                                                                                                                                         |
| `vesuvius.finalize_outputs`   | `models.run.finalize_outputs`      | performs softmax / argmax / none on the blended array and writes a final `uint8` volume                                                                                                                   |
| `vesuvius.inference_pipeline` | `models.run.vesuvius_pipeline`     | runs the three steps above (predict → blend → finalize) in sequence                                                                                                                                       |
| `vesuvius.compute_st`         | `structure_tensor.run_create_st`   | computes structure tensors on input data and derives eigen-values/vectors                                                                                                                                 |
| `vesuvius.napari_trainer`     | `napari_trainer.main_window`       | launches a Napari window for interactive training and inference                                                                                                                                           |
| `vesuvius.proofreader`        | `utils.vc_proofreader.main`        | opens a Napari window that loads local / remote image-label arrays and extracts training patches                                                                                                          |
| `vesuvius.voxelize_obj`       | `scripts.voxelize_objs`            | converts input `.obj` meshes to voxel grids and outputs `.tif` stacks                                                                                                                                     |
| `vesuvius.refine_labels`      | `scripts.edt_frangi_label`         | refines surface or fibre labels with a custom Frangi-based filter                                                                                                                                         |
| `vesuvius.render_obj`         | `rendering.mesh_to_surface`        | renders `.obj` meshes and outputs their surface-volume layers                                                                                                                                             |
| `vesuvius.flatten_obj`        | `rendering.slim_uv`                | flattens an `.obj` mesh using `slim_uv`                                                                                                                                                                   |
| `vesuvius.train`              | `models.run.train`                 | main entry point for training models                                                                                                                                                                      |




## 📓 Introductory notebooks
To get started, we recommend these notebooks that jump right in:

1. 📊 [Scroll Data Access](https://colab.research.google.com/github/ScrollPrize/vesuvius/blob/main/notebooks/example1_data_access.ipynb): an introduction to accessing scroll data using a few lines of Python!

2. ✒️ [Ink Detection](https://colab.research.google.com/github/ScrollPrize/vesuvius/blob/main/notebooks/example2_ink_detection.ipynb): load and visualize segments with ink labels, and train models to detect ink in CT.

3. 🧩 [Volumetric instance segmentation cubes](https://colab.research.google.com/github/ScrollPrize/vesuvius/blob/main/notebooks/example3_cubes_bootstrap.ipynb): how to access instance-annotated cubes with the `Cube` class, used for volumetric segmentation approaches.

## `vesuvius` does:
- **Data retrieval**: Fetches volumetric scroll data, surface volumes of scroll segments, and annotated volumetric instance segmentation labels. Remote repositories and local files are supported.
- **Data listing**: Lists the available data on [our data server](https://dl.ash2txt.org).
- **Data caching**: Caches fetched data to improve performance when accessing remote repositories.
- **Normalization**: Provides options to normalize data values.
- **Multiresolution**: Accesses and manages data at multiple image resolutions.

## `vesuvius` doesn't do:
- **Remote data modification**: The read-only library does not support modifying the original data.
- **Complex analysis**: While it provides access to data, it does not include built-in tools for complex data analysis or visualization.

## Installation

`vesuvius` can be installed with `pip`.
Then, before using the library for the first time, accept the license terms:
```sh
$ pip install vesuvius
$ vesuvius.accept_terms --yes
```

___
**Note:** 

The model framework utilized by `vesuvius` is _heavily_ inspired by [nnUNetv2](https://github.com/MIC-DKFZ/nnUNet). The default configuration will use the same blocks and construct the same encoders/decoders as the default nnUNetv2 ResEncUNet. As such, a significant portion of the code from the nnUNetv2 repository is duplicated here, with modifications to enable multi-task support with dynamic decoder branches, as well as other data formats. 

Additionally, the augmentations provided within this package are from another of [MIC-DKFZ's](https://github.com/MIC-DKFZ) fantastic array of machine learning libraries, in this case [batchgeneratorsv2](https://github.com/MIC-DKFZ/batchgeneratorsv2). 

Copying the modules directly into `vesuvius` was a choice of end-user friendliness, as we were using highly modified branches of both libraries, which created conflicts if an end user were to attempt to run any of our models.

_**Detailed documentation for training and inference are located in [the docs folder](docs/docs)**_
___


### Napari based training

1. Run `vesuvius.napari_trainer`
2. Add an image to the viewer
3. Add a label layer, with the suffix as the name of your target (ex : `32_ink`)
4. Optionally, add a label layer with the suffix `_mask` (ex : `32_mask`)
5. Set your training configuration and hit `run training`
6. Infer on the same layer, or another by importing an image, selecting it in the inference widget, and hitting `Run Inference`


![alt text](docs/docs/images/napari_trainer.png)

___

### Proofreading labels

1. Update the config in `/utils/vc_proofreader/config.py` with the proper paths
2. Run `vesuvius.proofreader`
3. Select your desired patch size and min labeled percentage 
4. Click `run` 
5. Approve patches with `a` or by checking the box
6. Skip patches or continue with `spacebar` or `next pair`
7. Patches are saved in the output dir, and their locations in the .json progress file

![alt text](docs/docs/images/proofreader.png)

### Training with `vesuvius.train`

Supported model types:
- Single task, single class semantic segmentation / regression
- Single task, multi-class semantic segmentation / regression
- Multi-task, single-class semantic segmentation / regression
- Multi-task, multi-class semantic segmentation / regression

By default, when provided with a single channel (binary) input label, the model will output 2 channels (fg/bg), but can adapt to any number of input channels.

**Place your data in the following format:** 
```
data/
  images/
    volume1.zarr (or .tif, .png, .jpg)
    volume2.zarr
  labels/
    volume1_ink.zarr (or tif, .png, .jpg) 
    volume1_hz_fiber.zarr ( if you want an additional task for the same volume )
    volume2_ink.zarr
   ```
**Begin training with:**
```bash
 vesuvius.train -i /path/to/data --batch-size 4 --patch-size 128,128,128 --model-name /path/to/save/checkpoints --config-path /vesuvius/models/configuration/multi-task_config.yaml
```
There are a number of other optional parameters, which you can find with `--help`

When this command is run: 
- a ConfigManager class will be instantiated, which will take the arguments given and the configuration file, and store these as properties.
- The trainer class will execute its class method `__build_model`, which will create an instance of `NetworkFromConfig`
- `NetworkFromConfig` will dynamically determine the number of pooling operations, stages, feature map sizes, operation dimensionality, and other specified parameters
- The trainer class will execute its `_configure_dataset` method, instantiating the `DatasetOrchestrator` with the adapter that matches `data_format`
- The trainer class will execute the rest of the setup required for training, through the following additional class methods:
  - `_build_loss`
  - `_get_optimizer`
  - `_get_scheduler`
  - `_get_scaler`
  - `_configure_dataloaders`

- The training loop will begin

Training will output the current losses in the `tqdm` progress bar, and will save a "debug" png or gif (depending on dimensionality) in the checkpoint directory. 

By default, the last 10 checkpoints are saved. This is not a smart way to do it, and will be changed to just store the last 3 + last 2 best validation. 

Training will run for 1,000 epochs by default, with 200 batches/epoch. This can be modified through the configuration file, which can be optionally provided to `vesuvius.train`, and some examples are provided in the [models folder](src/vesuvius/models/configuration/)

### Running inference with `vesuvius.inference_pipeline`

Detailed documentation is available in [the inference readme](docs/docs/inference.md)

The inference process is a bit convoluted. Due to our focus on very large volumetric data, it becomes very difficult for writes to keep pace with multi-gpu systems, necessitating a map-reduce style of inference where we store intermediate logits before blending.  It **stores a very large amount of intermediate data** , in the form of float16 logits from patches. In the case that you do not have a lot of storage space, it might make sense to borrow some of the functions from the inferer, and rewrite the inference process to write directly to uint8 final arrays.

The inference process is in 3 parts -- inference, blending, and finalization

1. `inference.py` creates a vc_dataset, which reads data from the Volume class in volume.py
 
    - patches are ran through forward passes in batches, and either rotation or mirroring TTA is applied
    - patches are stored in intermediate 'logits' arrays, named by their gpu rank, in a zarr array of shape `patch_z, patch_y, patch_x * num_patches`. It's just a zarr array that is very long and skinny. the reason for this is that writing to overlapping chunks in zarr arrays (which we have because we take 50% of the patch size as overlap during inference) is difficult when you have to keep pace with your gpus to avoid unbounded memory growth. We went with a map-reduce style here to avoid this entirely.
2. the logits arrays are read by `blending.py` chunkwise, along with their 8 neighbors (which contribute to their final values), and accumulated into a "weights array"
   - this accumulation array is normalized chunkwise by the number of patches which contributed to its values
3. the blended array is read by `finalize_outputs.py` , which applies the activation functions (softmax, argmax, or none), casts to uint8, and writes the final zarr

This pipeline can be performed individually, through `vesuvius.predict` , `vesuvius.blend_logits` and `vesuvius.finalize_outputs`, respectively. It also can be performed sequentially with `vesuvius.inference_pipeline` 

On a single machine, running the full pipeline is recommended. You can use local volumes or remote (s3, http) volumes. To begin inference, run
```
vesuvius.inference_pipeline --input /path/to/input.zarr --output /path/to/output.zarr --model hf://scrollprize/surface_recto --batch-size 4
```
You can use local models or models hosted on huggingface hub, so long as they are either nnUNetv2 models or models created with the vesuvius trainer. We have some of our fiber and surface models hosted [here](https://huggingface.co/collections/scrollprize/representation-67e1b44299d5c18f5845874f), but you could use your own as well. 

To use a local model point at either the `nnUNet_results` folder which contains the `dataset.json` and the `fold` directories, or if using a vesuvius model, the `checkpoint.pth` file

### Rendering and Flattening objs
Documentation is provided in [the rendering folder](src/vesuvius/rendering/README.md)
