Metadata-Version: 2.4
Name: em-database
Version: 0.4.0
Summary: A database for electron microscopy data using pooch to download and manage files.
Author-email: Carter Francis <cartsfrancis@gmail.com>
License-Expression: MIT
Project-URL: Homepage, https://cssfrancis.github.io/em_data/
Project-URL: Documentation, https://cssfrancis.github.io/em_data/datasets.html
Project-URL: Repository, https://github.com/CSSFrancis/em_data
Project-URL: Issues, https://github.com/CSSFrancis/em_data/issues
Keywords: electron microscopy,database,materials science
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Physics
Classifier: Topic :: Scientific/Engineering :: Image Processing
Requires-Python: >=3.12
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pooch
Requires-Dist: pyyaml
Requires-Dist: tqdm>=4.67.1
Provides-Extra: widget
Requires-Dist: anywidget>=0.9; extra == "widget"
Provides-Extra: tests
Requires-Dist: pytest; extra == "tests"
Requires-Dist: pytest-cov; extra == "tests"
Requires-Dist: hyperspy; extra == "tests"
Requires-Dist: quantem; extra == "tests"
Requires-Dist: anywidget>=0.9; extra == "tests"
Provides-Extra: doc
Requires-Dist: sphinx; extra == "doc"
Requires-Dist: sphinx-rtd-theme; extra == "doc"
Requires-Dist: sphinx-gallery; extra == "doc"
Requires-Dist: quantem; extra == "doc"
Requires-Dist: hyperspy; extra == "doc"
Requires-Dist: pydata_sphinx_theme; extra == "doc"
Requires-Dist: sphinx_design; extra == "doc"
Dynamic: license-file

EM Data
-------

This is a simple project for aggregating different Electron Microscopy files which are hosted over different sources.  It uses pooch to download datasets and should be
used as a way to host simple example datasets for method validation.

Data is stored in a file "User/em_database" but this can be also set to a custom location.

List of datasets https://cssfrancis.github.io/em_data/datasets.html

## Installation

```bash
pip install em-database
```

## Usage

Every dataset is a class under `em_database.data`.  Calling `download()` fetches the
file to the data directory, verifies its checksum, and returns a path handle.  Files
that are already present are not downloaded again.

```python
import em_database.data as data
import hyperspy.api as hs

path = data.LayeredCuNb4DSTEM().download()
s = hs.load(path, lazy=True)
```

By default the download runs on a background thread so a notebook cell returns
immediately.  The handle it returns *is* the file path, so you can hand it straight
to a loader as above — it only blocks at the moment the file is actually opened.
Call `path.done()` to check progress without blocking, or `path.result()` to wait
explicitly.  Pass `download(background=False)` to block and get the path as a plain
string instead.

## Settings

Configuration lives in a live, matplotlib-`rcParams`-style object,
`em_database.settings`, seeded at import from `~/.em_database/settings.yaml`.
Change it in memory for an immediate effect, and persist it to remember the
choice across sessions:

```python
import em_database

em_database.settings["data_dir"] = "/big/disk/em_data"  # takes effect now
em_database.settings.save()                             # remember it next time
```

In Jupyter you can also edit them interactively — `display(em_database.settings)`
renders a panel to set (and save) the data directory directly.

The data directory defaults to `~/em_database`. Convenience helpers wrap the
common case — `set_data_dir` persists by default:

```python
em_database.get_data_dir()                       # current location
em_database.set_data_dir("/big/disk/em_data")    # set + persist
em_database.set_data_dir("/scratch", persist=False)  # one-off, in-memory only
em_database.reset_data_dir()                     # back to the default, forget the choice
```

No environment variable is needed. Only values you explicitly set are written to
the file, so the default is never frozen in — it keeps being obeyed even if it
changes. For a one-off override (e.g. CI) the legacy `EM_DATABASE_DATA_DIR` is
still honored, and `EM_DATABASE_CONFIG` relocates the settings file.

### Shared, system-wide data

Datasets can be installed once for every user. `download()` and `filepath()`
look in the shared/system locations **first**, then your own data directory, and
only download (into your directory) if the file is nowhere to be found — so a
shared copy is reused instead of refetched. Shared locations come from the
`EM_DATABASE_SHARED_DIR` environment variable (an `os.pathsep`-separated list), a
`shared_data_dirs` list in your settings, or a system config file
(`/etc/em_database/settings.yaml`, or `%PROGRAMDATA%\em_database\settings.yaml`
on Windows).

## Adding a dataset

Datasets are described by a YAML file in `em_database/datasets/`, one entry per file,
validated against `em_database/datasets/json-schema.json`.  The class name is generated
from the top-level key:

```yaml
MyDataset:
  description: What the data is, how it was acquired and how it is calibrated.
  source: https://zenodo.org/records/<record>/files
  checksum: md5:<hash>
  file: MyDataset.zspy
  data_size: 1.2 GB
  technique: 4D-STEM
  license: CC-BY-4.0
```

Open an issue with the new dataset template, or add the YAML file directly.
