Metadata-Version: 2.5
Name: pyautoprocess
Version: 0.5.2
Summary: Automated processing and analysis of MicroED data
Project-URL: Homepage, https://github.com/theNelsonLab/autoprocess
Project-URL: Issues, https://github.com/theNelsonLab/autoprocess/issues
Author-email: Dmitry Eremin <eremin@caltech.edu>, Jessica Burch <hosea@caltech.edu>, Sam Foxman <sfoxman@caltech.edu>
License-Expression: GPL-3.0-or-later
License-File: LICENSE
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Requires-Python: >=3.10
Requires-Dist: mrcfile~=1.5
Requires-Dist: numpy~=2.0
Requires-Dist: scikit-image>=0.25.0
Requires-Dist: scipy>=1.7.0
Requires-Dist: seremi==0.0.1
Requires-Dist: tifffile>=2025.1.10
Provides-Extra: dev
Requires-Dist: pytest==8.4.2; extra == 'dev'
Description-Content-Type: text/markdown

# pyautoprocess

A Python package for automated processing and analysis of MicroED (Micro-Electron Diffraction) data.

## Overview
This package provides a comprehensive suite of tools for automated MicroED data processing:
- **autoprocess**: Core script for automated MicroED data processing
- **image_process**: Preconverted image processing and quality analysis tool
- **monitorED**: Active file monitor that watches for incoming data and triggers processing automatically
- **batch_reprocess**: Batch reprocessing tool with specific space group and unit cell parameters
- **mrc2tif**: Utility for converting MRC files to TIF format
- **ser2tif**: Utility for converting SER files to TIF format
- **tvips2tif**: Utility for converting TVIPS files to TIF format

## Installation

Install from PyPI:
```bash
pip install pyautoprocess
```

Or install from source:
```bash
git clone https://github.com/theNelsonLab/autoprocess.git
cd autoprocess
pip install .

## Requirements

### Python Dependencies
- Python >= 3.10
- numpy ~= 2.0
- mrcfile ~= 1.5
- tifffile >= 2025.1.10
- seremi == 0.0.1
- scipy >= 1.7.0
- scikit-image >= 0.25.0

### External Software Dependencies
1. **XDS Software Suite**: Required for crystallographic data processing
   - Must be accessible via the `xds` command
   - Available from [xds.mr.mpg.de](https://xds.mr.mpg.de/)

2. **Pointless**: Required for space group analysis (pointless)
   - Part of the CCP4 software suite

## Script Details

### 1. autoprocess

#### Description
Primary tool for automated MicroED data processing using the XDS suite. Handles image conversion, indexing, integration, space group analysis, and scaling with intelligent parameter optimization.

#### Key Features
- **File Processing**: Processes .mrc/.ser/.tvips files from specified paths or current directory
- **Automatic Image Conversion**: Converts source files to XDS-compatible TIF format with validation
- **Diffraction Quality Analysis**: Optional frame quality assessment and intelligent frame selection (--dqa)
- **Dynamic XDS Optimization**: Automatic parameter adjustment for indexing, integration, and scaling
- **Advanced Space Group Analysis**: Intelligent space group optimization using lattice analysis
- **Pointless Integration**: Optional CCP4 pointless analysis for space group validation (--pointless)
- **Parallel Processing**: Support for parallel XDS execution (--parallel)
- **Comprehensive Logging**: Detailed processing logs and progress tracking
- **Microscope Configurations**: Built-in support for multiple instrument configurations
- **Reprocessing Control**: Skip already processed files or force reprocessing (--reprocess)

#### Usage
```bash
autoprocess [paths] [options]

Positional Arguments:
  paths                    Path(s) to process: single .mrc/.ser/.tvips file, folder containing files,
                          or multiple files/folders. If not specified, processes all files
                          in current directory.

Microscope Configuration:
  --microscope-config CONFIG  Choose instrument configuration (default: default)
  --config-file FILE          JSON of extra microscope configurations, added to the
                              built-in set (a built-in name is replaced entirely).
                              See "Custom microscope configurations"
  --list-configs              Print the available configuration names, one per line
                              (including any from --config-file), and exit

Processing Control:
  --reprocess                 Reprocess files even if they have been processed before
  --retry-failed              Try again datasets that failed in an earlier run (without it
                              they are skipped and the exit status is 1)
  --pointless                 Run pointless for space group analysis
  --parallel                  Use parallel XDS (xds_par) instead of serial XDS
  --dqa                       Enable diffraction quality analysis and frame selection
  --verbose                   Enable verbose logging for detailed conversion validation

XDS Parameters Override:
  --rotation-axis AXIS        Override rotation axis
  --frame-size SIZE          Override frame size
  --signal-pixel VALUE       Override signal pixel value (XDS parameter)
  --min-pixel VALUE         Override minimum pixel value (XDS parameter)
  --background-pixel VALUE   Override background pixel value (XDS parameter)
  --pixel-size VALUE        Override pixel size value
  --wavelength VALUE        Override wavelength value
  --beam-center-x VALUE     Override beam center X coordinate
  --beam-center-y VALUE     Override beam center Y coordinate
  --file-extension EXT      Override input file extension

Experimental Parameters Override:
  --detector-distance VALUE  Override detector distance (in mm)
  --exposure VALUE           Override exposure time
  --rotation VALUE          Override rotation value

Resolution Control:
  --res-range VALUE         Manual resolution range in Angstroms (overrides calculated values)
  --min-res VALUE          Minimum resolution for XSCALE in Angstroms (independent from XDS)

Experimental Features (opt-in, off by default):
  --auto-rotation-axis     Derive the rotation-axis sign from the tilt-direction token in the
                           filename (e.g. P50toN-50, n60top10). A positive-to-negative sweep
                           keeps the configured axis; a negative-to-positive sweep negates every
                           component of it. Falls back to the configured axis, with the reason
                           logged, whenever the direction cannot be established.
  --seed N                 Seed the indexing-retry search so a failed first-pass indexing
                           reproduces exactly. Without it those retries use random parameters,
                           so a dataset that fails first-pass indexing can return a different
                           space group and unit cell on every run.
  --beam-center            Detect the beam centre from the diffraction data and use it for
                           ORGX/ORGY instead of the microscope-config value. Probes the first,
                           middle and last frame of the processed range and combines them;
                           falls back to the configured centre whenever the result is not
                           trustworthy. Writes auto_process/beam_center.LP recording how the
                           value was chosen.

Advanced XDS Parameters:
  --friedel BOOL           Set Friedel's law for XDS (default: true). Accepts true/false,
                           yes/no, 1/0, on/off in any case; anything else exits with status 2
```


### 2. image_process

#### Description
Processing tool for pre-converted crystallography images with backup management, quality analysis, and flexible format support.

#### Key Features
- **Pre-Converted Processing**: Process existing TIF/IMG image files without conversion
- **Format Flexibility**: Support for both TIF and SMV (.img) formats (--smv)
- **Quality Analysis Integration**: Optional diffraction quality analysis with frame selection (--dqa)
- **Frame Trimming**: Precise frame range control with --trim-front and --trim-end
- **Backup Management**: Automatic backup system with organized archive folders
- **Multi-Path Processing**: Process multiple directories or specific paths simultaneously
- **Microscope Configurations**: Support for all standard microscope configurations
- **Processing Isolation**: Separate output folders (auto_process_direct) to avoid conflicts

#### Usage
```bash
image_process [paths] [options]

Positional Arguments:
  paths                    Path(s) to process: folders containing pre-converted images.
                          If not specified, processes all suitable folders in current directory.

File Format Options:
  --smv                    Process SMV (.img) files instead of TIF files

Frame Selection:
  --trim-front N           Number of frames to trim from the start of the range (default: 0)
  --trim-end N             Number of frames to trim from the end of the range (default: 0)
  --dqa                    Enable diffraction quality analysis and frame selection

Microscope Configuration:
  --microscope-config CONFIG  Choose instrument configuration (default: default)
  --config-file FILE          JSON of extra microscope configurations, added to the
                              built-in set (a built-in name is replaced entirely).
                              See "Custom microscope configurations"
  --list-configs              Print the available configuration names, one per line
                              (including any from --config-file), and exit

Processing Control:
  --pointless                 Run pointless for space group analysis
  --parallel                  Use parallel XDS (xds_par) instead of serial XDS
  --verbose                   Enable verbose logging

XDS Parameters Override:
  --rotation-axis AXIS        Override rotation axis
  --frame-size SIZE          Override frame size
  --signal-pixel VALUE       Override signal pixel value
  --min-pixel VALUE         Override minimum pixel value
  --background-pixel VALUE   Override background pixel value
  --pixel-size VALUE        Override pixel size value
  --wavelength VALUE        Override wavelength value
  --beam-center-x VALUE     Override beam center X coordinate
  --beam-center-y VALUE     Override beam center Y coordinate

Experimental Parameters Override:
  --detector-distance VALUE  Override detector distance (in mm)
  --exposure VALUE           Override exposure time
  --rotation VALUE          Override rotation value

Resolution Control:
  --res-range VALUE         Manual resolution range in Angstroms (overrides calculated values)
  --min-res VALUE          Minimum resolution for XSCALE in Angstroms (independent from XDS)

Experimental Features (opt-in, off by default):
  --auto-rotation-axis     Derive the rotation-axis sign from the tilt-direction token in the
                           filename (e.g. P50toN-50, n60top10). A positive-to-negative sweep
                           keeps the configured axis; a negative-to-positive sweep negates every
                           component of it. Falls back to the configured axis, with the reason
                           logged, whenever the direction cannot be established.
  --seed N                 Seed the indexing-retry search so a failed first-pass indexing
                           reproduces exactly. Without it those retries use random parameters,
                           so a dataset that fails first-pass indexing can return a different
                           space group and unit cell on every run.
  --beam-center            Detect the beam centre from the diffraction data and use it for
                           ORGX/ORGY instead of the microscope-config value. Probes the first,
                           middle and last frame of the processed range and combines them;
                           falls back to the configured centre whenever the result is not
                           trustworthy. Writes auto_process/beam_center.LP recording how the
                           value was chosen.

Advanced XDS Parameters:
  --friedel BOOL           Set Friedel's law for XDS (default: true). Accepts true/false,
                           yes/no, 1/0, on/off in any case; anything else exits with status 2

Examples:
  # Process TIF images in current directory
  image_process

  # Process SMV files with frame trimming
  image_process --smv --trim-front 5 --trim-end 10 /path/to/images

  # Quality analysis with multiple paths
  image_process --dqa /data/sample1 /data/sample2 /data/sample3

  # Advanced processing with parameter overrides
  image_process --parallel --pointless --signal-pixel 6 --trim-front 3 /path/to/data
```

### 3. monitorED

#### Description
Active file monitor for MicroED data collection sessions. Watches for incoming data files or folders and automatically triggers `autoprocess` or `image_process` when new data arrives. Designed to run during a data collection session, processing datasets as they are acquired.

#### Key Features
- **Two Processing Modes**: Monitor for raw movie files (`--autoprocess`) or pre-converted image folders (`--image-process`)
- **Filename Validation**: Only processes files matching the expected naming schema (`sample_distance_rotation_exposure[_extra].ext`)
- **File Stability Detection**: Waits for files to finish writing before processing (size/mtime stability check)
- **Subdirectory Monitoring**: Optional one-level subdirectory watching (`--watch-subdirs`)
- **Inactivity Timeout**: Automatically terminates after configurable idle period (default: 2 hours)
- **Expected Count**: Optionally stop after processing a specific number of files (`--expect-count`)
- **Persistent Tracking Log**: Records processed files in `monitored_tracking.log` to avoid reprocessing across restarts
- **Flag Passthrough**: All `autoprocess` / `image_process` flags are forwarded directly to the child command

#### Usage
```bash
monitorED (--autoprocess | --image-process) [monitor options] [processing options...]

Monitor Options:
  --autoprocess             Monitor for raw movie files (.mrc/.ser/.tvips) and run autoprocess
  --image-process           Monitor for folders with images/ subdirectory and run image_process
  --watch-subdirs           Also monitor immediate subdirectories (1 level deep)
  --timeout SECONDS         Inactivity timeout in seconds (default: 7200 = 2 hours)
  --expect-count N          Stop after processing this many files/folders

Processing Options:
  All remaining flags are forwarded to the selected command (autoprocess or image_process).
  See their respective --help for details.

Examples:
  # Monitor current directory for new .mrc/.ser files, run autoprocess
  monitorED --autoprocess --microscope-config default

  # Monitor with subdirectories, expect 10 datasets, image_process mode
  monitorED --image-process --watch-subdirs --expect-count 10 --microscope-config default

  # Custom timeout (1 hour) with passthrough flags
  monitorED --autoprocess --timeout 3600 --parallel --dqa --microscope-config Arctica-CETA-ser-SM
```

### 4. batch_reprocess

#### Description
Advanced tool for batch reprocessing of crystallography data with specific space group and unit cell parameters. Supports both manual parameter specification and smart automatic detection mode.

#### Key Features
- **Smart Detection Mode**: Automatically detect optimal parameters from existing processing results (--smart)
- **Manual Parameter Mode**: Specify exact space group and unit cell parameters (--manual, default)
- **Interactive Input**: Prompts for missing parameters when not provided via command line
- **Flexible Reindexing**: Support for custom reindexing matrices
- **Batch Processing**: Process multiple datasets with consistent or adaptive parameters
- **Archive Management**: Organized backup and folder management for reprocessed data

#### Usage
```bash
batch_reprocess [paths] [options]

Positional Arguments:
  paths                    Path(s) to reprocess: folders containing processed data

Processing Mode:
  --manual                 Manual parameter specification mode (default)
  --smart                  Smart detection mode - automatically detect parameters

Crystallographic Parameters (Manual Mode):
  --space-gr NUMBER        Space group number
  --a VALUE               Unit cell parameter a (Å)
  --b VALUE               Unit cell parameter b (Å)
  --c VALUE               Unit cell parameter c (Å)
  --alpha VALUE           Unit cell angle alpha (degrees)
  --beta VALUE            Unit cell angle beta (degrees)
  --gamma VALUE           Unit cell angle gamma (degrees)
  --folder NAME           Subfolder name for reprocessed data
  --reidx "MATRIX"        Reindexing matrix as space-separated string
                          (e.g., "1 0 0 0 0 1 0 0 0 0 1 0")

Examples:
  # Smart mode - auto-detect parameters
  batch_reprocess --smart /path/to/data/folders

  # Manual mode with specific parameters
  batch_reprocess --space-gr 19 --a 50.1 --b 60.2 --c 70.3 \
                  --alpha 90 --beta 90 --gamma 90 --folder reprocess_P212121

  # Interactive mode (prompts for missing parameters)
  batch_reprocess /path/to/data
```

### 5. mrc2tif

#### Description
Precision utility for converting MRC movie files to TIF format with comprehensive verification, statistics, and data integrity checking.

#### Key Features
- **Multi-Format Support**: Single and multi-frame MRC file processing
- **Data Type Preservation**: Intelligent data type handling and conversion
- **Pedestal Addition**: Optional pedestal value addition with range validation
- **Raw Conversion Mode**: Preserve original data without type conversion (--raw)
- **Comprehensive Verification**: Frame-by-frame conversion validation with statistical analysis
- **Detailed Logging**: Split logging (file and console) with conversion statistics
- **Recursive Processing**: Process files in subdirectories (--recursive)
- **Custom Naming**: Flexible output file naming with --tif-name option
- **Data Range Validation**: Automatic checking for uint16 overflow/underflow
- **Directory Management**: Automatic 'images' subdirectory creation

#### Usage
```bash
mrc2tif [options]

Options:
  --folder PATH           Path to folder containing MRC files (default: current directory)
  --ped VALUE            Pedestal value to add to each pixel (default: 0)
  --tif-name NAME        Base name for output TIF files (default: same as MRC filename)
  --recursive            Search for MRC files recursively in subdirectories
  --raw                  Convert data without any type conversion or modifications

Examples:
  # Convert all MRC files in current directory
  mrc2tif

  # Convert with pedestal addition
  mrc2tif --ped 100 --folder /path/to/mrc/files

  # Recursive conversion with custom naming
  mrc2tif --recursive --tif-name sample_data --folder /data/root

  # Raw conversion preserving original data types
  mrc2tif --raw --folder /path/to/data
```

### 6. ser2tif

#### Description
Specialized utility for converting SER (Serial Electron Microscopy) movie files to TIF format using the seremi library for precise SER file handling and frame extraction.

#### Key Features
- **SER Format Expertise**: Native SER file format support using seremi library
- **TIA Compatibility**: Full compatibility with TIA and other SER-generating software
- **Frame Extraction**: Automatic individual frame extraction and processing
- **Data Integrity**: Comprehensive conversion verification and validation
- **Pedestal Support**: Optional pedestal value addition with validation
- **Raw Conversion**: Raw data conversion mode preserving original formats (--raw)
- **Detailed Logging**: Split logging system with comprehensive statistics
- **Recursive Processing**: Search subdirectories for SER files (--recursive)
- **Custom Naming**: Flexible output file naming control
- **Directory Management**: Automatic 'images' subdirectory organization

#### Usage
```bash
ser2tif [options]

Options:
  --folder PATH           Path to folder containing SER files (default: current directory)
  --ped VALUE            Pedestal value to add to each pixel (default: 0)
  --tif-name NAME        Base name for output TIF files (default: same as SER filename)
  --recursive            Search for SER files recursively in subdirectories
  --raw                  Convert data without any type conversion or modifications

Examples:
  # Convert all SER files in current directory
  ser2tif

  # Convert with pedestal and custom naming
  ser2tif --ped 50 --tif-name converted_frames --folder /data/ser_files

  # Recursive search with raw conversion
  ser2tif --recursive --raw --folder /microscopy/data
```

### 7. tvips2tif

#### Description
Utility for converting TVIPS movie files to TIF format with comprehensive verification and data integrity checking.

#### Key Features
- **TVIPS Format Support**: Native TVIPS file format support
- **Frame Extraction**: Automatic individual frame extraction and processing
- **Data Integrity**: Comprehensive conversion verification and validation
- **Pedestal Support**: Optional pedestal value addition with validation
- **Raw Conversion**: Raw data conversion mode preserving original formats (--raw)
- **Detailed Logging**: Split logging system with comprehensive statistics
- **Recursive Processing**: Search subdirectories for TVIPS files (--recursive)
- **Custom Naming**: Flexible output file naming control
- **Directory Management**: Automatic 'images' subdirectory organization

#### Usage
```bash
tvips2tif [options]

Options:
  --folder PATH           Path to folder containing TVIPS files (default: current directory)
  --ped VALUE            Pedestal value to add to each pixel (default: 0)
  --tif-name NAME        Base name for output TIF files (default: same as TVIPS filename)
  --recursive            Search for TVIPS files recursively in subdirectories
  --raw                  Convert data without any type conversion or modifications

Examples:
  # Convert all TVIPS files in current directory
  tvips2tif

  # Convert with pedestal and custom naming
  tvips2tif --ped 100 --tif-name converted_frames --folder /data/tvips_files

  # Recursive search with raw conversion
  tvips2tif --recursive --raw --folder /microscopy/data
```

## Experimental Features

Both features below are **opt-in and off by default**, and both fall back to the microscope
configuration whenever they cannot produce a trustworthy answer. Neither can ever leave you with
a guessed value: the fallback is always the configured one, and the reason is always logged.

### --auto-rotation-axis

The XDS rotation axis depends on which way the goniometer swept, and the sweep is recorded in the
filename as a whole underscore field: `P50toN-50`, `n60top10`, `p70to0`, `N40toP40`. A
positive-to-negative sweep uses the configured axis unchanged; a negative-to-positive sweep uses
its negation -- every component, not just x (`-0.8290 -0.5592 0` becomes `0.8290 0.5592 0`).

The direction is decided by arithmetic on the signed angles rather than by reading the letters,
so `0toP25`, `P25to0` and `P50toN0` all resolve correctly. The letter carries the sign, making
the minus in `N-50` redundant.

**Scope, honestly:** across 199,606 movie filenames in the lab's archive, 880 carry a direction
token, and only 21 of those are negative-to-positive. So this flag changes nothing for the vast
majority of datasets -- it logs "no tilt-direction token, keeping configured axis" and moves on.
The flip path is supported by relatively few crystals in one detector family, and no
negative-to-positive `.mrc` dataset is known to exist, so that combination is untested. The log
says so loudly whenever a flip actually happens; check the indexing result when it does.

### --beam-center

Estimates the direct-beam position from the frames themselves rather than trusting the
per-microscope constant. The estimator registers the pattern against its own 180-degree rotation
(Friedel centrosymmetry), so it does not assume the beam is the brightest feature -- which
matters when a beam stop makes the centre dark.

The first, middle and last frame of the processed range are probed, because either end of a movie
can be blank. Results below a confidence threshold are discarded, survivors are combined, and
frames that disagree with each other are rejected outright. If all three fail, the 25% and 75%
positions are tried before giving up.

**When it actually helps (measured, 12 datasets):**

Use it when indexing is poor or the configured beam centre is suspect. Do **not** expect it to
improve a run that already works.

The estimator is accurate: where it fires, it lands 0.8-5.8 px from the beam position XDS itself
refines to, against 11.6-83.1 px for the configured default -- a median error of roughly 2 px
versus 20 px. But that accuracy usually does not change the result, because **XDS refines the
beam centre itself and converges to the same answer regardless of where it starts.** Across the
test set, the refined position reached from the configured centre and from the detected centre
agreed to 0.0-0.6 px. The starting value is effectively discarded.

So the honest summary is: quality-neutral on most datasets (5 flat, 1 better, 1 worse across the
set), with two cases where it mattered:

- `ss-jacobsen-mov16`, configured centre 27.5 px off: Rmeas 24.2% -> 19.0%, I/sigma 3.23 -> 4.23.
- `Grossular-mov2`, configured centre 83 px off: the only dataset where XDS's refinement DIVERGED
  from the configured start (to a position ~80 px away), while the detected start kept it sane.
  Rmeas 133% -> 107% -- still unusable, but it is the difference between lost and merely bad.

Notes and limits:
- The result is anchored to the configured centre and bounded to about +/-100 px around it, so
  this **refines a roughly-right configuration; it cannot rescue a badly wrong one.** A grossly
  wrong config value shows up as "detection failed", not as a correction.
- Detection sometimes declines on exactly the datasets that would benefit: two datasets whose
  configured centre was 20-30 px off fell back rather than detecting. The confidence gate is
  deliberately conservative, and falling back is always safe, but it means the feature does not
  yet reliably catch the case it is best suited to.
- Explicit `--beam-center-x` / `--beam-center-y` take precedence, per axis. Supplying only one
  lets detection supply the other; supplying both skips detection entirely.
- Adds roughly 3 seconds per dataset.
- Not supported for SMV (`.img`) input in `image_process`.
- Every run writes `auto_process/beam_center.LP` with the method, confidence, frame count,
  cross-frame spread, unrounded value and shift from the configured centre -- including when it
  falls back, since knowing the attempt was made and declined is the useful record later.

### --seed

Not experimental, and unrelated to the two features above -- but worth understanding.

When the first indexing attempt fails, autoprocess retries up to ten times with **randomly
chosen** `BACKGROUND_PIXEL`, `SIGNAL_PIXEL` and `MINIMUM_NUMBER_OF_PIXELS_IN_A_SPOT` values.
That is a reasonable search strategy, but it means such a dataset **is not reproducible**:
reprocessing the same movie can return a different answer.

This is not hypothetical. Four runs of one real dataset under a single unchanged version gave:

| run | ISa | Rmeas | completeness | space group |
|-----|-----|-------|--------------|-------------|
| 1-4 | 1.06 - 3.39 | 72.9% - 257.2% | 42.5% - 90.0% | 1 or 5, different unit cell each time |

Another dataset indexed successfully in only 1 of 4 attempts.

`--seed N` pins that search without changing what it explores, so a run can be reproduced
exactly. The seed is applied **per dataset**, so a movie's result does not depend on how many
other movies preceded it in the same invocation. Datasets that index on the first attempt are
unaffected either way -- they never enter the retry loop.

Recommended whenever a result needs to be reproducible: a published structure, a regression
comparison, or any dataset whose log shows "Screening new indexing values".

A seed makes **failure** reproducible too. If a seeded run fails to index, re-running with the
same seed repeats that failure exactly -- try a different seed, or drop the flag, to let the
retry search explore other parameters.

## File Naming Convention
### For .mrc/.ser/.tvips Files
```
sample-name_distance_rotation_exposure_additional-notes.ext
```
Examples:
- `sample-mov1_960_0.3_3_n60top10_g8sp10_cryo.ser`
- `Lysozyme-NAG2-DC-xtal-05_960_1p5_0p6_p40ton60_g8sp7_bin4_0_movie.mrc`

Fields:
- `sample-name`: Sample identifier (may contain hyphens)
- `distance`: Detector distance in mm
- `rotation`: Rotation speed in degrees/second
- `exposure`: Exposure time in seconds
- `additional-notes`: Optional extra metadata (ignored by parser)

**Note**: Decimal points in numeric fields can use either `.` or `p` as separator (e.g., `1p5` is treated as `1.5`).

**The three numeric fields must actually be numeric.** Four underscore-separated fields are not
enough on their own: `20260513_98917_0_movie.mrc` has four, but its "exposure" is the word
`movie`, so it is skipped with an explanation rather than processed with nonsense. The same
applies when the sample name itself contains an underscore, since that shifts every field along.
`autoprocess` and `monitorED` apply the same test, so they agree on which files are valid.

Use `--id SAMPLE_ID` to process a file whose name does not follow the convention: the sample name
comes from the flag, and the three numeric values from `--detector-distance` / `--rotation` /
`--exposure` or the microscope config.

## Directory Structure
```
working_directory/
├── sample_name/
│   ├── images/
│   │   └── (converted image files)
│   ├── auto_process/
│   │   └── (XDS processing files)
│   └── batch_reprocess/
│       └── (reprocessed data files)
├── autoprocess_logs/
│   └── (processing log files)
└── logs/
    └── (conversion log files)
```

## Exit Codes

`autoprocess` and `image_process` report the outcome through their exit status. **These codes are
a stable contract**: callers such as REyes and `monitorED` depend on them, and they will not be
renumbered.

| code | meaning |
|------|---------|
| 0 | every dataset attempted succeeded, or was already processed successfully |
| 1 | at least one dataset failed, in this run or (for `autoprocess`) in an earlier run and was not retried |
| 2 | the command itself was wrong: an unknown `--microscope-config`, an invalid `--config-file`, a bad `--friedel` value, `--id` with more than one input file |
| 3 | nothing usable to process was found: no movie files (or image folders), only file names that do not follow the naming convention, or a given path that is missing or empty |

> **Breaking change from 0.3.x.** Up to 0.3.x (the last release on PyPI before 0.5.1) both tools
> always exited 0. Non-zero codes 1 and 2 arrived in 0.5.0; code 3 in 0.5.2, which also changed
> an empty folder from exit 0 to 3.

Details:

- Precedence when several apply: 2, then 1, then 0, then 3. A run that processed some datasets
  and failed one exits 1.
- Files whose names do not follow the convention are logged and skipped. They do not fail a run
  that found real datasets; a folder holding datasets alongside unrelated `.mrc` output is
  ordinary. Only when nothing usable is found does the run exit 3.
- To tell outcomes apart per dataset, read the result files described next rather than the exit
  status or the presence of `autoprocess_logs/`, which is created at start-up.

## Result files

For every dataset it handles, `autoprocess` writes `autoprocess_logs/<dataset>_status.json` in
the directory it was run from, replacing any earlier record for that dataset:

```json
{
  "schema_version": 1,
  "dataset": "sample-mov1",
  "status": "failed",
  "reason": "indexing failed: no XPARM.XDS after 10 retries",
  "source_file": "/data/run1/sample-mov1_960_0.3_3.mrc",
  "output_dir": "/data/run1/sample-mov1",
  "outputs": {
    "XDS.INP": "/data/run1/sample-mov1/auto_process/XDS.INP"
  },
  "pyautoprocess_version": "0.5.2",
  "updated_at": "2026-09-17T14:03:12"
}
```

- `status` is `success`, `failed` or `skipped`. `skipped` means the dataset was already processed
  by a version that wrote no record; a dataset already recorded as `success` keeps that record.
- `reason` is a short human-readable explanation, e.g. `processing completed`,
  `no good-quality frames found (--dqa)`, `integration failed: no INTEGRATE.HKL`.
- `outputs` maps each key output file that exists to its absolute path. The files looked for are
  `XDS.INP`, `XPARM.XDS`, `INTEGRATE.HKL`, `CORRECT.LP`, `XDS_ASCII.HKL`, `<dataset>.ahkl`,
  `<dataset>.hkl`, `stats.LP` and `pointless.LP`, all in `<output_dir>/auto_process/`.
- No record is written for files whose names are not datasets (they do not follow the naming
  convention).
- Fields may be added in later versions; `schema_version` changes only if an existing field
  changes meaning.

### Failed datasets are remembered

A dataset whose record says `failed` is **not** processed again by a later run: it is skipped with
a log line giving the recorded reason, and the run exits 1. This keeps a monitor that restarts from
re-running XDS on a dataset known to fail. To try again:

- `--retry-failed` retries only datasets recorded as failed;
- `--reprocess` reprocesses everything, including successes.

A record only counts for the exact source file it names, so a same-named dataset in another folder
is unaffected. Successful datasets are also listed in `autoprocess_logs/autoprocess_tracking.log`,
as before; failures are never written there.

## F30 TVIPS data: two configurations

TVIPS movies are converted to TIF exactly as recorded, with no flip or rotation. The lab's
historical `.img` files (made with `tvips2smv`) hold the same frames mirrored left-right, so the
same detector geometry is written differently for the two kinds of input:

| configuration | use for | rotation axis | beam centre (X, Y) |
|---|---|---|---|
| `F30-TVIPS-SM` | raw `.tvips` movies (`autoprocess`) | `-0.8290 0.5592 0` | 1028, 1020 |
| `F30-TVIPS-SMV-SM` | `.img` frames made by `tvips2smv` (`image_process --smv`) | `-0.8290 -0.5592 0` | 1020, 1020 |

The beam-centre X is counted from the other edge. The rotation axis is not simply the X component
negated: an axis is an axial vector, and mirroring the image also reverses the sense of rotation,
so it becomes (X, -Y, -Z). Negating only X (`0.8290 -0.5592 0`) still indexes, but gives a
wrong-handed near-solution with slightly distorted cells and lower ISa. Using the SMV geometry on
raw `.tvips` frames can even "index" into a meaningless triclinic cell with ISa near 1.

## Custom microscope configurations

`--config-file FILE` adds configurations to the built-in set. The file is a JSON object mapping each
configuration name to its settings. An entry whose name matches a built-in configuration replaces
it entirely; nothing is inherited from the built-in. Every entry is validated when the file is
loaded, and any problem is a usage error (exit 2) naming the entry and the field.

A complete example with one new configuration:

```json
{
  "Lab-Talos-Apollo": {
    "rotation_axis": "-1 0 0",
    "frame_size": 4096,
    "signal_pixel": 7,
    "min_pixel": 7,
    "background_pixel": 4,
    "pixel_size": 0.008,
    "wavelength": "0.0251",
    "beam_center_x": 2048,
    "beam_center_y": 2040,
    "file_extension": ".mrc",
    "value_range_min": 10.0,
    "value_range_max": 65535.0,
    "detector_distance": "560",
    "rotation": "1",
    "exposure": "1",
    "background_range_start": 1,
    "background_range_end": 10
  }
}
```

| field | required | type |
|-------|----------|------|
| `rotation_axis` | yes | string of three numbers, e.g. `"-1 0 0"` |
| `frame_size` | yes | integer (pixels) |
| `signal_pixel`, `min_pixel`, `background_pixel` | yes | integer |
| `pixel_size` | yes | number (mm) |
| `wavelength` | yes | number or numeric string (Å) |
| `beam_center_x`, `beam_center_y` | yes | number (pixels) |
| `file_extension` | yes | `".mrc"`, `".ser"` or `".tvips"` |
| `value_range_min`, `value_range_max` | no | number |
| `detector_distance` (mm), `rotation` (deg/s), `exposure` (s) | no | number or numeric string; used when the filename does not supply the value |
| `background_range_start`, `background_range_end` | no | integer (frame) |
| `microscope_config` | no | string; ignored, the key is the name |

Any other field is rejected as unknown, so a misspelling such as `beam_centre_x` is caught rather
than ignored. A validation error looks like:

```
Config file lab.json is invalid:
  entry 'Lab-Talos-Apollo', field 'frame_size': must be an integer (pixels), got '4096'
  entry 'Lab-Talos-Apollo', field 'beam_center_x': missing (required; a number (pixels))
  entry 'Lab-Talos-Apollo', field 'beam_centre_x': unknown field
```

### Listing configuration names

Use either of these rather than reading the package's `data/microscope_configs.json`, which is
internal:

```bash
autoprocess --list-configs                          # built-in names, one per line
autoprocess --list-configs --config-file lab.json   # plus the names the file adds
```

```python
import pyautoprocess
pyautoprocess.list_microscope_configs()                    # ['Arctica-CETA-ser-SM', ...]
pyautoprocess.list_microscope_configs("lab.json")          # raises ValueError if the file is invalid
```

`--list-configs` exits 0 and creates no files or folders.

## Error Handling
- All scripts include comprehensive error handling and logging
- Detailed logs are generated in the respective log directories
- Processing statistics and verification results are recorded
- Failed processes are clearly identified in the logs

## Contributing
Contributions are welcome! Please submit issues and pull requests to the project repository.

## License
This project is licensed under the GPL-3.0-or-later License.

## Authors and Contributors
- **Sam Foxman** (sfoxman@caltech.edu) - Package maintainer and development lead
- **Dmitry Eremin** (eremin@caltech.edu) - Core development and enhancements
- **Jessica Burch** - Original autoprocess.py implementation

## Acknowledgments
- Nelson Lab at Caltech for ongoing support and development
- XDS developers for data processing capabilities
- CCP4 Software Suite for crystallographic tools

## Version History
- **v0.5.2**: Contract for callers such as REyes
  - **Breaking:** a run that finds nothing usable to process now exits **3** instead of 0 (or 1
    when a path was given), for both `autoprocess` and `image_process`. Exit codes 0-3 are now a
    documented, stable contract
  - New per-dataset result files, `autoprocess_logs/<dataset>_status.json`, recording status
    (`success`, `failed`, `skipped`), a reason naming the stage that failed, and the key output
    files produced
  - **Behaviour change:** failed datasets are remembered. A later run skips a dataset recorded as
    failed and exits 1 instead of re-running XDS; new `--retry-failed` tries failures again, and
    `--reprocess` still reprocesses everything
  - `--config-file` entries are checked against a documented schema. Errors name the entry and the
    field, report every problem at once, and reject unknown (e.g. misspelled) fields
  - New `--list-configs` (with or without `--config-file`) and
    `pyautoprocess.list_microscope_configs()` to discover configuration names
  - `--friedel` accepts true/false, yes/no, 1/0 and on/off in any case, and rejects anything else
    with exit 2. Previously any value other than `true` silently meant false
  - **Fixed F30 geometry for raw `.tvips` movies.** `F30-TVIPS-SM` had been calibrated on
    `tvips2smv` `.img` frames, which are mirrored left-right relative to the raw frames that
    `autoprocess` converts. Processing `.tvips` therefore used the wrong rotation axis and beam
    centre, often ending in a meaningless triclinic cell. `F30-TVIPS-SM` now carries the
    raw-orientation geometry (axis `-0.8290 0.5592 0`, beam centre X 1028) and gives the same cells
    and comparable statistics as the SMV runs; the previous values move to the new
    `F30-TVIPS-SMV-SM`, for `.img` frames. **If you ran `image_process --smv` with `F30-TVIPS-SM`,
    switch to `F30-TVIPS-SMV-SM`.** See "F30 TVIPS data"
- **v0.5.1**: Configuration and robustness fixes
  - `--config-file` now works. It was accepted but never read, so a user's own microscope
    configuration was silently ignored. Its entries are added to the built-in configurations;
    an entry with a built-in name replaces that configuration entirely. The file is validated
    on load, and a missing or malformed file is a usage error (exit 2)
  - A missing or unreadable packaged configuration, or an unknown microscope name, now stops the
    run with an error. Previously both fell back silently to generic default settings, which
    processed the data with the wrong detector geometry while appearing to succeed
  - Output is written beside the path given, not beside a symlink's target
  - `--beam-center` accepts a consensus of the frame estimates instead of rejecting the whole set
    whenever the two most distant estimates disagree, which had discarded correct centres
  - The distance, rotation and exposure fields of a filename must be numeric; a name that fails
    is skipped with a message naming the field. Such skipped names only cause a non-zero exit
    when nothing usable was found
  - First release on PyPI since 0.3.0, so it is the first PyPI release with the
    `F30-TVIPS-SM` configuration (added in 0.3.2), `--min-res`, TVIPS support, monitorED and
    the other changes from 0.3.2 to 0.5.0 listed below
- **v0.5.0**: Experimental auto-detection, reproducibility, and correctness fixes
  - **Breaking:** `autoprocess` and `image_process` now return meaningful exit codes (1 failed,
    2 usage error) instead of always exiting 0; failed datasets are no longer recorded as
    processed. See "Exit Codes"
  - New `--seed N`: makes the random indexing-retry search reproducible. Without it, a dataset
    that fails first-pass indexing can return a different space group and unit cell on every run
  - New opt-in `--auto-rotation-axis`: derives the rotation-axis sign from the tilt-direction
    token in the filename, per dataset, falling back to the configured axis
  - New opt-in `--beam-center`: detects the beam centre from the data, falling back to the
    configured value, and records its reasoning in `auto_process/beam_center.LP`. Measured as
    quality-neutral on most datasets, since XDS refines the centre itself; useful mainly where
    the configured centre is far enough off to send that refinement astray
  - Fixed `F30-TVIPS-SM` rotation axis, which had a sign error (`-0.8290 0.5592 0` ->
    `-0.8290 -0.5592 0`). Determined by running all four sign combinations through XDS on three
    datasets: the shipped value FAILED TO INDEX ENTIRELY on all three, so this configuration was
    unusable without a manual `--rotation-axis` override. (Those tests used `tvips2smv` `.img`
    frames. For raw `.tvips` movies the right value is in fact `-0.8290 0.5592 0`; 0.5.2 splits
    the two into `F30-TVIPS-SM` and `F30-TVIPS-SMV-SM`.)
  - Fixed a missing `import json` in autoprocess.py that made every processor construction
    silently fail to load the Bravais-lattice data, papered over by three separate workarounds
  - Rotation axis and beam centre can now vary per dataset within a single run, instead of being
    fixed for the whole invocation
  - Minimum Python raised to 3.10 and scikit-image to 0.25 (the previous `>=3.8` claim was
    already unsatisfiable, since `numpy~=2.0` does not support 3.8)
- **v0.4.2**: `--id` flag for sample-name override in autoprocess
  - New `--id SAMPLE_ID` flag (autoprocess only) overrides the sample name parsed from the filename
  - With `--id`, files that lack the conventional `sample_distance_rotation_exposure` underscore structure are still processed, provided the missing numeric fields can be supplied via CLI flags or microscope config defaults
  - Without `--id`, unconventional filenames remain skipped (unchanged behavior)
  - image_process is unchanged (it already derives the sample name from the folder name)
- **v0.4.1**: Microscope-config defaults and reprocessing fixes
  - Added `detector_distance`, `rotation`, `exposure`, `background_range_start`, `background_range_end` to every microscope configuration
  - New parameter precedence: CLI override > filename-parsed value > microscope config default
  - `--background-range` CLI default is now sourced from the active microscope config
  - Fixed `_setup_movie_directories` so re-runs over a partially-cleaned sample folder still create the missing `images/` and `auto_process/` subdirs
  - Fixed `iterate_opt` writing back a truncated copy of `XDS.LP`, which corrupted the log and broke initial-ISa parsing on subsequent runs
  - `--reprocess` now backs up any existing `auto_process/` to `processing_backups/` before each run so the indexing-retry pipeline always starts from a clean state
- **v0.4.0**: monitorED and filename parsing improvements
  - Added monitorED: active file monitor for automated data collection sessions
  - monitorED supports both autoprocess and image_process modes with flag passthrough
  - File stability detection prevents processing files still being written
  - Persistent tracking log avoids reprocessing across restarts
  - Fixed filename parsing to support 'p' as decimal separator (e.g., `1p5` → `1.5`)
  - Updated Talos-Apollo-P microscope configuration
- **v0.3.2**: Resolution control and TVIPS support
  - Added dynamic XSCALE resolution shell commenting with --min-res argument
  - Enabled TVIPS file format support throughout processing pipeline
  - Added F30-TVIPS-SM microscope configuration
  - Fixed empty auto_process folder backup failures
  - Unified auto_process folder structure with backward compatibility migration
  - Added --friedel parameter for Friedel's law control
- **v0.2.0**: Major refactoring and DQA implementation
  - Implemented quality analysis and diffraction assessment
  - Added modular architecture with separate core, config, and UI modules
- **v0.1.x**: Enhanced autoprocess functionality
  - Restructured as installable Python package (pyautoprocess)
  - Added image_process tool for pre-converted image processing
  - Enhanced error handling and logging capabilities
  - Added support for frame trimming and SMV format
  - Added pointless and parallel processing flags
  - Improved parameter handling and microscope configurations
- **v0.0.x**: Initial development releases
  - Core autoprocess.py functionality
  - Basic batch reprocessing capabilities
  - MRC to TIF conversion utilities
