# dbprocessing

# dbprocessing README

`dbprocessing` is a Python-based, database-driven process controller which automates the production of derived data products upon the arrival of new input data. Although originally written for Heliophysics data, it is intended to be flexible enough to manage most forms of digital time-series data.

## What dbprocessing does

Given a description of relationship between data files, a set of codes to process data files to derived products, and input files themselves, `dbprocessing` iteratively runs the appropriate codes to make all possible output files.

`dbprocessing` delegates the details of producing files to mission-specific processing codes. A processing code must have a command-line interface and produce a single output file from one or more inputs. There are no language restrictions; dbprocessing has been used with C, Interactive Data Language (IDL), Java, and Python.

Support for a file format requires about 30 lines of Python to identify the product and extract required metadata, which can support many different products

When new versions of codes or input files are provided, codes are re-run to ensure all outputs are up to date.

## Current Status

`dbprocessing` has been used in production for nine years in several different projects; however, this has always been with the direct support of the developers.

`dbprocessing` should be considered to be in an early beta state. It is not currently suitable for use without developer support; however, if you are considering dbprocessing to support a mission or project, the developers would be happy to work with you. The developers are working daily to improve the maturity of the code, documentation, and the infrastructure supporting development.

## Relationship to SpacePy

Several `dbprocessing` developers are also SpacePy developers. The SpacePy organization is hosting `dbprocessing` and providing community support as `dbprocessing` is prepared for the public and grows its own community. The SpacePy developers are not, as a whole, responsible for `dbprocessing`.

`dbprocessing` is not a component of SpacePy, nor does it require SpacePy. SpacePy is generally useful in processing Heliophysics data, e.g. in the codes that `dbprocessing` manages.

## Development

Development of `dbprocessing` is primarily supported by the projects which make use of it to deliver data. Development is performed in the public github repository at <https://github.com/spacepy/dbprocessing/>.

Release: 0.1.0 Doc generation date: Feb 10, 2022