Metadata-Version: 2.3
Name: pyfile-prod
Version: 0.2.3
Summary: A set of Python command-line utilities for managing data files from Campbell Scientific and Spectrum instruments
Requires-Dist: pandas>=3.0.3
Requires-Dist: pyfiglet>=1.0.4
Requires-Dist: typer>=0.27.0
Requires-Dist: psycopg2-binary>=2.9.12
Requires-Dist: collection>=0.1.6
Requires-Python: >=3.12
Description-Content-Type: text/markdown

# pyFile

`pyFile` is a Python command-line application for working with data files from
dataloggers. It can:

- Split or merge data files
- Identify gaps in timestamped measurements
- Upload data from supported TOA5 files to the project's database

## Requirements

- Python 3.12 or newer
- To upload file data to the database, you must run `pyFile` on the tower's
  Raspberry Pi

## Installation

`pyFile` is available on [PyPI](https://pypi.org/project/pyfile-prod/). Install
it with:

```bash
pip install pyfile-prod
```

## Commands

Use `--help` with `pyfile` or any command to display usage information and
available options:

```bash
pyfile --help
pyfile arrange --help
pyfile report --help
pyfile uploaddb --help
```

### Arrange files

Arrange and split file data by day, month, or year, or merge the data into one
file. All files must belong to the same dataset and have the same table columns.
Supported file formats are TOA5, SWD, and CSV.

```bash
pyfile arrange --mode d <directory or file paths separated by spaces>
pyfile arrange --mode m <directory or file paths separated by spaces>
pyfile arrange --mode y <directory or file paths separated by spaces>
pyfile arrange --mode a <directory or file paths separated by spaces>
```

### Report missing data

Find gaps between consecutive timestamps. For example:

```bash
pyfile report --expected-gap 30 --time-unit minutes <directory or file paths separated by spaces>
```

Supported time units are `milliseconds`, `seconds`, `minutes`, and `hours`.
The report can be printed in the terminal, saved as CSV, or both.

Compare two previously saved reports to list the gaps found in only one file:

```bash
pyfile report compare <first report.csv> <second report.csv>
```

The comparison is printed in the terminal and can optionally be saved as CSV.

### Upload TOA5 data to the database

Upload one or more supported, uncompressed TOA5 `.dat` files data to the database. Run this on the production Raspberry Pi:

```bash
pyfile uploaddb <file paths or directory>
```

Currently supported filenames are:

- `FERLD01_dendroTable.dat` through `FERLD06_dendroTable.dat`
- `FERLD01_sapflowTable.dat` through `FERLD06_sapflowTable.dat`
- `FERLD07_dendroTable.dat`
- `FERLD07_soilTable.dat`
- `FERLD08_dendroTable.dat`
- `FERLD08_soilTable.dat`
- `fluxcsformat.dat`
- `fluxnotesabridged.dat`
- `fluxstorage.dat`
- `intavg.dat`
- `particules.dat`

The idea is to take a valid TOA5 file with the data that you want to upload and give it the corresponding name from the ones above. When you upload a file to the database, duplicate records are ignored, and only
new records are inserted.

## Project layout

```text
src/pyfile_prod/
├── commands/          # Typer command definitions
├── utils/
│   ├── database/      # PostgreSQL queries and connection pool
│   ├── files.py       # Input detection, parsing, and collections
│   ├── processors.py  # Database upload transformations
│   ├── reporter.py    # Missing-data detection
│   ├── divider.py     # Period-based output splitting
│   └── merger.py      # File merging
├── enums.py
└── main.py
```
