Metadata-Version: 2.4
Name: ema-data-access
Version: 0.11.0
Summary: EMA Data Access
License: MIT
License-File: LICENSE
Author: EMA PDC Developers
Requires-Python: >=3.12,<4
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Programming Language :: Python :: 3.15
Provides-Extra: dev
Provides-Extra: test
Requires-Dist: pre-commit (>=4.0.0,<5.0.0) ; extra == "dev"
Requires-Dist: pytest (>=6.2.5) ; extra == "test"
Requires-Dist: pytest-cov (>=4.0.0,<5.0.0) ; extra == "test"
Requires-Dist: requests (>=2.28.0,<3.0.0)
Requires-Dist: ruff (>=0.2.1) ; extra == "dev"
Description-Content-Type: text/markdown

# ema-data-access

Lightweight Python tools to query and access EMA data.

## Setup

### Python environment (Poetry)

1. [Install Poetry](https://python-poetry.org/docs/#installation) if you don't have it:
   ```
   curl -sSL https://install.python-poetry.org | python3 -
   ```

2. Create a virtual environment in the project directory:
   ```
   python3 -m venv venv
   source venv/bin/activate
   ```

3. Install dependencies:
   ```
   poetry install --extras "dev test"
   ```

4. Install pre-commit hooks:
   ```
   poetry run pre-commit install
   ```

### Python environment (pip)

1. Create a virtual environment in the project directory:
   ```
   python3 -m venv venv
   source venv/bin/activate
   ```

2. Install the package, with the `dev`/`test` extras, from this checkout:
   ```
   pip install -e ".[dev,test]"
   ```

3. Install pre-commit hooks:
   ```
   pre-commit install
   ```

## File naming conventions

Every file in the EMA archive must match one of the naming conventions
below.

`<payload>` is one of `rept`, `mista`, `embirs`, `emacs`, `prb`, `cnc`, `pnc`,
`irc`. `<source>` is one of `flight`, `softsim`, `flatsat`. `<data_level>` is
one of `l0`, `l1`, `l1a`, `l1b`, `l2`, `l2a`, `l2b`, `l3`, `ql`.

| Table | Convention |
| --- | --- |
| `ancillary` | `ema_l1_anc_sc_<apid>_<YYYYMMDD>.csv` |
| `manifest` | `<payload>_manifest_<YYYYMMDDHHMM>.txt`, or `moc_manifest_<YYYYMMDDHHMM>.txt` for a payload-less MOC manifest |
| `housekeeping` | `ema_l0_hsk_<payload>_<source>_<YYYYMMDD>.pkts` |
| `science` (L0) | `ema_l0_sci_<payload>_<source>_<YYYYMMDD>.pkts` |
| `science` (L1a+) | `ema_<payload>_<data_level>_<YYYYMMDDtHHMMSS>_<descriptor>_<pred_rec>_v<NN>-<NN>.fits`, where `<pred_rec>` is `p` (predicted) or `r` (reconstructed) and `v<NN>-<NN>` is the two-digit major and minor version, both required (e.g. `ema_embirs_l1a_20240101t000000_observing-mode-info_p_v01-00.fits`) |
| `mission_events` | `ema_mission_events_<start_date:YYYYMMDD>_<end_date:YYYYMMDD>.xml` |

### SPICE kernel naming conventions

SPICE kernels follow NAIF conventions rather than the `ema_l*_*` pattern above.
`<start_date>`/`<end_date>` are `YYYYMMDD`. `<version>` is free-form
alphanumeric (e.g. `v001`) for the first three conventions below, and
digits-only for the rest.

| Convention | Pattern | Example |
| --- | --- | --- |
| Spacecraft ephemeris | `ema_<type>_<start_date>_<end_date>_<version>.bsp`, `<type>` one of `pred`, `recon`, `ref` | `ema_recon_20240101_20240201_v001.bsp` |
| Attitude | `ema_<type>_<start_date>_<end_date>_<version>.bc`, `<type>` one of `rck`, `pck` | `ema_rck_20240101_20240201_v001.bc` |
| Body ephemeris | `ema_<body>_<version>.bsp`, `<body>` one of `sun`, `venus`, `earth`, `mars`, `wes`, `chi`, `roc`, `va28`, `rc76`, `sg6`, `jus` | `ema_sun_v001.bsp` |
| Planetary ephemeris | `<type><version>.bsp`, `<type>` one of `de`, `mar` | `de440.bsp` |
| Leapseconds | `naif<version>.tls` | `naif0012.tls` |
| Planetary constants | `pck<version>.tpc` or `.bpc` | `pck00011.tpc` |
| Spacecraft clock / frames | `ema_<type>_<version>.tsc` or `.tf`, `<type>` one of `sclk`, `fk` | `ema_sclk_0012.tsc` |

## Command Line Utility

### Query the ancillary table

Query the ancillary table for files matching a set of filters. An API key is
optional — without one, only `released` files are returned.

```bash
$ EMA_API_KEY=<your-api-key> ema-data-access --url <url> query-ancillary --apid 1234 --file-extension csv
```

or with CLI flags

```bash
$ ema-data-access --url <url> --api-key <your-api-key> query-ancillary --apid 1234 --file-extension csv
```

Other available filters: `--file-name`, `--timetag-start`, `--timetag-end`,
`--version`, `--md5checksum`. Results are returned as JSON.

Under the hood, this is equivalent to:

```bash
$ curl -H "x-api-key: $EMA_API_KEY" "<url>/query_ancillary?apid=1234&file_extension=csv"
```

### Query the housekeeping table

Query the housekeeping table for files matching a set of filters. An API key
is optional. Without one, only `released` files are returned.

```bash
$ EMA_API_KEY=<your-api-key> ema-data-access --url <url> query-housekeeping --payload mista
```

Other available filters: `--file-name`, `--timetag-start`, `--timetag-end`,
`--version`, `--md5checksum`. Results are returned as JSON.

Under the hood, this is equivalent to:

```bash
$ curl -H "x-api-key: $EMA_API_KEY" "<url>/query_housekeeping?payload=mista"
```

### Query the science table

Query the science table for files matching a set of filters. An API key is
optional. Without one, only `released` files are returned.

```bash
$ EMA_API_KEY=<your-api-key> ema-data-access --url <url> query-science --payload embirs --data-level l1a
```

Other available filters: `--file-name`, `--timetag-start`, `--timetag-end`,
`--descriptor`, `--pred-rec`, `--file-extension`, `--major-version`,
`--minor-version`, `--md5checksum`. Results are returned as JSON.

Under the hood, this is equivalent to:

```bash
$ curl -H "x-api-key: $EMA_API_KEY" "<url>/query_science?payload=embirs&data_level=l1a"
```

### Query the mission events table

Query the mission_events table for event files matching a set of filters. An
API key is optional. Without one, only `released` files are returned.

Events span a date range, so `--start-date` and `--end-date` define a query
window and any event whose own range overlaps that window is returned.

```bash
$ EMA_API_KEY=<your-api-key> ema-data-access --url <url> query-mission-events --start-date 20240101 --end-date 20240110
```

Other available filters: `--file-name`, `--version`, `--md5checksum`.
Results are returned as JSON.

Under the hood, this is equivalent to:

```bash
$ curl -H "x-api-key: $EMA_API_KEY" "<url>/query_mission_events?start_date=20240101&end_date=20240110"
```

### Query the manifest table

Query the manifest table for files matching a set of filters. Manifest rows
are public, so no API key is required.

```bash
$ ema-data-access --url <url> query-manifest --payload embirs
```

Other available filters: `--file-name`, `--timetag-start`, `--timetag-end`.
Results are returned as JSON.

`--payload moc` matches MOC manifests, which have no payload of their own
(`moc_manifest_<YYYYMMDDHHMM>.txt`).

Under the hood, this is equivalent to:

```bash
$ curl "<url>/query_manifest?payload=embirs"
```

### Query the spice table

Query the spice table for kernel files matching a set of filters. An API key
is optional, but not needed in practice — SPICE kernels are always
`released`.

```bash
$ ema-data-access --url <url> query-spice --file-root naif
```

Other available filters: `--file-name`, `--min-date-j2000`,
`--max-date-j2000`, `--min-date-datetime`, `--max-date-datetime`,
`--delivery-date-start`, `--delivery-date-end`, `--od-number`, `--version`,
`--limit`. Results are returned as JSON.

Under the hood, this is equivalent to:

```bash
$ curl "<url>/query_spice?file_root=naif"
```

### Build a SPICE metakernel

Build a metakernel covering a time window, given in seconds past J2000.

```bash
$ ema-data-access --url <url> metakernel --start-time 0 --end-time 100000 > mission.tm
```

Other available options: `--kernel-types` (comma-separated `kernel_type`
names, e.g. `ephem_reconstructed,ephem_predicted`) to restrict which kernels
are included, `--list-files` to get the file names instead of metakernel
text, and `--require-coverage` to error instead of returning a partial
metakernel if the window isn't fully covered.

Under the hood, this is equivalent to:

```bash
$ curl "<url>/metakernel?start_time=0&end_time=100000" -o mission.tm
```

### Download the kernels and furnish the metakernel

A metakernel only lists kernel file names — it doesn't bundle the kernels
themselves. Download every file it references into the same directory as the
metakernel, then furnish it with
[SpiceyPy](https://github.com/AndrewAnnex/SpiceyPy). SPICE kernels are always
released, so no API key is required.

```python
import ema_data_access
import spiceypy

start_time, end_time = 0, 100000
kernel_dir = "kernels"

file_names = ema_data_access.metakernel(
    start_time=start_time, end_time=end_time, list_files=True
)
ema_data_access.download(file_names, destination=kernel_dir)

with open(f"{kernel_dir}/mission.tm", "w") as f:
    f.write(ema_data_access.metakernel(start_time=start_time, end_time=end_time))

spiceypy.furnsh(f"{kernel_dir}/mission.tm")
```

### Upload files

Upload a single local file, or every file in a directory, to the EMA data
archive. File names must match a known EMA naming convention, and uploading
requires an API key with developer-level access — request one from the EMA
PDC team.

```bash
$ EMA_API_KEY=<your-api-key> ema-data-access --url <url> upload path/to/ema_l1_anc_sc_1234_20240101.csv
```

or with CLI flags

```bash
$ ema-data-access --url <url> --api-key <your-api-key> upload path/to/dir
```

A directory's files can be any mix of types. Subdirectories and hidden files
are skipped, and rejected files are listed in an error after the rest upload.

Under the hood, this requests presigned upload URLs and then PUTs each file,
equivalent to:

```bash
$ RESPONSE=$(curl -s -X POST -H "x-api-key: $EMA_API_KEY" -H "Content-Type: application/json" \
    -d '{"filenames": ["ema_l1_anc_sc_1234_20240101.csv"]}' <url>/upload)
$ UPLOAD_URL=$(echo "$RESPONSE" | python3 -c "import json,sys; print(json.load(sys.stdin)['results'][0]['upload_url'])")
$ curl -X PUT -H "Content-Type:" -T path/to/ema_l1_anc_sc_1234_20240101.csv "$UPLOAD_URL"
```

The `-H "Content-Type:"` (empty) is required — the presigned URL isn't signed with a
content-type, and curl's guessed one will cause a signature mismatch against S3.

### Download files

Download one or more files from the EMA data archive by name. Unreleased files require
an API key with at least team-level access.

```bash
$ EMA_API_KEY=<your-api-key> ema-data-access --url <url> download ema_l1_anc_sc_1234_20240101.csv
```

or with CLI flags

```bash
$ ema-data-access --url <url> --api-key <your-api-key> download ema_l1_anc_sc_1234_20240101.csv
```

By default, files are saved in the current directory. Pass `--destination`
to save them in another directory:

```bash
$ ema-data-access --url <url> download naif0012.tls de440.bsp --destination path/to/dir
```

Under the hood, this requests presigned
download URLs for all the files in one call, then downloads each one:

```bash
$ curl -X POST -H "x-api-key: $EMA_API_KEY" -H "Content-Type: application/json" \
    -d '{"filenames": ["naif0012.tls", "de440.bsp"]}' "<url>/download"
```

Each entry in the returned `results` has a `download_url`. Fetch each one
without the API key; the presigned URL carries its own authorization:

```bash
$ curl -o naif0012.tls "<download_url>"
```

## Importing as a package

```python
import ema_data_access

ema_data_access.config["DATA_ACCESS_URL"] = "<url>"
ema_data_access.config["API_KEY"] = "<your-api-key>"

results = ema_data_access.query_ancillary(apid=1234, file_extension="csv")

results = ema_data_access.query_housekeeping(payload="mista")

results = ema_data_access.query_science(payload="embirs", data_level="l1a")

results = ema_data_access.query_mission_events(
    start_date="20240101", end_date="20240110"
)

results = ema_data_access.query_manifest(payload="embirs")

results = ema_data_access.query_spice(file_root="naif")

ema_data_access.upload("path/to/ema_l1_anc_sc_1234_20240101.csv")

ema_data_access.download(["naif0012.tls", "de440.bsp"], destination="path/to/dir")
```

## Running tests

```
pytest
```

