Metadata-Version: 2.4
Name: RDEFpy
Version: 1.0.0
Summary: Reader and writer for the CCSDS Delta-DOR Raw Data Exchange Format (RDEF, CCSDS 506.1-B-1).
Author-email: Andrea Togni <antogni@moreheadstate.edu>
License-Expression: MIT
Project-URL: Documentation, https://rdefpy.readthedocs.io/en/latest/
Project-URL: Repository, https://github.com/atogni/RDEFpy
Project-URL: Issues, https://github.com/atogni/RDEFpy/issues
Keywords: CCSDS,RDEF,Delta-DOR,DSN,radio science,open-loop
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Programming Language :: Python :: 3.15
Classifier: Programming Language :: Cython
Classifier: Operating System :: OS Independent
Classifier: Intended Audience :: Science/Research
Classifier: Topic :: Scientific/Engineering :: Astronomy
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy>=1.26
Provides-Extra: browse
Requires-Dist: matplotlib>=3.8; extra == "browse"
Dynamic: license-file

# RDEFpy

[![Documentation Status](https://readthedocs.org/projects/rdefpy/badge/?version=latest)](https://rdefpy.readthedocs.io/en/latest/)

Open-source Python reader and writer for the product files of the CCSDS
Delta-DOR Raw Data Exchange Format (RDEF), CCSDS 506.1-B-1 (June 2013), also
published as ISO 20208:2015. It covers the standard's §5: binary one-second
records of complex I/Q with full downconversion metadata, as recorded by
open-loop receivers for radio science and Delta-DOR. The standard's ASCII
observation files (§4) are read and validated; writing them, and checks
across files (§4.2.4 version, §6.2 file names), are not supported.

**Documentation: https://rdefpy.readthedocs.io/en/latest/** (user guide,
examples, API reference).

## Introduction

Open-loop recordings of spacecraft signals serve radio-science experiments,
radiometric tracking and Delta-DOR (delta differential one-way ranging), a
navigation technique derived from very long baseline interferometry (VLBI).
The Consultative Committee for Space Data Systems (CCSDS) defines a common
format for these recordings, the Delta-DOR Raw Data Exchange Format (RDEF):
one-second binary records of complex in-phase/quadrature (I/Q) samples with
their downconversion metadata, required when stations of different agencies
share an observation. To our knowledge no licensed open-source library reads
and writes RDEF, so each station or research group builds and validates its
own; VLBI stations, which record in the VLBI Data Interchange Format (VDIF),
face this before joining a navigation campaign. RDEFpy is a Python reader and
writer for RDEF product files that depends on NumPy alone and decodes every
sample size the standard defines, with tests pinning each against the
standard's bit layout. On a laptop workstation it read recordings about
72,000 times faster than real time for 1-kHz, 16-bit data (6 h), about 3,100
times for 100-kHz, 16-bit data (2 h, 2.9 GB), and about 9 times for 100-MHz,
2-bit wideband data (60 s, 3.0 GB). Records above 1 kHz stay on disk until
accessed, so memory stays bounded on multi-gigabyte files. Stations and
research groups can produce and consume RDEF without in-house format
software, and a VLBI station can add it to a VDIF-based workflow without
changing its recording or correlation pipeline.

## Features

- Reads and writes RDEF product files (`.prd`).
- Decodes every record header field and the I/Q samples of all supported sample sizes.
- Handles multi-gigabyte files: records at or below 1 kHz are loaded;
  faster records stay on disk and are read on access, one at a time.
- Extracts the carrier frequency before downconversion per integration
  time: fractional Fourier transform plus Successive Parabolic
  Interpolation (Togni et al., 2025, Signal Processing 237, 110094).
  Blocks run in parallel within a memory budget (`memory_limit`, by default
  half of the memory the process can still take), counted at about 400 bytes per block sample.
- Depends on NumPy alone. An optional Cython extension speeds up I/Q
  unpacking and carrier extraction; without a compiler the package falls
  back to an equivalent NumPy path.

## Installation

RDEFpy needs Python 3.11 or later and NumPy.

From PyPI:

```
pip install rdefpy
```

The prebuilt wheels include the compiled Cython extension. Where no wheel
matches your platform, pip builds from source as below.

The record browser (`rdefpy.browse`) also needs matplotlib:

```
pip install "rdefpy[browse]"
```

From GitHub, latest version:

```
pip install git+https://github.com/atogni/RDEFpy
```

or from a clone (add `-e` for an editable install when developing):

```
git clone https://github.com/atogni/RDEFpy
cd RDEFpy
pip install .
```

A source build compiles the optional Cython extension, which needs a C compiler:
Microsoft C++ Build Tools on Windows, `gcc` on Linux (e.g. `build-essential`),
or the Xcode command-line tools on macOS (`xcode-select --install`). Without
one the build prints a warning and installs the NumPy path only. To check
that the extension was built:

```
python -c "import rdefpy.prd.unpack_cyth"
```

No error means it is in use.

## Quick start

```python
from rdefpy import read_product, write_product

pf = read_product("scan.prd")  # records above 1 kHz stay on disk until accessed
pf.info()                      # file summary: time span, rate, bits, band
print(pf[0])                   # first record: header fields

for rec in pf:                 # one one-second record at a time
    x = rec.data.samples       # complex I + jQ, NumPy array
    print(rec.header.timetag, rec.data.power_db)

buf = None                     # faster at high rates: decode straight to complex
for rec in pf:                 # and refill one array (copy anything you keep)
    buf = rec.read_samples(out=buf)

x = pf.samples[:1024]                    # any sample range, across records
t = pf.timetags[:1024]                   # datetime64[ns] of each sample
f = pf.downconversion_frequency[:1024]   # Hz, at each sample (§5.2.6)
phi = pf.downconversion_phase[:1024]     # turns, variable part (§5.2.6.2)
f0 = pf.record_downconversion_frequency  # Hz, at each record's first sample
f_mean = pf.mean_downconversion_frequency  # Hz, mean over each record

from rdefpy import extract_carrier       # FrFT + SPI, samples untouched
c = extract_carrier(pf, integration_time=1.0, max_rate_hz_s=10.0)
c["time"], c["frequency_before_downconversion"]           # one row per block; time = block midpoint

part = pf.select(start_epoch="2026-02-08T12:00:00", stop_epoch="2026-02-08T12:00:10")
write_product("part.prd", part)
```

An observation file (`.obs`) lists the product files of each scan:

```python
from rdefpy import read_observation, read_product

for scan in read_observation("session.obs"):
    for entry in scan.products:
        pf = read_product(entry.datafile)
```

More in the [quick start](https://rdefpy.readthedocs.io/en/latest/quickstart.html)
and the [user guide](https://rdefpy.readthedocs.io/en/latest/).

## Performance

Time to read a file and decode every sample to integer I/Q, by sample size,
sample rate and recording duration. Median of 10 runs, files in the OS
cache, 2026-11-06. Intel Core Ultra 9 285HX laptop (24 cores, 64 GB RAM),
Windows 11, Python 3.15.0, NumPy 2.5.4, RDEFpy at commit c7b5e55 with the
Cython extension, 16 threads.

![Decode time per file](https://raw.githubusercontent.com/atogni/RDEFpy/main/docs/realtime.png)

The same measurement for four typical files, with and without the compiled
Cython extension (without it RDEFpy falls back to pure NumPy). Median of 10
runs, same machine and versions, RDEFpy at commit 1e7991f.

![Cython vs pure NumPy](https://raw.githubusercontent.com/atogni/RDEFpy/main/docs/cases.png)

The ratios in the introduction come from the IVS GM2026 poster benchmark
(Cython back-end).

## Testing

```
uv run python -m pytest
```

The tests pin the low-level decoding for every sample size against the
standard's bit layout.

`tests/reference/` holds small synthetic product files with their expected
decoding as text (headers, samples, hex dump, downconversion, DSN 0222
agency block), written from a plain transcription of the standard, not from
RDEFpy. Use them to check another decoder: see
[Checking another decoder](https://rdefpy.readthedocs.io/en/latest/reference_files.html).

## Contributing

Issues and pull requests are welcome. The low-level decoding (the `2k+1`
transform of §5.3.6, the bit offsets, the header struct) is the format itself
and is meant to stay as it is; changes there need a reference to the standard.
Performance improvements in decoding, reading, or processing data are welcome.

## Publications using RDEFpy

- Togni, A., Malphrus, B., Arnold, B., Mayo, D., Sanchez-Net, M., & Asmar, S.
  (2026). Extending VLBI Station Capabilities for Delta-DOR and Navigation via
  Open-Source RDEF Software Compliant with CCSDS Standards. Poster, 14th
  General Meeting of the International VLBI Service for Geodesy and Astrometry
  (IVS GM2026), Würzburg.
  https://indico.astro.uni-wuerzburg.de/event/2/contributions/143/

## License

MIT. See `LICENSE`.
