Metadata-Version: 2.4
Name: addereq-dm
Version: 2.1.1
Summary: SDK for geophysics time-series data access, lightweight processing, and plotting
Author-email: WANG Qinglin <chd_wql@qq.com>
License-Expression: MIT
Project-URL: Homepage, https://github.com/chdwql/addereq-dm
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Scientific/Engineering
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: requests>=2.25.0
Requires-Dist: pandas>=1.3.0
Requires-Dist: gmssl>=3.2.0
Requires-Dist: tenacity>=8.0.0
Requires-Dist: pydantic>=2.0.0
Provides-Extra: viz
Requires-Dist: matplotlib>=3.3.0; extra == "viz"
Requires-Dist: numpy>=1.21.0; extra == "viz"
Provides-Extra: dev
Requires-Dist: pytest>=6.0; extra == "dev"
Requires-Dist: pytest-cov>=2.0; extra == "dev"
Requires-Dist: black>=21.0; extra == "dev"
Requires-Dist: flake8>=3.8; extra == "dev"
Requires-Dist: mypy>=0.910; extra == "dev"
Dynamic: license-file

# addereq-dm

Dameng-backed earthquake precursor time-series data access, processing, and plotting SDK. This package is the API-based successor to the Oracle-oriented `addereq` package.

## Install

Python 3.9 or newer is required.

Install the package with plotting support. The distribution name is `addereq-dm` and the Python import name is `addereq_dm`:

```bash
pip install "addereq-dm[viz]"
```

## Connection config

For CLI usage, the recommended place for credentials is a user-level config file instead of the project directory.

Default config file location:
- Windows: `%APPDATA%\\addereq-dm\\config.env`
- Linux/macOS: `~/.config/addereq-dm/config.env`

Initialize the config from CLI with your own values:

```bash
addereq-dm init-config \
  --base-url http://your-api-server:8080 \
  --app-id your_app_id \
  --secret your_secret
```

If any required value is omitted, the CLI will prompt for it interactively. `timeout` defaults to `10`.

Or initialize from Python:

```python
from addereq_dm import initialize_user_config

initialize_user_config(
    base_url="http://your-api-server:8080",
    app_id="your_app_id",
    secret="your_secret",
)
```

Example written file:

```env
DM_API_BASE_URL="http://your-api-server:8080"
DM_API_APP_ID="your_app_id"
DM_API_SECRET="your_secret"
DM_API_TIMEOUT="10"
```

The CLI resolves values in this order:
- command-line arguments
- `--env-file` if provided
- the default user config file above
- current process environment variables
- built-in defaults such as `DM_API_TIMEOUT=10`

Business parameters such as station, point, item, sample rate, and time range are still expected to be passed explicitly in code or CLI commands.

## Quick start

```python
from addereq_dm import create_geophysics_client

api = create_geophysics_client(
    base_url="http://your-api-server:8080",
    app_id="your_app_id",
    secret="your_secret",
)
```

## Core fetch API

```python
df = api.ts.fetch_dys(
    station="taian_center",
    point="1",
    item="vertical_z",
    source_sample_rate="02",
    output_sample_rate="02",
    start_time="2025-11-01 00:00:00",
    end_time="2025-11-03 00:00:00",
)
```

The Python API uses explicit names for the two sample-rate concepts:
- `source_sample_rate`: raw input sample rate
- `output_sample_rate`: output sample rate for the returned series

For the CLI, use the corresponding options `--source-sample-rate` and
`--output-sample-rate`. The configuration variables are
`DM_SOURCE_SAMPLE_RATE` and `DM_OUTPUT_SAMPLE_RATE`.

Accepted sample-rate aliases currently include:
- `01`, `minute`, `min`
- `02`, `second`, `sec`
- `60`, `hour`, `h`
- `90`, `day`, `d`
- common Chinese aliases are also supported in code

`source_sample_rate` must match the rate of the data stored upstream. The
`output_sample_rate` may request a coarser result, for example source seconds
to output minutes. It cannot recover second-level observations from
minute-level source data.

The Chinese aliases map as follows:

| Alias | Code | Meaning |
| --- | --- | --- |
| `分钟值` | `01` | minute-level data |
| `秒钟值` | `02` | second-level data |
| `整点值` | `60` | hourly data |
| `日值` | `90` | daily data |

### DYS and DYU

Use `fetch_dys()` for DYS raw data:

```python
dys = api.ts.fetch_dys(
    station="taian_center",
    point="1",
    item="vertical_z",
    source_sample_rate="02",
    output_sample_rate="01",
    start_time="2025-11-01 00:00:00",
    end_time="2025-11-02 00:00:00",
)
```

Use `fetch_dyu()` for DYU preprocessed data. Its call shape is the same;
the `kind` is selected by the method name:

```python
dyu = api.ts.fetch_dyu(
    station="taian_center",
    point="1",
    item="vertical_z",
    source_sample_rate="02",
    output_sample_rate="01",
    start_time="2025-11-01 00:00:00",
    end_time="2025-11-02 00:00:00",
)
```

Use `fetch_dys_with_report()` or `fetch_dyu_with_report()` when you need
success/failure details for multiple targets or long time ranges.

### Fetch parameters

The common parameters for `fetch_dys()`, `fetch_dyu()`, and their
`*_with_report()` variants are:

| Parameter | Required | Description |
| --- | --- | --- |
| `station` | Yes | Station ID or station name; use `"all"` for all stations. |
| `point` | No | Point ID, point number, or point name. Omit to match multiple points. |
| `item` | Yes | Item ID or item name; use `"all"` for all items. |
| `source_sample_rate` | No | Actual source-data rate, default `"02"`. |
| `output_sample_rate` | No | Returned-data rate; the Python API default is `"02"`, so set it explicitly when needed. |
| `start_time` | Yes | Start time in `YYYY-MM-DD HH:MM:SS` format. |
| `end_time` | Yes | Exclusive end time in `YYYY-MM-DD HH:MM:SS` format. |
| `max_targets` | No | Maximum number of resolved station/point/item targets, default `50`. |
| `allow_partial` | No | Keep successful targets or chunks when another request fails. |
| `max_workers` | No | Concurrent target workers; use `1` for serial requests. |

`fetch_dys()` and `fetch_dyu()` return a normalized pandas DataFrame, or
`None` when no target can be resolved. The `*_with_report()` variants return
`(dataframe, report)` so partial failures can be inspected.

Supported input styles:
- `station`: station id or station name
- `point`: point id, point number, or point name; may be omitted
- `item`: item id or item name

When resolution fails or the query is ambiguous, the resolver returns candidate suggestions to help narrow the scope.

Long-range requests are handled automatically. When the requested range spans more than 30 natural days, the SDK splits the request into multiple upstream calls, merges the returned frames, sorts by timestamp, and deduplicates boundary rows. Chunk boundaries follow an exclusive `end_time` rule, so each chunk covers `[start_time, end_time)` and the next chunk starts exactly at the previous chunk's `end_time`.

Use the next midnight when requesting a complete final day. For example, data for November 1-2 should use `end_time="2025-11-03 00:00:00"`.

## Scope keywords

The following keywords are supported for broad queries:
- `"all"`
- `"*"`
- common Chinese equivalents for “all” are also supported in code

## Lightweight processing

```python
stats = api.ts.summarize(df)
report = api.ts.quality_report(df)
hourly = api.ts.resample(df, "1h")
centered = api.ts.center(hourly)
detrended = api.ts.detrend(df, method="mean")
cleaned = api.ts.clip_outliers(df, zscore=3.0)
differenced = api.ts.difference(df)
robust = api.ts.hampel(df, window_size=15, threshold=5.0)
complete = api.ts.expand_timeline(df)
aligned = api.ts.align([df1, df2], labels=["station_a", "station_b"])
```

The processing layer normalizes sentinel missing values such as `999999` into real missing values before summaries, quality checks, resampling, alignment, and plotting.

Frames containing multiple logical series are processed independently by `STATIONID`, `POINTID`, and `ITEMID`. Multi-series summaries and quality reports include a `series` list with per-series results. Pass `group_cols=[]` only when an intentional whole-frame calculation is required.

Normalized output uses `SAMPLERATE` for the actual returned series rate and `SOURCE_SAMPLERATE` for the upstream source rate. `VALUE` is the canonical value column; `OBSVALUE` remains as a compatibility mirror for existing `addereq` workflows.

`expand_timeline` uses the sample rate to insert timestamps omitted by the API. `MISSING_REASON="missing_record"` identifies inserted rows, while `MISSING_REASON="sentinel"` identifies explicit upstream values such as `999999`.

## Optional metadata snapshot

Online resolution and plotting use live API metadata plus process-local memory caching. They do not automatically read or write persistent metadata. If offline plotting is required, explicitly create a station, point, item, and unit snapshot under the same user configuration directory as `config.env`.

Snapshot namespaces are SHA-256 hashes of the normalized API base URL; credentials are never written into snapshot files or keys. Refresh failures are reported directly and never hidden by silently using old metadata.

```bash
addereq-dm metadata path
addereq-dm metadata status
addereq-dm metadata refresh
addereq-dm metadata refresh --station-id 37001
addereq-dm metadata clear
```

## Optional batch fetch report

```python
df, report = api.ts.fetch_dys_with_report(
    station="all",
    point="1",
    item="3123",
    source_sample_rate="02",
    output_sample_rate="02",
    start_time="2025-11-01 00:00:00",
    end_time="2025-11-03 00:00:00",
    max_targets=20,
    allow_partial=True,
)
```

`report` contains:
- a local request identifier for tracing one aggregated SDK fetch
- total elapsed time and elapsed time per target
- target count
- success count
- failure count
- successful targets
- failed targets with error summaries
- requested chunk count
- successful chunk count
- failed chunk count and failed time ranges

With `allow_partial=True`, successful chunks remain available when another chunk for the same target fails. The failures remain visible in `report`.

## Plotting

Plotting accepts a DataFrame returned by `fetch_*()` or restored with the
export/import helpers. It uses the metadata returned by the API to label
stations, points, items, and units.

Plot grouped by item:

```python
api.plot.plot_by_items(
    df,
    prefix="demo_",
    fig_label="_item",
    show_mean=True,
    overlay_earthquakes=True,
    xlabel="Time",
)
```

Plot grouped by station and point:

```python
api.plot.plot_by_stations(
    df,
    prefix="demo_",
    fig_label="_station",
    show_mean=False,
    overlay_earthquakes=False,
    ylabel="Displacement",
)
```

### Plot parameters

Common parameters for `plot_by_items()` and `plot_by_stations()` include:

| Parameter | Description |
| --- | --- |
| `ts` | Normalized time-series DataFrame. |
| `prefix` | Filename prefix for generated figures. |
| `fig_label` | Extra label added to generated filenames. |
| `plot_type` | Plot style, normally `"line"`. |
| `show_mean` | Draw the mean line when `True`. |
| `show_std` | Draw standard-deviation bands when `True`. |
| `start_time`, `end_time` | Limit the displayed time range without refetching data. |
| `invert_y` | Reverse the y-axis when `True`. |
| `figure_format` | Output format, default `"png"`. |
| `overlay_earthquakes` | Add earthquake markers when earthquake data is supplied. |

`plot_by_items()` creates figures grouped by item and orders station/point
subplots by station. `plot_by_stations()` creates figures grouped by station
and point and orders item subplots by item. Additional matplotlib axis options
can be passed through as keyword arguments.

Plotting dependencies are loaded lazily, so importing the package without `matplotlib` is still supported when plotting is not used.

Subplot ordering rules are stable:
- `plot_by_items` sorts station subplots by `STATIONID`
- `plot_by_stations` sorts item subplots by `ITEMID`

## CLI

Initialize user config:

```bash
addereq-dm init-config --base-url http://your-api-server:8080 --app-id your_app_id --secret your_secret
```

Resolve station, point, or item:

```bash
addereq-dm resolve --station taian_center --point 1 --item vertical_z
```

Fetch data and export to file:

```bash
addereq-dm fetch \
  --station taian_center \
  --point 1 \
  --item 3123 \
  --start-time "2025-01-01 00:00:00" \
  --end-time "2025-01-01 01:00:00" \
  --kind dys \
  --allow-partial \
  --workers 2 \
  --out data.csv \
  --summary
```

CSV, JSON, and Parquet exports include a `<data-file>.metadata.json` sidecar with the query, data kind, labels, units, and fetch report. The SDK uses this sidecar automatically when the file is imported or plotted locally:

```python
from addereq_dm import export_timeseries, import_timeseries

export_timeseries(df, "data.parquet", metadata={"project": "weekly-review"})
restored = import_timeseries("data.parquet")
```

`--workers` enables controlled concurrency across independent resolved targets. It defaults to `1` and is capped at `8`; each target's 30-day windows remain sequential, and merged output keeps target order deterministic.

Plot an existing file:

```bash
addereq-dm plot --input data.csv --output-dir figures
```

Use an explicit persistent metadata snapshot instead of the export sidecar:

```bash
addereq-dm plot --input data.csv --offline-metadata --output-dir figures
```

Local files exported by the SDK restore their embedded labels and units automatically. `--offline-metadata` explicitly replaces those labels with the persistent snapshot and is the only plotting mode that reads that snapshot.

Inspect a local file without API configuration:

```bash
addereq-dm quality --input data.csv
```

Fetch and plot directly:

```bash
addereq-dm plot \
  --station taian_center \
  --point 1 \
  --item 3123 \
  --start-time "2025-01-01 00:00:00" \
  --end-time "2025-01-01 01:00:00" \
  --kind dys \
  --output-dir figures
```

Use `--kind dyu` for processed data. CP is intentionally not exposed until its upstream API is stable.

See [ROADMAP.md](ROADMAP.md) for compatibility, rate limiting, and reporting work.
