Metadata-Version: 2.5
Name: signal-dataset
Version: 0.2.3
Summary: An open, profile-free format and Python library for large-scale multidimensional signal datasets.
Project-URL: Documentation, https://superpose-labs.github.io/signal-dataset/
Project-URL: Issues, https://github.com/superpose-labs/signal-dataset/issues
Project-URL: Repository, https://github.com/superpose-labs/signal-dataset
Author: Superpose Labs
License-Expression: Apache-2.0
License-File: LICENSE
Requires-Python: <3.14,>=3.11
Requires-Dist: array-record<0.9,>=0.8.3
Requires-Dist: numpy<2.5,>=1.26
Requires-Dist: safetensors<0.9,>=0.5
Provides-Extra: gcs
Requires-Dist: google-cloud-storage<4,>=3; extra == 'gcs'
Provides-Extra: s3
Requires-Dist: boto3<2,>=1.35.69; extra == 's3'
Description-Content-Type: text/markdown

# Signal Dataset

Immutable, indexed storage for multidimensional signal records on local filesystems, GCS, and S3.
Numerical fields use SafeTensors; ArrayRecord provides random access.

> **Status:** experimental 0.x software. The Python API may make documented breaking changes
> between minor releases. Persisted-format compatibility is versioned separately.

```python
import numpy as np
import signal_dataset as sds

record = sds.Record(
    id="capture-0042",
    fields={
        "iq": sds.Field(
            np.zeros((4, 4096), dtype=np.complex64),
            axes=(sds.Axis("channel", 4), sds.Axis("time", 4096)),
        )
    },
    metadata={"sample_rate_hz": 20_000_000},
)

shard = sds.write_shard([record], "captures.sds", work_id="worker-000")
dataset = sds.publish(
    "captures.sds",
    [shard],
    dataset_id="captures",
    snapshot_id="run-001",
)

assert dataset[0].id == "capture-0042"
assert dataset.record_metadata[0]["iq"].shape == (4, 4096)

for descriptor in dataset.iter_record_metadata():
    print(descriptor.id)

for full_record in dataset.iter_records():
    assert full_record["iq"].data.shape == (4, 4096)
```

`dataset.record_metadata[index]` reads one aligned metadata entry without fetching signal tensors.
`dataset.iter_record_metadata()` streams all metadata in logical order with bounded shard-store
requests and keeps memory bounded to `StorageOptions.records_per_read_batch` records. The shipped
ArrayRecord store serves each request with one underlying reader lifetime; custom stores control
their own `read_many()` implementation.
`dataset.iter_records()` provides the same ordered, shard-batched traversal for full records,
including tensor payloads and normal record validation.
Readers follow generation-pinned manifests and never list directories or GCS prefixes.

See the [quickstart](docs/quickstart.md), [annotation example](docs/how-to/annotations.md), and
[distributed-writing guide](docs/how-to/distributed-writing.md).

## Install

```bash
uv add signal-dataset
uv add 'signal-dataset[gcs]'  # GCS transport
uv add 'signal-dataset[s3]'   # S3 transport
```

Equivalent pip commands are `pip install signal-dataset`,
`pip install 'signal-dataset[gcs]'`, and `pip install 'signal-dataset[s3]'`.

Python 3.11–3.13 on Linux and macOS are supported. Windows is not currently supported because the
local atomic-publication implementation uses POSIX filesystem primitives.

## Design

- Immutable, root-last publication: readers observe no dataset or a complete snapshot.
- Lazy ordinal random access without directory or bucket-prefix discovery.
- Tensor-free aligned metadata reads.
- Storage and shard-container extension contracts.
- Profile-free records: no modality, training framework, or split policy is embedded in the core.

The project does not provide mutable datasets, distributed scheduling, authorization, retention,
or batching policy. See the [architecture](docs/architecture.md), [format compatibility
contract](docs/concepts/compatibility.md), and [roadmap](ROADMAP.md).

Signal Dataset is use-case agnostic. It has no training-framework, modality, or split-policy
dependency.

## Community

Read [CONTRIBUTING.md](CONTRIBUTING.md) before proposing changes. Use GitHub Issues for reproducible
bugs and design discussion, and follow [SECURITY.md](SECURITY.md) for private vulnerability reports.
Participation is governed by the [Code of Conduct](CODE_OF_CONDUCT.md).
