Metadata-Version: 2.5
Name: datastore-provider
Version: 0.2.0
Summary: A Python/Polars provider for reading folders of data files (JSONL, CSV, Parquet) as tables.
Project-URL: Homepage, https://github.com/TheRAFLab/datastore_provider
Project-URL: Repository, https://github.com/TheRAFLab/datastore_provider
Project-URL: Issues, https://github.com/TheRAFLab/datastore_provider/issues
Author: The RAF Lab
License-Expression: MIT
License-File: LICENSE
Keywords: csv,dataframe,datastore,jsonl,parquet,polars
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Database
Classifier: Topic :: Scientific/Engineering :: Bio-Informatics
Requires-Python: >=3.10
Requires-Dist: polars>=1.43.2
Description-Content-Type: text/markdown

# Datastore Provider

A small Python/[Polars](https://pola.rs) provider for reading folders of data files (JSONL, CSV, Parquet) as tables.

A "datastore" here is just a directory tree: each table is a subdirectory, and each table is made up of one or more sharded files in a single format. `datastore_provider` maps a JSON config onto that layout and hands you a Polars `LazyFrame` (or `DataFrame`) per table, so you can query across the shards without caring how many files there are or where they live.

```
<DATASTORE_ROOT>/
├── <table_name>/           <- one directory per table
│   ├── <shard>.jsonl       <- one or more files, all in the table's format
│   ├── <shard>.jsonl
│   └── ...
└── <other_table>/
    ├── <shard>.parquet
    └── ...
```

## Installation

With [uv](https://docs.astral.sh/uv/):

```bash
uv add datastore-provider
```

Or with pip:

```bash
pip install datastore-provider
```

Requires Python 3.10 or newer. The only dependency is [Polars](https://pola.rs).

To work on the library itself rather than install it, see
[Development](#development).

## Configuration

Configuration is a plain JSON document (or any `dict`). The minimum is a datastore root and a list of tables:

```json
{
    "DATASTORE_ROOT": "/path/to/datastore",
    "TABLES": [
        { "name": "samples", "format": "jsonl" },
        { "name": "measurements", "format": "csv" },
        { "name": "annotations", "format": "parquet" }
    ]
}
```

A table can also declare the schema it should be read with — see
[Schemas](#schemas):

```json
{
    "name": "records",
    "format": "jsonl",
    "schema": {
        "id": "Int64",
        "cdr3": "String",
        "aliases": "List(String)"
    }
}
```

| Key | Required | Description |
| --- | --- | --- |
| `DATASTORE_ROOT` | yes | Base path for the datastore. Local path or remote URI (e.g. `s3://my-bucket/datastore`). |
| `TABLES` | yes | List of table definitions. |
| `TABLES[].name` | yes | Table name, and the name of its subdirectory under `DATASTORE_ROOT`. |
| `TABLES[].format` | yes | One of `jsonl`, `csv`, `parquet`. All files in the table must share this format. |
| `TABLES[].schema` | no | Columns and dtypes to read the table with, as `{"column": "Dtype"}`. See [Schemas](#schemas). |
| `S3_ENABLED` | no | Set `true` to pass S3 credentials through to Polars. Defaults to `false`. |
| `AWS_ACCESS_KEY_ID` | no | Used when `S3_ENABLED` is true. |
| `AWS_SECRET_ACCESS_KEY` | no | Used when `S3_ENABLED` is true. |
| `AWS_REGION` | no | Used when `S3_ENABLED` is true. Defaults to `us-east-1`. |

Files are discovered by globbing `{DATASTORE_ROOT}/{name}/*.{format}`, so adding a shard to a table is just a matter of dropping a file into the directory.

## Usage

```python
from datastore_provider import DatastoreProvider
from datastore_provider.helpers import load_config

config = load_config("config.json")
provider = DatastoreProvider(config)

# Lazy by default — nothing is read until you collect
samples = provider.load_table("samples").collect()

# Or read eagerly
annotations = provider.load_table("annotations", mode="eager")
```

The config can equally be built in code, without a file on disk:

```python
provider = DatastoreProvider({
    "DATASTORE_ROOT": "/path/to/datastore",
    "TABLES": [{"name": "samples", "format": "jsonl"}],
})
```

Because lazy mode returns a Polars `LazyFrame`, predicates and projections are pushed down into the scan — only the columns and rows you ask for are read off disk:

```python
import polars as pl

subset = (
    provider.load_table("samples")
    .filter(pl.col("category") == "reference")
    .select("id", "label")
    .collect()
)
```

Every row carries a `source_file` column recording which file it came from, which is useful when a table is sharded per-sample or per-experiment.

## Schemas

By default Polars infers a table's schema from the files it scans. A table can
instead declare its columns and dtypes, either in the config or at the call
site:

```python
# In the config, so every caller gets the same read
provider = DatastoreProvider({
    "DATASTORE_ROOT": "/path/to/datastore",
    "TABLES": [{
        "name": "records",
        "format": "jsonl",
        "schema": {"id": "Int64", "cdr3": "String", "aliases": "List(String)"},
    }],
})

# Or per call, which wins over the configured schema
records = provider.load_table("records", schema={"id": pl.Int64, "cdr3": pl.String})
```

Configured schemas are resolved when the provider is built, so a dtype typo
fails there rather than on first scan, and the resolved schemas are available
as `provider.schemas`.

### Why a sharded jsonl table usually needs one

**A jsonl table sharded one record per file effectively requires an explicit
schema.** Polars infers a schema from the first files it globs, so with one
record per file it only ever sees a single record. Any field that is `null` or
`[]` in that record is inferred as `Null` or `List(Null)`, and the scan then
fails on the first shard that carries a real value:

```
ComputeError: got non-null value for NULL-typed column: DSAIYN
```

`infer_schema_length` cannot help, because there is nothing more in that first
file for Polars to look at. The failure also depends on which shard happens to
be globbed first, so a table can read cleanly for months and then break when a
sparse record is added — declare the schema and it cannot happen.

### Writing dtypes

A dtype is either a Polars dtype, when the schema is built in code, or the name
of one, so a schema can live in a JSON config. Names are matched
case-insensitively, both as Polars spells them and as it prints them:

| Written as | Resolves to |
| --- | --- |
| `"Int64"`, `"i64"` | `pl.Int64` |
| `"String"`, `"str"`, `"Utf8"` | `pl.String` |
| `"Boolean"`, `"bool"` | `pl.Boolean` |
| `"List(String)"`, `"list[str]"` | `pl.List(pl.String)` |

Lists nest to any depth. Anything Polars can express but a name cannot — a
struct, a fixed-size array, an enum — has to be passed to `load_table()` as a
real dtype.

The `source_file` column is added by the provider, so a schema must not declare
it; doing so raises `ValueError`.

### What a schema means

The schema is handed straight to the Polars scanner, so its exact meaning is
the scanner's:

- **jsonl** — authoritative. Columns the schema leaves out are not read, and
  columns the data lacks come back null.
- **csv** — must match the number of columns in the file, and sets their
  dtypes.
- **parquet** — validates and casts against the file's own schema, so it is a
  check rather than a necessity.

## API

### `DatastoreProvider(config: dict)`

Builds a provider from a config dictionary. If `S3_ENABLED` is true, AWS credentials are collected into the `storage_options` passed to every Polars scan; otherwise `storage_options` is `None`. Any schemas declared by tables are resolved to Polars dtypes and exposed as `provider.schemas`, keyed by table name.

The configuration is validated up front, so typos fail here rather than on first use. Raises `ValueError` if `DATASTORE_ROOT` is unset, if `TABLES` is not a list, or if any table definition is not a dict, is missing `name` or `format`, declares an unsupported format, or declares a schema that cannot be resolved.

### `load_table(table_name: str, mode: str = "lazy", schema: Mapping | None = None)`

Scans every file belonging to `table_name` and returns a Polars `LazyFrame` (`mode="lazy"`) or `DataFrame` (`mode="eager"`). `mode` is typed as a `Literal`, so a typo is caught by a type checker before it runs.

`schema` maps column names to dtypes, and overrides the table's configured schema. When it is `None` the table's configured schema is used, and failing that Polars infers one. See [Schemas](#schemas).

Raises `ValueError` if the table is not in the config, if `mode` is not `lazy` or `eager`, or if the schema cannot be resolved. Table formats and configured schemas are validated when the provider is built.

### `build_table_path(table_name: str) -> str`

Returns `{DATASTORE_ROOT}/{table_name}`. Raises `ValueError` if the table is unknown or `DATASTORE_ROOT` is missing.

### `build_glob_string(table_name: str, format: str) -> str`

Returns the glob used to find the table's files: `{DATASTORE_ROOT}/{table_name}/*.{format}`.

### `helpers.load_config(config_path: str) -> dict`

Reads and parses a JSON config file.

## Development

```bash
git clone https://github.com/TheRAFLab/datastore_provider.git
cd datastore_provider
uv sync
uv run pytest
uv run ruff check .
```

The test suite builds a throwaway datastore under `tmp_path` covering all three
formats, so it needs no fixture data on disk. CI runs the same two commands
against Python 3.10 through 3.14.

## License

MIT — see [LICENSE](LICENSE).
