Metadata-Version: 2.4
Name: dirigent-parquet
Version: 0.16.3
Summary: The parquet format pack for dirigent: the convert.arrow codec, on pyarrow.
License-Expression: LicenseRef-Proprietary
License-File: LICENSE
Requires-Dist: dirigent-common==0.16.3
Requires-Dist: dirigent-plugin==0.16.3
Requires-Dist: pyarrow>=21.0.0
Requires-Python: >=3.13
Description-Content-Type: text/markdown

# dirigent-parquet

The parquet format pack for dirigent: one codec, `convert.arrow`, trading parquet with the
three text spellings `convert.std` already trades between.

| From | To | What it does |
| --- | --- | --- |
| `parquet` | `json`, `ndjson`, `csv` | Each row becomes one record, every value in its JSON spelling. |
| `json`, `ndjson`, `csv` | `parquet` | Each record becomes one row, under a schema inferred from the records. |

The record model is `convert.std`'s: a sequence of flat records. What parquet adds is
types, and the codec is honest about them in both directions:

- **Writing** infers one type per column -- boolean, int64, float64, or string -- from the
  records. Integers and floats unify to a float column, because JSON calls both a number;
  any other mix is refused naming the column. A column every row leaves null is written as
  a nullable string column. A csv source carries no types, so csv to parquet writes string
  columns and nothing else.
- **Reading** gives every value its JSON spelling: timestamps, dates and times come back as
  ISO strings, decimals as strings, and a float that is NaN or infinite as null. A nested
  column, or one holding raw bytes, is refused naming it -- flattening is a reshape, and a
  reshape belongs to a jq step that knows what the flattening should mean.

A conversion is a storage-object operation, like `storage.copy`: it reads the object at
`source` and writes the one at `target`, and neither side is ever a value the step carries.
A run that means to look at the records reads them with `storage.read`.

The pack ships separately because `convert.std` is deliberately on the standard library
and nothing else; this codec stands on [pyarrow](https://arrow.apache.org/docs/python/).
Nothing else in the workspace may depend on it.
