Metadata-Version: 2.5
Name: fixedwidthparser
Version: 2.0.0
Summary: Read and write fixed-width files through the FixedWidthParser NativeAOT library
Project-URL: Homepage, https://github.com/GabrielMarquezMatte/FixedWidthParser
Author: Gabriel Matte
License: MIT
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Requires-Python: >=3.10
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == 'dev'
Provides-Extra: numpy
Requires-Dist: numpy>=1.21; extra == 'numpy'
Provides-Extra: pandas
Requires-Dist: pandas>=2.0; extra == 'pandas'
Requires-Dist: pyarrow>=14.0; extra == 'pandas'
Provides-Extra: polars
Requires-Dist: polars>=1.0; extra == 'polars'
Provides-Extra: pyarrow
Requires-Dist: pyarrow>=14.0; extra == 'pyarrow'
Description-Content-Type: text/markdown

# fixedwidthparser

Read and write fixed-width files from Python through FixedWidthParser's NativeAOT library: no .NET runtime,
parallel reads, results as NumPy views, Arrow, pandas or polars.

```bash
pip install "fixedwidthparser[numpy]"    # fw.read: NumPy views
pip install "fixedwidthparser[pandas]"    # or [pyarrow], [polars]
```

Wheels ship for Windows x64, macOS (arm64, x64) and Linux (glibc and musl, x64 and arm64). The package installs
as `fixedwidthparser` and imports as `fixedwidth`. pyarrow has no musllinux wheels, so on Alpine and other
musl systems Arrow, pandas, polars and writing need a pyarrow built for them.

```python
import fixedwidth as fw

# B3's COTAHIST quote history: 245-byte lines, a header (00) and a trailer (99) around the quotes (01).
layout = fw.Layout([
    fw.Column("tipreg", 0, 2, fw.String),
    fw.Column("data_pregao", 2, 8, fw.Date, format="yyyyMMdd"),
    fw.Column("codneg", 12, 12, fw.String),
    fw.Column("preabe", 56, 13, fw.Float64, scale=2),   # N(11)V99: two implied decimals
    fw.Column("quatot", 152, 18, fw.Int64),
])
options = fw.ReadOptions(header_lines=1, skip_invalid=True)   # the trailer fails to parse and is skipped

df = fw.read_pandas("COTAHIST_A2023.TXT", layout, options)
```

## Reading

| Function | Returns |
|---|---|
| `fw.read(source, layout, options)` | `fw.Table`: NumPy views over the native columns, no copy |
| `fw.read_arrow(source, layout, options)` | `pyarrow.Table` (needs pyarrow) |
| `fw.read_pandas(source, layout, options)` | `pandas.DataFrame` (needs pyarrow and pandas) |
| `fw.read_polars(source, layout, options)` | `polars.DataFrame` |
| `fw.read_batches(source, layout, fw.ReaderOptions(batch_rows=...))` | an iterator of `fw.Table` |
| `fw.read_arrow_stream(source, layout, fw.ReaderOptions(...))` | an Arrow PyCapsule stream for `polars.DataFrame(...)` or `pyarrow.RecordBatchReader.from_stream(...)` |

`source` is a path, `bytes`/`bytearray`/`memoryview`/`io.BytesIO` to read from memory (a `BytesIO` in place,
from its position), or any stream with `readinto()` or `read()`: an open file, a pipe, `gzip.open(...)`,
`zipfile.ZipFile(...).open(...)`. `read_batches` and `read_arrow_stream` pull from a stream as the batches are
taken, in constant memory; the whole-table reads read it into memory first, since they split it into byte ranges.

```python
import zipfile

with zipfile.ZipFile("COTAHIST_A2023.ZIP") as archive, archive.open("COTAHIST_A2023.TXT") as stream:
    for batch in fw.read_batches(stream, layout, fw.ReaderOptions(read=options, batch_rows=100_000)):
        ...
```

Every optional setting lives in
`fw.ReadOptions`: `header_lines`, `skip_invalid`, `degree_of_parallelism` (0 = one thread per processor,
1 = sequential), `encoding` (a code page or `"latin-1"`, `"cp1252"`, ...), `max_line_length` and
`skip_byte_order_mark`. A parallel read gives the same rows, in the same order, as a sequential one.

A line that fails to parse raises `fw.ParseError` with `line_number` and `column`, unless `skip_invalid` is set.

`fw.Table` maps a column name to a NumPy array (a list for strings), and has `num_rows`, `column_names`,
`lines_read`, `lines_skipped`, `validity(name)` (a bool array, `True` where the row holds a value, or `None`
when the column has no nulls) and `to_arrow()`, which shares the native buffers. Without pyarrow,
`read_polars` reads sequentially through the Arrow stream.

## Threads and asyncio

Every function releases the GIL for its native work, so reads and writes on several threads, or through
`asyncio.to_thread`, run in parallel:

```python
frames = await asyncio.gather(*(asyncio.to_thread(fw.read_polars, path, layout, options) for path in paths))
```

- A `fw.Layout` is immutable and can be shared by every thread at once.
- A failure is raised on the thread whose call failed, with that call's message and line number.
- A `read_batches` iterator, and the stream it reads, belong to one thread at a time.
- A stream's `read()` is called on the thread doing the read and takes the GIL while it runs; Python's files,
  sockets and `zlib` release it again while they wait or decompress, so other threads keep going.
- `degree_of_parallelism=0` gives each call one thread per processor. When many calls already run at once, pass
  `1`: the calls are the parallelism, and splitting each one too only makes them queue for the same cores.

## Columns

`fw.Column(name, start, length, type, *, nullable, scale, format, trim_char, trim_mode, alignment, padding, overflow, dictionary)`.
Positions are bytes. Types: `fw.String`, `fw.Int64`, `fw.Float64`, `fw.Decimal` (int64 unscaled, Arrow
`decimal128(18, scale)`), `fw.Date`, `fw.Time`, `fw.Timestamp`. `scale` is the number of implied decimals:
`0000000123456` with `scale=2` reads as `1234.56`. `format` uses .NET date syntax (`yyyyMMdd`, `HHmmss`).
`trim_mode`, `alignment` and `overflow` take `fw.TrimMode`, `fw.Alignment` and `fw.Overflow`.

`dictionary=True` on a `fw.String` column hands it to Arrow dictionary-encoded, so `read_pandas` gives a
`category` column and `read_polars` a `Categorical` one: for a column of few distinct values (a market code, a
currency, a record type) far less memory, and faster group-bys. The encoding costs a pass over the column, about
3 ms per million rows per column, run in parallel across columns. `fw.read`'s Table keeps plain strings, and its
`to_arrow()` encodes the same columns. Categorical columns write like strings.

`fw.Layout(columns, fw.LayoutOptions(line_length=..., filler=...))` sets how written lines are laid out:
`line_length` past the last column pads each line, and `filler` (default a space) fills the gaps between
columns.

## Writing

```python
fw.write(df, "out.txt", layout, fw.WriteOptions(degree_of_parallelism=0))
data = fw.write_bytes(df, layout)

with gzip.open("out.txt.gz", "wb") as stream:   # or an open file, a socket's makefile("wb"), ...
    fw.write(df, stream, layout)
```

`fw.write` takes a path or a binary stream with `write()`. A stream gets the lines in order, a batch per call on
the calling thread, while the next batches format in parallel, so memory stays at a batch per thread; writing
to an open file is as fast as writing to its path. For the bytes themselves, `write_bytes` beats an `io.BytesIO`,
which copies itself each time it grows.

Takes pandas, polars, pyarrow or anything with `__arrow_c_stream__` (needs pyarrow), matching columns by name.
`padding="0"` writes zero-padded numbers with the sign leftmost; a null writes as its column filled with
`trim_char`. Parallel writes are byte-identical to serial ones. `fw.WriteOptions` also takes `encoding`,
`newline` (default `"\n"`) and `batch_rows`.

A `fw.Decimal` column stores its value times `10**scale` as an int64. Decimals and integers are scaled
exactly, and a decimal with more digits than `scale` raises. Floats round half to even to `scale`.
A value that is NaN, infinite or outside int64 once scaled raises before anything is written.

## Building from source

    python python/scripts/build_native.py      # requires the .NET 10 SDK; publishes the library into src/fixedwidth/_lib
    pip install -e "python[dev,pandas,polars]"
    pytest python/tests

To point at a library built elsewhere, set `FIXEDWIDTH_NATIVE_LIB` to its full path.
