Metadata-Version: 2.4
Name: nfscache
Version: 0.1.2
Summary: Concurrency-safe, NFS-friendly Parquet cache for Polars DataFrames
Author: Torbjörn Sjögren
License-Expression: MIT
License-File: LICENSE
Keywords: cache,concurrency,dataframe,nfs,oracle,parquet,polars
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: POSIX
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Database
Classifier: Topic :: System :: Filesystems
Requires-Python: <3.14,>=3.13
Requires-Dist: numpy<3,>=2.4.6
Requires-Dist: polars<2,>=1.41.2
Requires-Dist: pyarrow<25,>=24.0.0
Provides-Extra: oracle
Requires-Dist: oracledb<4,>=3.4.2; extra == 'oracle'
Description-Content-Type: text/markdown

# nfscache

Prototype shared-filesystem cache for `DataContainer` objects whose payload is
a Polars `DataFrame`. The original target is NFS, and the locking now also
covers SMB/Windows shares (the lock-directory removal and `mkdir` retry paths
tolerate the racier removal semantics of SMB).

The cache stores container data as Parquet on a shared filesystem. Cold loads
can read from any slow source, for example Oracle, MySQL, or a local parquet
file. Warm loads use `polars.read_parquet`.

## Install

```bash
uv add nfscache
```

The Oracle SQL source path requires the `oracledb` driver, available as the
`oracle` extra:

```bash
uv add nfscache[oracle]
```

## Usage

Create an `NFSCache` pointed at a directory on the shared filesystem, then wrap
your cold-load function with a decorator. The wrapped function only runs on a
cache miss; warm hits are served from the Parquet cache.

The example below uses the Oracle SQL source path, so it needs the `oracle`
extra (`uv add nfscache[oracle]`). For a file-backed source that works with
the base install, use `@nfscache.parquet` (see
`nfscache/util/main.py`).

```python
from pathlib import Path

import oracledb
import pyarrow as pa
import pyarrow.parquet as pq

from nfscache.nfs_cache import NFSCache

nfscache = NFSCache(Path("__cache__/nfs"))
nfscache.connect_factory = lambda: oracledb.connect(
    user="SOMEUSER",
    password="cache",
    dsn="localhost:1521/FREEPDB1",
)


# SQL source streamed straight to the cache parquet file. On a warm hit, the
# function body is skipped and the validated cache file is copied to `filename`
# through an atomic `*.part` + `os.replace` export.
@nfscache.sql_parquet
def stream(sql: str, filename: Path, connection) -> None:
    table = pa.table({"example": [1, 2, 3]})  # replace with cursor.fetchmany()
    pq.write_table(table, filename)


with nfscache.connect_factory() as connection:
    stream(
        "select * from DATA_CONTAINER_DEMO",
        Path("DATA_CONTAINER_DEMO.parquet"),
        connection,
    )
```

For SQL sources, set `nfscache.connect_factory` (a `Callable[[], connection]`)
and use `@nfscache.sql_parquet` when the cold load should stream directly into
the cache parquet file. The first argument is the SQL string, and the second is
the requested output filename. The cache key is derived from the normalized SQL
and the source version from `MAX(ORA_ROWSCN)` plus the row count. See
`nfscache/database/oracle_streaming.py` for a complete Oracle streaming example.
Use `@nfscache.sql` instead when the cold load returns a materialized
`DataContainer`; see `nfscache/database/oracle_read.py`.

## Current Functionality

- Decorator API: `@nfscache.sql_parquet`, `@nfscache.sql`, and
  `@nfscache.parquet`.
- Stores `DataContainer.data.rows_data_pl` as a Parquet cache file, or streams
  SQL results directly into a cached Parquet file with `@nfscache.sql_parquet`.
- Reads cached objects with the fast Polars parquet reader.
- Writes cached objects with `pyarrow.parquet.ParquetWriter`.
- Writes through unique `*.part` files, then atomically replaces the final file
  with `os.replace`.
- Exports `@nfscache.sql_parquet` results by copying the validated cache file to
  an output `*.part` file, then atomically replacing the requested output path.
- Cleans up partial cache files on write failure.
- Uses a per-cache-key mkdir-based read/write lock: warm readers create
  per-reader tokens and can overlap, while writers and invalidations block new
  readers and wait for active readers to finish.
- Lock tokens include `lock.json` metadata with hostname, PID, UUID,
  `created_at`, and `last_seen`; held locks heartbeat `last_seen`, and stale
  reader/writer tokens are broken after `stale_lock_seconds`.
- The default stale lock timeout is 30 minutes, sized for cold Oracle reads that
  can take around 10 minutes while still heartbeating as live work.
- Adds an authoritative metadata sidecar:

```text
__cache__/nfs/parquet/A_TEST_1048576.parquet
__cache__/nfs/parquet/A_TEST_1048576.parquet.meta.json
```

- Metadata includes source key/version, parquet byte size, parquet SHA-256, row
  count, column count, schema hash, writer version, created time, and normalized
  `source_sql` for SQL-backed entries.
- Readers reject missing, stale, unsupported, or corrupt metadata and validate
  parquet size/checksum/row count/schema hash before returning a warm hit.
- Invalidates stale cache entries when the source version changes.
- For file path arguments, the default source version is a SHA-256 content hash.
- SQL sources use normalized SQL for cache keys and `COUNT(*)` plus
  `MAX(ORA_ROWSCN)` as the Oracle version token for the detected `FROM` table.
- Cold loads re-read the source version before and after loading and retry if
  the source changes during the read.

## Demo

From a source checkout, run:

```bash
uv run --no-cache --no-sync python -m nfscache.util.main
```

`main.py` runs:

1. clear `__cache__`
2. generate parquet source data
3. cold load and write cache
4. warm cache load
5. regenerate parquet source data
6. reload because the source hash changed
7. warm cache load again

Expected shape:

```text
Clearing cache: __cache__
Generating: parquet/A_TEST_1048576.parquet...
Reading: parquet/A_TEST_1048576.parquet...
Returning cached object: parquet/A_TEST_1048576.parquet sha=<first 40 chars>...
Generating: parquet/A_TEST_1048576.parquet...
Ignoring cache entry: parquet/A_TEST_1048576.parquet: stale source version
Reading: parquet/A_TEST_1048576.parquet...
Returning cached object: parquet/A_TEST_1048576.parquet sha=<first 40 chars>...
```

## Swarm Test

`swarm_file.py` tests a multi-client environment with process-level concurrency.
It mixes cache gets with source regeneration to simulate clients reading while
the source data changes.

Run the default swarm:

```bash
uv run --no-cache --no-sync python -m nfscache.util.swarm_file
```

Default behavior:

- 4 client processes
- 12 get waves
- 6 source regenerations
- generations are injected throughout the get waves
- final warm check after all waves complete

Useful smaller run:

```bash
uv run --no-cache --no-sync python -m nfscache.util.swarm_file \
  --clients 3 \
  --generators 1 \
  --gets-per-client 6 \
  --generations 3 \
  --n-rows 1024 \
  --cols 6 \
  --n-int-cols 2 \
  --n-str-cols 1 \
  --data-dir /tmp/parquet-nfs-wave-swarm-parquet \
  --cache-dir /tmp/parquet-nfs-wave-swarm-cache
```

Swarm output includes:

- source generation hash
- cold `Reading: ...` reloads after invalidation
- warm `Returning cached object: ... sha=...` hits
- final multi-client warm check

## SQL Swarm Test

`swarm_sql.py` tests the same process-level concurrency path for Oracle-backed
SQL reads. It creates an Oracle table, runs client reads through `@nfscache.sql`,
and rewrites the table between read waves so the cache has to invalidate and
reload under load.

Start Oracle first, then run:

```bash
uv run --no-cache --no-sync python -m nfscache.util.swarm_sql
```

Useful smaller run:

```bash
uv run --no-cache --no-sync python -m nfscache.util.swarm_sql \
  --clients 2 \
  --writers 1 \
  --gets-per-client 3 \
  --generations 2 \
  --n-rows 128 \
  --batch-size 64 \
  --table SWARM_SQL_TEST \
  --cache-dir /tmp/parquet-nfs-swarm-sql-cache
```

SQL swarm output includes Oracle cold reads, writer SCNs, stale SQL cache
invalidation, warm cache hits, and a final multi-client warm check.

## Tests

Run focused unit tests:

```bash
uv run --no-cache --no-sync python -m unittest discover -s tests
```

The tests cover authoritative metadata, corrupted metadata/parquet recovery,
normalized SQL metadata, overlapping warm readers, and writer-preference
locking. A syntax check for all modules:

```bash
uv run --no-cache --no-sync python -m compileall -q nfscache tests
```

## Generate Parquets

Generate or replace test parquet files:

```bash
uv run --no-cache --no-sync python -m nfscache.util.generate_parquets
```

The generator writes to a unique `*.part` file and atomically replaces the final
parquet when the write is complete.

By default, content changes on every run. Use `--seed` for reproducible data:

```bash
uv run --no-cache --no-sync python -m nfscache.util.generate_parquets --seed 123
```

## Oracle SQL Cache

The Oracle demos run from a source checkout and need the `oracledb` driver. Sync
the `oracle` extra into the environment first:

```bash
uv sync --extra oracle
```

Start the local Oracle demo container:

```bash
./build_and_run.sh [--wipe]
```

Populate the demo table:

```bash
uv run --no-cache --no-sync python -m nfscache.database.oracle_write_container
```

Stream Oracle SQL directly through the SQL Parquet cache:

```bash
uv run --no-cache --no-sync python -m nfscache.database.oracle_streaming \
  "select * from DATA_CONTAINER_DEMO" \
  DATA_CONTAINER_DEMO.parquet
```

Read into a materialized `DataContainer` through the SQL cache:

```bash
uv run --no-cache --no-sync python -m nfscache.database.oracle_read "select * from DATA_CONTAINER_DEMO"
```

SQL cache keys use normalized SQL plus requested columns. Metadata stores the
normalized `source_sql`, and source versions use `COUNT(*)` plus
`MAX(ORA_ROWSCN)` for the detected `FROM` table. `@nfscache.sql_parquet`
stores the streamed Parquet file in the cache first, then atomically exports a
copy to the requested output path.

## Production Notes

This is not yet production-grade enterprise software.

For Oracle on a shared filesystem (NFS, or an SMB/Windows share) with many
clients, the next important pieces are:

- validate `mkdir` lock tokens, writer intent, stale-lock recovery, and
  `os.replace` semantics on the actual mounts. The SMB/Windows code paths
  exist (resilient lock-dir removal, `mkdir` retry); what is missing is
  validation against a real SMB share and a mixed NFS-Linux + SMB-Windows
  client pool sharing one cache directory
- tie long Oracle reads to a documented consistent SCN/snapshot strategy
- add structured logs and metrics for hit/miss/reload, reader/writer lock wait,
  cold load duration, parquet write/read duration, and corruption/retry counts
- broaden automated failure tests for crashed lock holders, corrupted files,
  source changes during cold load, and multi-host / multi-protocol integration
  (NFS and SMB clients against one cache)
- add operational controls for cache retention, quotas, old `*.part` cleanup,
  version migration, compression, permissions, and bad-key runbooks
