Metadata-Version: 2.4
Name: jsonlkit
Version: 0.2.0
Summary: Fast, zero-dependency toolkit for reading, writing and managing JSONL files
Author-email: Your Name <you@example.com>
License-Expression: MIT
Project-URL: Homepage, https://github.com/your-username/jsonlkit
Project-URL: Documentation, https://github.com/your-username/jsonlkit/blob/main/docs/USAGE.md
Project-URL: Changelog, https://github.com/your-username/jsonlkit/blob/main/CHANGELOG.md
Project-URL: Issues, https://github.com/your-username/jsonlkit/issues
Keywords: jsonl,ndjson,json-lines,streaming,cli
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Libraries
Classifier: Typing :: Typed
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
Dynamic: license-file

# jsonlkit

A small, **zero-dependency** toolkit for reading, writing and managing
[JSONL / NDJSON](https://jsonlines.org/) files. Pure standard library, Python 3.8+.

The standard `json` module only knows `loads` and `dumps`; it has no idea what a
JSONL *file* is. jsonlkit adds the file-level tools you would otherwise write
(and debug) by hand.

```python
import jsonlkit

jsonlkit.write("users.jsonl", [{"id": 1, "city": "Pune"}, {"id": 2, "city": "Mumbai"}])
for user in jsonlkit.read("users.jsonl"):        # streaming, flat memory
    print(user)
jsonlkit.sort_file("users.jsonl", "sorted.jsonl", "city")   # works on files bigger than RAM
jsonlkit.to_csv("users.jsonl", "users.csv")
```

## Install

```bash
pip install jsonlkit
```

New to JSONL? Read the
[step-by-step user guide](https://github.com/your-username/jsonlkit/blob/main/docs/USAGE.md),
whose examples are all run by the test suite. Version history is in the
[changelog](https://github.com/your-username/jsonlkit/blob/main/CHANGELOG.md).

## The basics

1. **Easy:** `read`, `write`, `append`, `head`, `tail`, `count`, `filter_file`, `map_file`,
   `dedupe`, `merge`, `split`, `validate`, plus a `jsonlkit` command-line tool.
2. **Memory-friendly:** `read()` is a generator, so a multi-GB file needs about 1 MB of RAM.
3. **Safe:** `write()` is atomic (temp file, then replace). A crash or a bad record never
   leaves a half-written file, and you can filter or sort a file in place. `append()`
   repairs a missing final newline. BOMs, `\r\n` and blank lines are handled.
4. **Clear errors:** a bad line raises `JSONLError` naming the file and line number
   (or skip bad lines with `skip_errors=True`).
5. **Gzip built in:** any path ending in `.gz` is compressed/decompressed automatically.

## 8 extras the `json` module does not have

**1. Dotted-path access: `get`.** Dig into nested data without a chain of `try/except`.
The same paths work as `key=` in `dedupe`, `sort_file`, `select` and `value_counts`.

```python
jsonlkit.get(rec, "user.address.city", "unknown")
jsonlkit.get(rec, "items.0.price")        # list index
jsonlkit.get(rec, "tags.-1")              # last item
```

**2. Pick and rename fields: `select`.**

```python
jsonlkit.select("in.jsonl", "out.jsonl", ["id", "user.email"])
jsonlkit.select("in.jsonl", "out.jsonl", {"id": "id", "email": "user.email"}, fill_missing=True)
```

**3. Flatten and unflatten nested records.**

```python
jsonlkit.flatten({"a": {"b": 1}, "tags": ["x", "y"]})   # {'a.b': 1, 'tags.0': 'x', 'tags.1': 'y'}
jsonlkit.unflatten({"a.b": 1, "tags.0": "x"})           # {'a': {'b': 1}, 'tags': ['x']}
jsonlkit.map_file("in.jsonl", "flat.jsonl", jsonlkit.flatten)   # whole file
```

**4. Sort files bigger than memory: `sort_file`.** External merge sort: holds at most
`chunk_records` records in RAM, is stable, and supports several keys.

```python
jsonlkit.sort_file("big.jsonl", "sorted.jsonl", "user.age", reverse=True)
jsonlkit.sort_file("big.jsonl", "sorted.jsonl", ["country", "name"], chunk_records=50_000)
```

**5. Random samples in one pass: `sample`.** Reservoir sampling; only the chosen lines
are parsed, so it is very fast on big files. Use `seed` for reproducible results.

```python
jsonlkit.sample("big.jsonl", 100, seed=42)
```

**6. Batches: `batched`.** For bulk database inserts and API calls.

```python
for batch in jsonlkit.batched("users.jsonl", 500):
    db.insert_many(batch)
```

**7. Convert formats:** `to_csv`, `from_csv`, `to_json`, `from_json`.
Nested data becomes dotted CSV columns and back. `from_json` reads a huge
`[ ... ]` array element by element, so it needs almost no memory.

```python
jsonlkit.to_csv("users.jsonl", "users.csv")                       # columns found automatically
jsonlkit.to_csv("users.jsonl", "users.csv", ["id", "user.email"]) # or choose them
jsonlkit.from_csv("users.csv", "users.jsonl", infer_types=True, nested=True)
jsonlkit.from_json("export.json", "export.jsonl")                 # JSON array -> JSONL
jsonlkit.to_json("export.jsonl", "export.json")
```

**8. Look inside a file: `schema` and `value_counts`.**

```python
jsonlkit.schema("events.jsonl")
# {'records': 1000, 'fields': {'id': {'count': 1000, 'types': {'int': 1000}},
#                              'user.name': {'count': 990, 'types': {'str': 985, 'null': 5}}, ...}}
jsonlkit.value_counts("events.jsonl", "status", top=5)   # [('ok', 940), ('error', 55), ...]
```

## Full API

| Function | What it does |
| --- | --- |
| `read(path, *, skip_errors=False)` | Lazily yield one record per line |
| `write(path, records)` | Atomically write records; returns the count |
| `append(path, records)` | Append records; creates the file if needed |
| `head(path, n=10)` / `tail(path, n=10)` | First / last `n` records |
| `count(path, *, skip_blank=False)` | Count lines without parsing (blank lines included unless `skip_blank=True`) |
| `filter_file(src, dst, predicate)` | Keep records where `predicate(record)` is true |
| `map_file(src, dst, func)` | Transform every record |
| `dedupe(src, dst, key=None)` | Drop duplicates (whole record, a dotted field path, or a function) |
| `merge(sources, dst)` | Concatenate several files |
| `split(src, out_dir, lines=100_000)` | Split into parts without parsing |
| `validate(path, *, max_errors=100)` | Return `(valid_count, [(line_no, message), ...])` |
| `get(record, path, default=None)` | Dotted-path lookup |
| `flatten(record)` / `unflatten(flat)` | Nested dict to dotted keys and back |
| `select(src, dst, fields, *, fill_missing=False)` | Keep or rename fields |
| `sort_file(src, dst, key, *, reverse=False, chunk_records=100_000)` | External merge sort |
| `sample(path, n, *, seed=None)` | Uniform random sample, file order |
| `batched(path, size)` | Yield lists of `size` records |
| `to_csv` / `from_csv` / `to_json` / `from_json` | Format conversion |
| `schema(path, *, max_records=None)` | Fields and types found in a file |
| `value_counts(path, field, *, top=None)` | Most common values of a field |

## Command line

```bash
jsonlkit head data.jsonl -n 5
jsonlkit tail data.jsonl
jsonlkit count data.jsonl
jsonlkit validate data.jsonl                 # exit code 1 if any line is invalid
jsonlkit dedupe in.jsonl out.jsonl --key user.id
jsonlkit merge all.jsonl a.jsonl b.jsonl
jsonlkit split big.jsonl parts/ --lines 50000
jsonlkit select in.jsonl out.jsonl id email=user.email
jsonlkit sort in.jsonl out.jsonl --key country name --reverse
jsonlkit sample big.jsonl -n 20 --seed 1
jsonlkit to-csv in.jsonl out.csv [--fields id user.email]
jsonlkit from-csv in.csv out.jsonl --infer-types --nested
jsonlkit to-json in.jsonl out.json
jsonlkit from-json in.json out.jsonl
jsonlkit schema data.jsonl
jsonlkit counts data.jsonl status --top 5
python -m jsonlkit count data.jsonl          # same thing
```

## Benchmarks

Measured with `benchmarks/bench.py` (Python 3.13, Linux, 300,000 lines, 35 MB,
best of 3). **Your numbers will differ; run the script yourself.**

| Task | By hand with `json` | jsonlkit | Speedup |
| --- | ---: | ---: | ---: |
| write | 702 ms | 696 ms | about the same |
| read all records | 1220 ms | 1093 ms | 1.12x |
| count, vs. parsing every line | 540 ms | 20 ms | 27x |
| count, vs. a plain line-counting loop | 25 ms | 20 ms | 1.2x |
| head(10), vs. load everything then slice | 1126 ms | 0.2 ms | different algorithm |
| head(10), vs. a loop with `break` | 0.2 ms | 0.2 ms | about the same |
| sample(100), vs. parse everything then sample | 1290 ms | 66 ms | 20x |
| sort by a field | 1969 ms | 2968 ms | **0.66x (slower)** |
| JSON array to JSONL | 1277 ms | 1314 ms | about the same |

| Peak memory | By hand | jsonlkit |
| --- | ---: | ---: |
| read 300k records (list vs. streaming) | 242 MB | 1 MB |
| sort 300k records | 246 MB | 51 MB |
| JSON array to JSONL | 196 MB | 14 MB |

How to read this honestly: parsing and encoding are done by the standard `json`
module, so raw speed is close to a hand-written loop. The wins are *not loading
whole files* (flat memory), *not parsing what you do not need* (`count`, `head`,
`sample`), and not writing the boilerplate yourself. `sort_file` trades about
1.5x more time for about 5x less memory; if the whole file fits in RAM and you
only care about speed, `list.sort` is faster. For the fastest possible parsing,
a C-based library such as `orjson` beats anything built on `json`.

## Notes and limits

- Records are written compactly (`{"a":1}`) with non-ASCII characters kept as-is, in UTF-8.
- `count()` counts every line, blank ones included, unless you pass `skip_blank=True`.
- `split()` and `count()` work on raw lines and do not validate JSON; use `validate()`.
  `sample()` only parses the lines it picks.
- `dedupe()` keeps the set of unique keys in memory. `sort_file()` keeps one chunk in
  memory and opens one temporary file per chunk, so raise `chunk_records` for very large inputs.
- `to_csv` needs JSON objects as records. CSV has no types, so use `infer_types=True`
  when reading it back (numbers with leading zeros stay strings).
- `flatten`/`unflatten`: keys that already contain `.` can collide with nested ones;
  pass another `sep`. A level whose keys are exactly `0..n-1` is rebuilt as a list.
- Appending to a `.gz` file creates a multi-member gzip, which standard tools read fine.

## Development

Tested on Python 3.8, 3.9, 3.10, 3.11, 3.12 and 3.13.

```bash
pip install -e .
python -m unittest discover -s tests -v
python benchmarks/bench.py
```

## License

MIT
