Metadata-Version: 2.5
Name: parallelogram
Version: 1.5.1
Summary: Strict validator for fine-tuning datasets. Run before you train.
Project-URL: Homepage, https://parallelogram.dev
Project-URL: Source, https://github.com/Thatayotlhe04/Parallelogram
Project-URL: Documentation, https://github.com/Thatayotlhe04/Parallelogram/blob/main/cli/docs/agents.md
Project-URL: Changelog, https://github.com/Thatayotlhe04/Parallelogram/blob/main/cli/CHANGELOG.md
Project-URL: Issues, https://github.com/Thatayotlhe04/Parallelogram/issues
Author-email: Thatayotlhe Tsenang <tsenangthatayotlhe04@gmail.com>
Maintainer-email: Thatayotlhe Tsenang <tsenangthatayotlhe04@gmail.com>
License: Apache-2.0
License-File: LICENSE
Keywords: dataset,fine-tuning,linter,llm,validation
Classifier: Development Status :: 5 - Production/Stable
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Quality Assurance
Requires-Python: >=3.10
Requires-Dist: rich>=13.7
Requires-Dist: typer>=0.12
Provides-Extra: dev
Requires-Dist: pytest>=8; extra == 'dev'
Requires-Dist: ruff>=0.5; extra == 'dev'
Provides-Extra: tokenizer
Requires-Dist: tiktoken>=0.7; extra == 'tokenizer'
Requires-Dist: tokenizers>=0.15; extra == 'tokenizer'
Description-Content-Type: text/markdown

# parallelogram

> Strict validator for fine-tuning datasets. Run it before you train.

Every fine-tuning framework assumes your data is clean. None of them verify it.
Axolotl will start a run on malformed data and either crash mid-way or — worse —
complete silently while producing a broken model. TRL will truncate samples that
exceed the context window without telling you. Unsloth will train on duplicates
that cause your model to memorize instead of generalize.

`parallelogram` sits between your raw dataset and your training run. It hard-blocks
on anything that would silently corrupt training. **If it exits 0 with all rules enabled,
your run won't fail because of data.**

## Agents and Python (1.5)

```bash
parallelogram check train.jsonl --json --limit 50
parallelogram rules --json
parallelogram mcp
```

```python
from parallelogram import check, fix, rules

result = check("train.jsonl", limit=50)
payload = result.to_dict()
status = result.exit_code
fixed = fix("train.jsonl", output="clean.jsonl")
```

The API and CLI share their check/fix implementation. JSON retains
`schema_version: 1`, with exact `by_rule` counts and `issues_truncated`; limiting
issues never limits validation. Fixes require a separate output path and refuse
input/eval aliases. [Agent guide](https://github.com/Thatayotlhe04/Parallelogram/blob/main/cli/docs/agents.md): MCP tools, Claude Code/Codex/Cursor
configuration, Python errors, and a six-line AGENTS.md/CLAUDE.md snippet.

## Install

```bash
pip install parallelogram
```

For an **exact** context-window count, install the tokenizer extras — HuggingFace
`tokenizers` for open-weight models and `tiktoken` for OpenAI models:

```bash
pip install 'parallelogram[tokenizer]'
```

Without the extras (or for a model with no offline tokenizer, like Claude), the
context-window check still runs using an approximate length-based count.

## Use

```bash
parallelogram check data.jsonl
```

With a model-specific tokenizer for an exact context-window count — an OpenAI model,
or any HuggingFace repo or short alias (`mistral`, `qwen`, `llama-3`, …):

```bash
parallelogram check data.jsonl \
  --tokenizer Qwen/Qwen2.5-7B \
  --max-seq-len 8192
```

Omit `--tokenizer` and the check still runs with an approximate count, reported as
warnings instead of errors.

Write only the clean records to a new file:

```bash
parallelogram check data.jsonl --output clean.jsonl
```

OpenAI/Qwen chat JSONL (`messages` with `role`/`content`) and ShareGPT-style
datasets are accepted by default. ShareGPT-style records use
`{"conversations": [{"from": ..., "value": ...}, ...]}`; every rule runs
identically, and `--output` (with or without `--fix`) writes the surviving
records back in the source shape:

```bash
parallelogram check data.jsonl
```

Use `--format openai-chat`, `--format sharegpt`, or `--format prompt-completion` to force
one parser instead of auto-detection.

### Check Alpaca and preference (DPO) data

Alpaca records (`instruction`, optional `input`, `output`, plus LLaMA-Factory's
`system` and `history`) and preference pairs (`prompt`, `chosen`, `rejected`,
or TRL's implicit-prompt shape where `chosen` and `rejected` are full
conversations) are auto-detected too. Every rule checks both sides of a
preference pair:

```bash
parallelogram check dpo.jsonl
parallelogram check alpaca.jsonl --fix --output clean.jsonl   # written back as Alpaca
```

### Catch train/eval leakage

```bash
parallelogram check train.jsonl --eval eval.jsonl
```

Flags train records whose whole content, or whose prompt, also appears in the
eval set. `--fix` drops them from the train output; the eval file is never touched.
Eval records are auto-detected independently of `--format`. Matching trims message
edges but preserves case and internal whitespace. An empty eval file is allowed;
invalid JSON, invalid schema, or a read failure aborts with exit 3 and a stderr
diagnostic, before any output dataset is written.

### Check prompt/completion data

`prompt-completion` is auto-detected alongside OpenAI/Qwen chat and ShareGPT,
including files that mix these formats. Both string and conversational pairs
are accepted; either half may independently be a string or message list:

```json
{"prompt": "What is 2 + 2?", "completion": "4."}
{"prompt": [{"role": "user", "content": "Capital of Botswana?"}], "completion": [{"role": "assistant", "content": "Gaborone."}]}
```

```bash
parallelogram check prompt_completion.jsonl
parallelogram check prompt_completion.jsonl --format prompt-completion
parallelogram check prompt_completion.jsonl --fix --output repaired.jsonl
```

Missing halves and values other than strings/lists are schema errors. Fixed
output keeps standard pairs as strings and conversational pairs as lists.
Conversational completion is the trailing run of assistant messages, with earlier
turns in prompt. A mixed record keeps a string half when it still contains one
matching turn; otherwise that half becomes a list to preserve every message. Extra top-level keys
may be omitted by fixes.

## `--fix` — mechanical repair

When `parallelogram check` finds errors, `--fix` attempts to repair what it can without
touching the model. Mechanical fixes are free, local, and require no network call.

```bash
parallelogram check data.jsonl --fix --output clean.jsonl
```

Fixes applied in order:

1. **encoding** — strip BOM markers, replace mojibake (`donâ€™t` → `don't`)
2. **empty-content** — drop empty/whitespace-only message turns
3. **context-window** — truncate the longest user message until the record fits
4. **duplicates** — keep the first occurrence, drop subsequent

After fixes are applied, the dataset is **re-validated**. Anything still erroring is
dropped from the output. The CLI tells you exactly what was unchanged, fixed, and
dropped:

```
✓ encoding · 4 fixes
✓ duplicates · 12 fixes

✗ dropped:
    data.jsonl:23 → roles (unfixable)
    data.jsonl:147 → schema (unfixable)

547 records  531 unchanged  4 fixed  11 dropped  1 unparseable
```

Errors that need understanding the content (broken role sequences, incomplete
assistant turns) are not fixable mechanically. These will be addressed by a hosted
SLM tier in a future release. For now, they are dropped.

Use `--dry-run` to preview without writing:

```bash
parallelogram check data.jsonl --fix --dry-run
```

## `report` — dataset health for humans and CI

`check` answers "what is wrong, line by line". `report` answers "how healthy is
this dataset overall, and is it getting better or worse":

```bash
parallelogram report data.jsonl --tokenizer gpt-4o --max-seq-len 4096
```

One run prints: clean/error/warning record counts, issues by rule with fixable
counts, what `--fix` would emit/drop, token risk (records over budget and within
85% of it, labeled **exact** or **estimated**), duplicate clusters, and the shape
of the data (role counts, turns per record, conversations ending on assistant).

Three output modes:

```bash
parallelogram report data.jsonl              # pretty terminal
parallelogram report data.jsonl --json       # machine-readable (also the baseline format)
parallelogram report data.jsonl --markdown   # GitHub-flavored, for $GITHUB_STEP_SUMMARY
```

`--out PATH` additionally writes the report to a file.

### Fail a PR when dataset quality regresses

Save a baseline from your main branch, then gate PRs against it:

```bash
parallelogram report data.jsonl --json --out baseline.json   # on main
parallelogram report data.jsonl --baseline baseline.json     # on the PR
```

If quality regressed, the command exits **3** and lists exactly what got worse.
Comparison is **rate-based** (errors per record, duplicates per record, records
over token budget per record, fraction dropped by `--fix`, clean fraction), so a
dataset that grows is never punished for having more records — only for getting
proportionally worse.

```yaml
# .github/workflows/data.yml
- run: pip install 'parallelogram[tokenizer]'

# human-readable summary on the Actions run page
- run: parallelogram report data/train.jsonl --tokenizer mistral --max-seq-len 32768 --markdown >> "$GITHUB_STEP_SUMMARY"

# hard gates: fail on errors, and fail if quality regressed vs main
- run: parallelogram check data/train.jsonl --tokenizer mistral --max-seq-len 32768
- run: parallelogram report data/train.jsonl --tokenizer mistral --max-seq-len 32768 --baseline baseline/report.json
```

## GitHub Action

Validate a dataset and add a Markdown health report to the workflow summary:

```yaml
name: Validate training data
on: [push, pull_request]
permissions:
  contents: read
jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: Thatayotlhe04/Parallelogram@v1
        with:
          path: data/train.jsonl
          format: auto
          max-seq-len: '4096'
          fail-on: error
          # tokenizer: Qwen/Qwen2.5-7B
          # baseline: baseline/report.json
          # eval: data/eval.jsonl
          # python-version: '3.12'
```

`@v1` tracks the 1.x releases. The action installs the CLI from its own checkout, so
its version matches the selected ref. Setting `tokenizer` installs the tokenizer
extras. The required `path` points to your checked-out dataset; all other inputs
are optional, with defaults shown above (tokenizer and baseline default to empty).
Errors (exit 2), invalid eval inputs and baseline regressions (exit 3) always fail. Warning-only reports
(exit 1) pass by default; use `fail-on: warning` to fail on them too. The report is
appended to the job summary even when validation fails.

## Stability

From 1.0 until 2.0, command names and flags, rule ids, exit codes 0/1/2/3,
and JSON fields at `schema_version: 1` are stable. Removing or renaming any of
these requires a major release. New rules and formats may be added in minor
releases. See the [JSON output contract](https://github.com/Thatayotlhe04/Parallelogram/blob/main/cli/docs/json-output.md) for every field and its meaning.
Reports saved by 0.4.x remain usable as baselines.

## `--disable` and the exit-0 guarantee

Rules can be disabled by id (e.g. `--disable encoding`), but with three constraints:

- The `schema` rule cannot be disabled. Every other rule depends on its structural
  guarantees, and disabling it would let other rules silently no-op on malformed records.
- Unknown rule ids are rejected. Typos like `--disable encding` exit non-zero with a
  list of valid options rather than silently doing nothing.
- Whenever any rule is disabled, a loud stderr warning names exactly which ones, and
  reminds you that the exit-0 guarantee no longer applies. The terminal output and
  JSON report (`disabled_rules` field) both surface this so CI tooling can refuse to
  merge a PR that disabled rules.

The guarantee is precise: **a clean exit with all rules enabled** means your run won't
fail because of data. A clean exit with rules disabled means only that the rules you
left enabled passed — which may or may not be enough.

## Options

| Flag | Description |
|------|-------------|
| `--version` | Show the CLI version and exit (use before a command). |
| `--format`, `-f` | Dataset format: `auto` (default), `openai-chat`, `sharegpt`, `prompt-completion`, `alpaca`, or `preference`. |
| `--eval` | Eval set to check for train/eval overlap (`check` and `report`). |
| `--tokenizer`, `-t` | Model or tokenizer for the context-window check — an OpenAI model (`gpt-4o`), or an HF repo/alias (`Qwen/Qwen2.5-7B`, `mistral`). Optional: omit for an approximate count. |
| `--max-seq-len` | Token budget per record (default 4096). |
| `--output`, `-o` | Write error-free records to this file. With `--fix`, writes the repaired dataset. |
| `--fix` | Attempt mechanical repair of fixable issues. |
| `--dry-run` | With `--fix`, report what would change without writing. |
| `--json` | Machine-readable report on stdout. |
| `--limit` | Maximum check JSON issues to list; all counts remain exact. |
| `--disable` | Disable a rule by id. Repeatable. |
| `--no-color` | Plain output. |

## Exit codes

| Code | check | check --fix | report |
|------|-------|-------------|--------|
| `0`  | Clean. | All records emitted clean. | Clean. |
| `1`  | Warnings only. | Some records dropped (partial fix). | Warnings only. |
| `2`  | Errors. | Nothing fixable. | Errors. |
| `3`  | Operational failure, such as an unreadable or invalid `--eval` file. | Operational failure, such as an unreadable or invalid `--eval` file. | Invalid `--eval` input, or quality regressed vs `--baseline`. |

These are stable and map directly to CI gates without any extra wiring.
Informational notes (like "context-window counts are approximate" when running
without a tokenizer) never affect the exit code — a clean dataset exits 0 on a
default install.

## Rules

| id | severity | catches |
|----|----------|---------|
| `schema` | error | malformed records, missing fields, wrong types |
| `roles` | error | bad role sequences (system out of place, no alternation, doesn't end on assistant) |
| `empty-content` | error | empty or whitespace-only message content |
| `context-window` | error / warning | records exceeding `max_seq_len` (TRL truncates these silently) — error with an exact tokenizer, warning when the count is approximate |
| `duplicates` | error | exact-content duplicate records (memorization → poor generalization) |
| `encoding` | warning | BOM markers, mojibake patterns |
| `secrets` | error | API keys, tokens and private keys (OpenAI, Anthropic, AWS, GitHub, Hugging Face, Slack, Google, Stripe). `--fix` redacts them; matches are never printed in full |
| `pii` | warning | email addresses, phone numbers, Luhn-valid card numbers. Example domains and 555 numbers are ignored; `--fix` replaces them with placeholders |
| `preference` | error | preference (DPO) pairs whose chosen and rejected answers are identical |
| `eval-overlap` | error | with `--eval`: train records whose content or prompt also appears in the eval set. `--fix` drops them |

## Status

v1.5.1 — stable CLI and Python API, agent-sized JSON output, rule discovery and a local MCP server. No telemetry, no network, no upload boundary.

## Roadmap

- ~~`--fix` mechanical tier (dedupe, truncate, normalize encoding)~~ ✓ shipped in v0.2
- ~~Model-specific tokenizers (tiktoken/HF) with approximate fallback~~ ✓ shipped in v0.3
- ~~ShareGPT format (`{"conversations": [...]}`)~~ ✓ shipped in v0.4
- ~~`report` command + CI regression gate (`--baseline`, exit 3)~~ ✓ shipped in v0.4.1
- ~~Direct ShareGPT-style auto-detection~~ ✓ shipped in v0.4.2
- ~~Prompt/completion format~~ shipped in v1.0

## License

Apache-2.0
