Metadata-Version: 2.4
Name: codebookpy
Version: 0.1.0
Summary: LLM-powered codebook reader for exploratory data analysis
Author-email: ethand05hi <you@example.com>
License: MIT
Project-URL: Homepage, https://github.com/ethand05hi/codebookpy
Project-URL: Repository, https://github.com/ethand05hi/codebookpy
Project-URL: Issues, https://github.com/ethand05hi/codebookpy/issues
Keywords: codebook,data analysis,eda,survey data,pdf,llm
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE.txt
Requires-Dist: pdfplumber>=0.10
Requires-Dist: pandas>=1.5
Requires-Dist: rich>=13.0
Requires-Dist: pikepdf>=8.0
Provides-Extra: anthropic
Requires-Dist: anthropic>=0.20; extra == "anthropic"
Provides-Extra: openai
Requires-Dist: openai>=1.0; extra == "openai"
Provides-Extra: ocr
Requires-Dist: pymupdf>=1.23; extra == "ocr"
Requires-Dist: pytesseract>=0.3; extra == "ocr"
Requires-Dist: Pillow>=10.0; extra == "ocr"
Provides-Extra: all
Requires-Dist: anthropic>=0.20; extra == "all"
Requires-Dist: openai>=1.0; extra == "all"
Requires-Dist: pymupdf>=1.23; extra == "all"
Requires-Dist: pytesseract>=0.3; extra == "all"
Requires-Dist: Pillow>=10.0; extra == "all"
Dynamic: license-file

# codebookpy

**LLM-powered codebook reader for exploratory data analysis.**

`codebookpy` lets you import a statistical codebook PDF into Python and look up variable descriptions, value codes, and data summaries alongside your DataFrame — all from a single function call.

---

## Installation

```bash
pip install codebookpy
```

Install with your preferred LLM backend:

```bash
pip install "codebookpy[anthropic]"   # Claude (recommended)
pip install "codebookpy[openai]"      # OpenAI
pip install "codebookpy[all]"         # everything including OCR support
```

### OCR support (for scanned/image-based PDFs)

If your codebook is a scanned PDF, install the OCR extras and Tesseract:

```bash
pip install "codebookpy[ocr]"
```

Then install the Tesseract engine:
- **Windows**: download from [UB-Mannheim](https://github.com/UB-Mannheim/tesseract/wiki)
- **macOS**: `brew install tesseract`
- **Linux**: `sudo apt install tesseract-ocr`

Update `TESSERACT_CMD` at the top of `core.py` to match your install path if needed (Windows only).

---

## Quick start

```python
import pandas as pd
from codebookpy import Codebook, cb_lookup

# Load your data
df = pd.read_csv("my_data.csv")

# Parse the codebook — pass df so the LLM anchors to your actual column names
cb = Codebook(
    "my_codebook.pdf",
    anthropic_api_key="sk-ant-...",   # or openai_api_key="sk-..."
    df=df,
)

# Look up variables — codebook info only
cb_lookup(cb, ["AGE", "SEX", "RACE"])

# Look up variables — with DataFrame summary
cb_lookup(cb, ["AGE", "SEX", "RACE"], df=df, summarize_df=True)
```

---

## API reference

### `Codebook(pdf_path, *, anthropic_api_key=None, openai_api_key=None, df=None, known_vars=None, verbose=True)`

Parses a codebook PDF using an LLM backend. Pass either `anthropic_api_key` or `openai_api_key` (not both).

| Parameter | Type | Description |
|---|---|---|
| `pdf_path` | `str \| Path` | Path to the codebook PDF |
| `anthropic_api_key` | `str` | Anthropic API key |
| `openai_api_key` | `str` | OpenAI API key |
| `df` | `DataFrame` | If provided, all column names are used to anchor the LLM during parsing |
| `known_vars` | `list[str]` | Explicit variable list (overrides `df` if both provided) |
| `verbose` | `bool` | Print parsing progress (default `True`) |

### `cb_lookup(codebook, var_list, df=None, summarize_df=False)`

Look up variables and print formatted summaries to the console.

| Parameter | Type | Description |
|---|---|---|
| `codebook` | `Codebook` | A parsed `Codebook` object |
| `var_list` | `list[str]` | Variable names to look up |
| `df` | `DataFrame` | Optional DataFrame for data summaries |
| `summarize_df` | `bool` | If `True` and `df` provided, prints dtype, missingness, and numeric/frequency/date summaries |

---

## How it works

1. **Text extraction** — `pdfplumber` extracts text from the PDF. If the PDF is malformed, `pikepdf` repairs it first. If it's a scanned image-based PDF, OCR kicks in automatically via `pymupdf` + `pytesseract`.
2. **Anchor-guided parsing** — Your DataFrame column names are passed to the LLM so it searches for each variable by name rather than guessing. Variables are sent in batches of 10 with the most relevant codebook sections.
3. **Rate limiting** — Requests are throttled automatically to stay within API rate limits.
4. **Lookup** — `cb_lookup()` queries the in-memory parsed result and renders a formatted panel per variable using `rich`.

---

## License

MIT
