Metadata-Version: 2.4
Name: codegauge
Version: 0.0.1
Summary: Toolkit for benchmarking, scoring, and comparing LLM-generated code.
Author-email: Okan Gumus <okanngumus4@gmail.com>
License-Expression: MIT
Project-URL: Homepage, https://github.com/okngms/codegauge
Project-URL: Issues, https://github.com/okngms/codegauge/issues
Keywords: llm,evaluation,benchmark,mutation-testing,code-quality,pytest
Classifier: Development Status :: 2 - Pre-Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development :: Testing
Classifier: Topic :: Software Development :: Quality Assurance
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Dynamic: license-file

# CodeGauge

Toolkit for benchmarking, scoring, and comparing LLM-generated code using automated
test harnesses, human rubrics, and mutation testing.

> **Status: early development.** This release reserves the package name. The first
> usable version is `0.1.0`. Nothing here is stable yet.

## What it will do

Three commands, one command-line tool:

- `codegauge bench` — runs a set of Python problems through several models, executes the
  generated code against a test suite in an isolated subprocess, and reports happy-path
  and edge-case pass rates separately.
- `codegauge prefs` — presents two solutions blind and in random order, records your
  first impression before the test results are revealed, then captures your final
  preference and written rationale as a JSONL preference dataset.
- `codegauge testeval` — asks a model to write unit tests for a target module, then
  measures whether those tests would actually catch a bug, using coverage and mutation
  testing.

## Install

```bash
pip install codegauge
```

## License

MIT
