Metadata-Version: 2.4
Name: synthbench
Version: 0.2.0
Summary: Synthetic datasets for ML benchmarking with controllable complexity, configurable corruptions, and full provenance.
Project-URL: Homepage, https://github.com/JanTeichertKluge/synth-bench
Project-URL: Documentation, https://JanTeichertKluge.github.io/synth-bench
Project-URL: Repository, https://github.com/JanTeichertKluge/synth-bench.git
Project-URL: Issues, https://github.com/JanTeichertKluge/synth-bench/issues
Project-URL: Changelog, https://github.com/JanTeichertKluge/synth-bench/releases
Author-email: Jan Teichert-Kluge <janteiklu@gmail.com>
License-Expression: MIT
License-File: LICENSE
Keywords: benchmarking,data-generating-process,dataset-generation,machine-learning,reproducibility,synthetic-data
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Typing :: Typed
Requires-Python: >=3.12
Requires-Dist: numpy>=2.0
Requires-Dist: scikit-learn>=1.5
Requires-Dist: scipy>=1.12
Provides-Extra: dev
Requires-Dist: pre-commit>=3.5; extra == 'dev'
Requires-Dist: pytest-cov>=7.0; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: ruff>=0.5; extra == 'dev'
Provides-Extra: docs
Requires-Dist: matplotlib>=3.7; extra == 'docs'
Requires-Dist: mkdocs-jupyter>=0.26; extra == 'docs'
Requires-Dist: mkdocs-material>=9.5; extra == 'docs'
Requires-Dist: mkdocs>=1.6; extra == 'docs'
Requires-Dist: mkdocstrings[python]>=1.0; extra == 'docs'
Requires-Dist: openml>=0.14; extra == 'docs'
Requires-Dist: pandas>=2.0; extra == 'docs'
Provides-Extra: io
Requires-Dist: pyarrow>=14.0; extra == 'io'
Provides-Extra: neural
Requires-Dist: torch>=2.1; extra == 'neural'
Description-Content-Type: text/markdown

<div align="center">
  <img src="https://raw.githubusercontent.com/JanTeichertKluge/synth-bench/main/icon.png" alt="synthbench" width="420">
</div>

---

[![PyPI](https://img.shields.io/pypi/v/synthbench.svg)](https://pypi.org/project/synthbench/)
[![Python](https://img.shields.io/pypi/pyversions/synthbench.svg)](https://pypi.org/project/synthbench/)
[![CI](https://github.com/JanTeichertKluge/synth-bench/actions/workflows/ci.yml/badge.svg)](https://github.com/JanTeichertKluge/synth-bench/actions/workflows/ci.yml)
[![Docs](https://img.shields.io/badge/docs-mkdocs-blue.svg)](https://JanTeichertKluge.github.io/synth-bench)
[![License: MIT](https://img.shields.io/badge/license-MIT-green.svg)](LICENSE)

synthbench generates synthetic datasets for benchmarking. You choose how complicated the
signal is, layer noise or missing data on top, and get back a dataset that carries a record
of how it was made. One integer seed reproduces the whole thing.

The point is knowing the answer in advance. On real data you can measure that your model
scored 0.83 and have no idea whether 0.85 was available or whether you already hit the
ceiling. Here the ceiling is in the metadata.

## Installation

```bash
pip install synthbench
```

Parquet serialization needs `pip install "synthbench[io]"`, and `RandomNeuralDGP` needs
`pip install "synthbench[neural]"` for PyTorch. Everything else works from the base install.

## Basic usage

```python
from synthbench import BenchPipeline, LinearDGP, MissingDataCorruptor

pipeline = BenchPipeline(
    LinearDGP(complexity="medium", task_type="classification"),
    corruptors=[MissingDataCorruptor(proportion=0.1, mechanism="mar")],
)
result = pipeline.run(n_samples=500, n_features=10, random_state=42)

print(result.X.shape)  # (500, 10)
print(result.metadata["bayes_error_analytic"])  # exact error floor
print(result.metadata["effective_rank"])  # feature space dimensionality
```

## Data-generating processes

Each one takes a `complexity` parameter and records which features actually carry signal.

| DGP | Signal |
|---|---|
| `LinearDGP` | Linear combination, sparsity and noise set by complexity |
| `PolynomialDGP` | Polynomial terms and interactions |
| `TreeDGP` | Axis-aligned splits, depth set by complexity |
| `FriedmanDGP` | The Friedman 1/2/3 benchmark functions |
| `AdditiveDGP` | Sum of univariate functions, GAM-style |
| `SparseDGP` | Explicit number of informative features via `k` |
| `GeometricDGP` | Moons, circles, spirals |
| `RandomNeuralDGP` | A randomly initialised MLP (needs `[neural]`) |

## Corruptors

`MeasurementNoise`, `Outlier`, `MissingData`, `Collinearity`, and `Categorical` transform the
feature matrix; `LabelNoiseCorruptor` goes in the separate `label_corruptors=` argument and
touches only `y`. Feature corruptors always run in a fixed order regardless of how you list
them, so two pipelines with the same components produce the same data. Each one records how
much of each feature's information it destroyed, in
`metadata["effective_feature_importances"]`.

Severity is a preset: `"low"`, `"medium"`, or `"high"`. Pass the underlying parameter
(`proportion`, `noise_level`, `n_bins`) when you want a specific value instead.

## More than two classes

Pass `n_classes` to any DGP except `FriedmanDGP`, with `class_weight="balanced"` or an
explicit list of priors.

```python
from synthbench import BenchPipeline, LinearDGP

result = BenchPipeline(
    LinearDGP(
        task_type="classification", n_classes=4, class_weight=[0.4, 0.3, 0.2, 0.1]
    )
).run(n_samples=1000, n_features=10, random_state=0)
```

How the classes are produced differs by DGP, and `metadata["label_mechanism"]` says which
you got. `LinearDGP`, `SparseDGP`, `TreeDGP`, and `RandomNeuralDGP` emit one latent score per
class and draw through a softmax, so the classes are unordered. `PolynomialDGP` and
`AdditiveDGP` are built on a fixed term basis with only one signal to work with, so theirs is
cut into ordered bins — fine for genuinely ordinal targets, misleading as a general multiclass
benchmark. `FriedmanDGP` refuses: its functions are specific published formulas, and a
multi-output variant would not be the thing anyone cites.

## How hard is the dataset?

Two numbers, and the difference between them matters:

- `bayes_error_analytic` is exact. The DGP draws each label from a known probability, so the
  irreducible error is computable rather than estimated. No model can do better. It reflects
  the labels only, so feature corruption does not change it.
- `bayes_error` is an empirical 1-NN leave-one-out error on the corrupted features. It moves
  when you corrupt things, which makes it useful for comparing severity levels against each
  other, but it is biased upwards in more than a few dimensions. It is not a bound.

For a balanced `LinearDGP` classification task the analytic floor is about 0.325, while the
1-NN estimate reads about 0.456 at n=500, p=10. Use the first when you need a floor and the
second when you need a relative signal.

## Ablations

`severity_sweep` and `difficulty_sweep` vary one axis and return a dict keyed by level.
`experiment_grid` crosses sample size, complexity, and severity. Seeds come from a nested
`SeedSequence` hierarchy, so cells are independent of each other but reproducible across
runs.

```python
from synthbench import LinearDGP, OutlierCorruptor, experiment_grid

grid = experiment_grid(
    LinearDGP,
    OutlierCorruptor,
    n_samples_list=[200, 500, 1000],
    complexities=["low", "medium", "high"],
    severities=["low", "medium", "high"],
    n_features=10,
    random_state=0,
    task_type="classification",
)

print(grid[(500, "high", "medium")].metadata["bayes_error_analytic"])
```

## Suites and serialization

`BenchSuite("easy-classification").run()` generates a curated collection in one call and
returns it keyed by label — handy as a shared baseline, since the suite name alone pins the
data. `BenchSuite.from_dict` and `BenchSuite.from_json` take your own specs.

`to_parquet` / `to_csv` round-trip the data and the full metadata, and
`BenchPipeline.from_metadata` rebuilds the pipeline from that metadata so a saved dataset can
be regenerated rather than shipped.

## Docs

Full reference and runnable notebooks at
[JanTeichertKluge.github.io/synth-bench](https://JanTeichertKluge.github.io/synth-bench).
