Metadata-Version: 2.1
Name: researchlab
Version: 0.0.2
Summary: Lightweight dataset/model benchmarking framework for ML experiments
Home-page: https://github.com/felipeangelimvieira/researchlab
License: BSD-3-Clause
Keywords: machine learning,benchmarking,sklearn,skpro
Author: felipeangelimvieira
Author-email: felipeangelim@gmail.com
Requires-Python: >=3.11,<3.14
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: BSD License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Provides-Extra: all
Provides-Extra: aws
Provides-Extra: skpro
Requires-Dist: boto3 (>=1.39.0,<2.0.0) ; extra == "aws" or extra == "all"
Requires-Dist: click (>=8.1.8,<9.0.0)
Requires-Dist: numpy (>=1.26.0)
Requires-Dist: pandas (>=2.1.0,<3.0.0)
Requires-Dist: scikit-base (>=0.13,<0.14) ; extra == "skpro" or extra == "all"
Requires-Dist: scikit-learn (>=1.3.0,<2.0.0)
Requires-Dist: skpro (==2.11.0) ; extra == "skpro" or extra == "all"
Project-URL: Repository, https://github.com/felipeangelimvieira/researchlab
Description-Content-Type: text/markdown

# researchlab

[![CI](https://github.com/felipeangelimvieira/researchlab/actions/workflows/ci.yml/badge.svg)](https://github.com/felipeangelimvieira/researchlab/actions/workflows/ci.yml)
[![PyPI](https://img.shields.io/pypi/v/researchlab.svg)](https://pypi.org/project/researchlab/)
[![Python](https://img.shields.io/pypi/pyversions/researchlab.svg)](https://pypi.org/project/researchlab/)
[![License](https://img.shields.io/pypi/l/researchlab.svg)](LICENSE)

**researchlab** is a lightweight benchmarking framework that runs every
`(dataset, model)` pair in your experiment grid, skips pairs that are
already done, and stores results as plain CSV files — locally or on S3.

It ships two ready-made runners:

| Runner                        | What it evaluates                                           |
| ----------------------------- | ----------------------------------------------------------- |
| `SklearnClassificationRunner` | Any sklearn-compatible classifier with K-fold CV            |
| `SkproRegressionRunner`       | Any skpro probabilistic regressor with proper scoring rules |

You can also plug in any library or custom evaluation logic by subclassing
`LocalRunner` or `BaseRunner`.

## Install

Core package (sklearn benchmarking, local storage):

```bash
pip install researchlab
```

With skpro probabilistic regression support:

```bash
pip install "researchlab[skpro]"
```

With AWS S3 storage and Batch/ECS submission:

```bash
pip install "researchlab[aws]"
```

Everything:

```bash
pip install "researchlab[all]"
```

## Quick start — sklearn classification

```python
# experiment.py
from pathlib import Path
from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import get_scorer
from sklearn.model_selection import StratifiedKFold
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

from researchlab import Dataset, DatasetPayload, Model, SklearnClassificationRunner


class BreastCancer(Dataset):
    name = "breast_cancer"

    def _load(self):
        b = load_breast_cancer(as_frame=True)
        return DatasetPayload(data=(b.data, b.target.to_frame(name="target")))


class LogReg(Model):
    name = "logistic_regression"

    def _instantiate(self, metadata):
        return make_pipeline(StandardScaler(), LogisticRegression(max_iter=2000))


runner = SklearnClassificationRunner(
    dataset_collection={"breast_cancer": BreastCancer()},
    model_collection={"logistic_regression": LogReg()},
    artifact_repository=Path("results/"),
    cv=StratifiedKFold(n_splits=5, shuffle=True, random_state=42),
    metrics={"accuracy": get_scorer("accuracy"), "roc_auc": get_scorer("roc_auc")},
)

if __name__ == "__main__":
    runner.cli()
```

```bash
python experiment.py run                    # run all pairs, skip done
python experiment.py run --count            # show done / remaining
python experiment.py run -d breast_cancer   # one dataset only
python experiment.py concatenate -o out.csv # merge all results
```

## Quick start — skpro probabilistic regression

```python
from pathlib import Path
from sklearn.datasets import load_diabetes
from sklearn.model_selection import KFold
from skpro.regression.residual import ResidualDouble
from sklearn.linear_model import LinearRegression

from researchlab import Dataset, DatasetPayload, Model, SkproRegressionRunner


class Diabetes(Dataset):
    name = "diabetes"

    def _load(self):
        b = load_diabetes(as_frame=True)
        return DatasetPayload(data=(b.data, b.target.to_frame(name="target")))


class LinearResidual(Model):
    name = "linear_residual"

    def _instantiate(self, metadata):
        return ResidualDouble(LinearRegression())


runner = SkproRegressionRunner(
    dataset_collection={"diabetes": Diabetes()},
    model_collection={"linear_residual": LinearResidual()},
    artifact_repository=Path("results/"),
    cv=KFold(n_splits=5, shuffle=True, random_state=42),
    metrics=["CRPS", "LogLoss"],
)

if __name__ == "__main__":
    runner.cli()
```

## Scaling out to AWS

Add the `AWSRunner` mixin to any runner to get `aws batch` and `aws ecs`
subcommands — no separate submission script needed:

```python
from researchlab.aws import AWSRunner, S3ArtifactRepository

class MyAwsRunner(AWSRunner, SklearnClassificationRunner):
    pass

runner = MyAwsRunner(
    ...
    artifact_repository=S3ArtifactRepository("s3://my-bucket/my-experiment"),
    container_command=["python", "/app/experiment.py"],
)
```

```bash
# submit every pair as a Batch job from your laptop
python experiment.py aws batch \
    --job-queue my-queue \
    --job-definition my-jobdef

# collect results after jobs finish
python experiment.py concatenate -o results.csv
```

## Examples

| Example                                    | What it shows            |
| ------------------------------------------ | ------------------------ |
| `examples/researchlab_classification/`     | Local sklearn benchmark  |
| `examples/researchlab_regression/`         | Local skpro benchmark    |
| `examples/researchlab_aws_classification/` | sklearn + S3 + Batch/ECS |
| `examples/researchlab_aws_regression/`     | skpro + S3 + Batch/ECS   |

## Documentation

Full documentation lives in [`docs/`](docs/) and is built with Quarto.

```bash
quarto render docs/
```

## Contributing

1. Fork the repository
2. Install dev dependencies: `poetry install --extras all --with dev`
3. Run tests: `poetry run pytest`
4. Open a pull request against `main` — CI runs automatically

## License

BSD 3-Clause


metrics = [RMSE()]

cv = KFold(n_splits=3, shuffle=True, random_state=42)

RUNNER = ExperimentRunner()
SAVE_BEST_MODEL = True
SAVE_BEST_MODEL_PICKLE = True
```

To store artifacts outside the experiment directory, set `ARTIFACT_ROOT` in
`experiments_config.py`.

- Local path: `ARTIFACT_ROOT = "/absolute/path/to/results"`
- Relative local path: `ARTIFACT_ROOT = "shared-results"`
- S3 path: `ARTIFACT_ROOT = "s3://my-bucket/experiments/exp1"`

If `ARTIFACT_ROOT` is omitted, storage defaults to the local directory
`<experiment_dir>/results/`.

If you need custom orchestration, define `RUNNER` or `runner` in
`experiments_config.py` with an instance of a `BaseRunner` subclass. The
built-in `ExperimentRunner` preserves the current behavior and can be
subclassed by overriding hooks such as config validation, dataset preparation,
task inference, evaluation, summary aggregation, or best-model selection.

## CLI

Run an experiment:

```bash
poetry run probabilistic_experiment path/to/experiment
```

Validate only:

```bash
poetry run probabilistic_experiment path/to/experiment --dry-run
```

Run specific models:

```bash
poetry run probabilistic_experiment path/to/experiment --model model_a --model model_b
```

Run specific datasets:

```bash
poetry run probabilistic_experiment path/to/experiment --dataset dataset_a --dataset dataset_b
```

Re-run datasets with existing outputs:

```bash
poetry run probabilistic_experiment path/to/experiment --include-evaluated
```

Concatenate all per-model fold outputs:

```bash
poetry run probabilistic_experiment concat path/to/experiment
```

If `--output` is omitted, the concatenated CSV is written to
`concatenated_results.csv` under the active artifact root.

Erase a model's stored artefacts and refresh summaries:

```bash
poetry run probabilistic_experiment path/to/experiment --erase-model model_a
```

Build and submit the AWS Batch example:

```bash
poetry run probabilistic_experiment aws build-image
poetry run probabilistic_experiment aws register-job-definition --image-uri ... --execution-role-arn ... --job-role-arn ...
poetry run probabilistic_experiment aws submit examples/aws_batch_local_classification --job-queue ... --job-definition ... --artifact-root s3://...
```

## Output Layout

The runner writes results to `path/to/experiment/results/<dataset>/`:

- `<model>.csv`: standardized fold-level results
- `<model>_data.pkl`: pickled fold data and predictions
- `metrics.csv`: dataset-level aggregated summary for all remaining models
- `best_model.json`: metadata for the best available model
- `best_model.pkl`: optional fitted best model pickle

If `ARTIFACT_ROOT` is configured, the same layout is written under that local or
S3 root instead of `path/to/experiment/results/`.

The runner uses a storage abstraction internally, so reads, writes, deletion,
and listing all go through the same backend interface. Local filesystem storage
is the default implementation; S3 is available by setting `ARTIFACT_ROOT` to an
`s3://...` URI.

The fold-level CSVs always include:

- `dataset`
- `model`
- `task`
- `fold`
- `status`
- `fit_time`
- `pred_time`
- one or more `test_<metric>` columns

## More

- Example config: [example/exp1/experiments_config.py](example/exp1/experiments_config.py)
- Local runnable example: [examples/local_classification/README.md](examples/local_classification/README.md)
- Detailed usage guide: [docs/USAGE.md](docs/USAGE.md)
- Artifact storage guide: [docs/ARTIFACTS.md](docs/ARTIFACTS.md)
- AWS Batch parallel tutorial: [docs/AWS_BATCH_TUTORIAL.md](docs/AWS_BATCH_TUTORIAL.md)
- Runner architecture: [experiment/__init__.py](src/probabilistic_experiment/experiment/__init__.py)
- Storage API module: [artifacts.py](src/probabilistic_experiment/artifacts.py)

