Metadata-Version: 2.4
Name: hydraboost
Version: 0.1.0
Summary: Bandit-allocated heterogeneous Newton boosting for tabular data
Project-URL: Homepage, https://github.com/reconstructive/hydraboost
Project-URL: Repository, https://github.com/reconstructive/hydraboost
Project-URL: Issues, https://github.com/reconstructive/hydraboost/issues
Project-URL: Changelog, https://github.com/reconstructive/hydraboost/blob/main/CHANGELOG.md
Author-email: Your Name <richardstevenball@gmail.com>
License: MIT License
        
        Copyright (c) 2026 Your Name
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
License-File: LICENSE
Keywords: bandit,boosting,gradient-boosting,machine-learning,scikit-learn,tabular
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Requires-Dist: lightgbm>=4.0
Requires-Dist: numpy>=1.24
Requires-Dist: scikit-learn>=1.3
Requires-Dist: scipy>=1.10
Provides-Extra: benchmark
Requires-Dist: catboost>=1.2; extra == 'benchmark'
Requires-Dist: pandas>=2.0; extra == 'benchmark'
Requires-Dist: xgboost>=2.0; extra == 'benchmark'
Provides-Extra: dev
Requires-Dist: pandas>=2.0; extra == 'dev'
Requires-Dist: pytest-cov>=4.1; extra == 'dev'
Requires-Dist: pytest>=7.4; extra == 'dev'
Requires-Dist: ruff>=0.6; extra == 'dev'
Description-Content-Type: text/markdown

# HydraBoost

Bandit-allocated heterogeneous Newton boosting for tabular data.

Most gradient boosting libraries assume the answer is always an axis-aligned
tree. HydraBoost does not. Each boosting round fits one weak learner drawn
from a pool of structurally different families, and a bandit allocates rounds
to whichever family is actually reducing validation loss. A dataset with smooth
or linear structure gets ridge, spline and RBF steps that a tree ensemble would
need many splits to approximate; a dense interaction surface gets deep trees.
You do not have to know which in advance, and you do not have to tune it to
find out.

```bash
pip install hydraboost
```

```python
from hydraboost import HydraBoostClassifier

clf = HydraBoostClassifier().fit(X_train, y_train)
proba = clf.predict_proba(X_test)

print(clf.arm_usage_)
# {'pair': 0.08, 'rbf': 0.14, 'ridge': 0.29, 'spline': 0.09,
#  'tree64': 0.19, 'tree8': 0.21}
```

That last line is a readout of which learner families kept earning their
rounds on your data.

Read it as a diagnostic, not as a structure detector. A family that solves its
part of the problem in a few rounds can end up with a small share: on purely
linear synthetic data, ridge fits the signal almost immediately and the
remaining rounds scatter across arms chasing noise. High share means "kept
helping for a long time", which is not the same as "matters most".

## How it works

Every round, one learner is fitted from one of six families on the current
Newton gradients and Hessians:

| Arm | Learner | Captures |
| --- | --- | --- |
| `tree8` | 8-leaf histogram tree | shallow interactions |
| `tree64` | 64-leaf histogram tree | deeper interactions |
| `treeXL` | 256-leaf histogram tree (large data only) | dense interaction surfaces |
| `ridge` | weighted ridge on standardised features | global linear and oblique structure |
| `spline` | componentwise cubic spline | smooth univariate effects |
| `rbf` | weighted ridge on k-means RBF prototypes | smooth local multivariate structure |
| `pair` | quantile-binned pairwise interaction grid | specific two-way interactions |

Arm selection is epsilon-greedy on an exponential moving average of realised
validation improvement, with annealed exploration. Tree arms use their own
per-leaf Newton optima; the other arms take a one-step Newton line search so
learners on different scales compose correctly.

On small datasets a single validation split is a poor signal for both early
stopping and arm feedback, so below `small_n` rows the estimator pools the
split back in and fits several members on rotating splits, averaging them.

## API

`HydraBoostClassifier` and `HydraBoostRegressor` are standard scikit-learn
estimators. They handle categorical columns, missing values, binary and
multiclass targets internally, and work inside `Pipeline`, `GridSearchCV`,
`cross_val_score` and the rest.

```python
from sklearn.model_selection import cross_val_score
from hydraboost import HydraBoostRegressor

scores = cross_val_score(HydraBoostRegressor(), X, y, cv=5)
```

Main parameters, all with defaults chosen a priori rather than tuned per
dataset:

| Parameter | Default | Meaning |
| --- | --- | --- |
| `learning_rate` | 0.09 | shrinkage per round |
| `n_rounds` | 800 | maximum boosting rounds |
| `patience` | 60 | early stopping patience |
| `subsample` | 0.8 | row subsample per round |
| `validation_fraction` | 0.15 | held out for early stopping and arm feedback |
| `small_n` | 2500 | below this, use bagged internal validation |
| `n_members` | 5 | members fitted in bagged mode |

## Benchmarks, and what they do and do not show

HydraBoost was developed against a 16-dataset suite and compared to tuned
XGBoost, LightGBM and CatBoost under a shared protocol: 5-fold cross
validation, identical folds and design matrices for every model, a 15 percent
inner validation split, and test folds never touched during fitting or
selection. Each opponent got 8 random hyperparameter configs with early
stopping. HydraBoost got one fixed config and no tuning.

On that suite HydraBoost had the best mean metric on 10 of 16 datasets and the
best mean rank (1.75, against CatBoost 2.19, LightGBM 2.88, XGBoost 3.19),
while fitting 1.5 to 23 times faster than the opponents' average.

**These numbers should be read with care, and the honest caveats are these.**
The algorithm went through four design iterations against that same suite, and
iterating against a benchmark inflates significance, so the reported p-values
are optimistic. CatBoost's depth grid was capped at 8 for single-threaded
compute budget, which handicaps it. Every model ran single-threaded. No
untouched large-scale benchmark (OpenML CC-18, Grinsztajn et al.) has been run
yet, and until one has, the fair claim is "competitive with tuned GBDTs out of
the box on small to mid-size tabular data", not "state of the art".

The full harness and per-fold results are in [`benchmarks/`](benchmarks/) so
you can rerun or contest any of it. Independent results on datasets I have
never seen are the most useful contribution anyone could make here, including
negative ones.

Known weak spots from the existing runs: small noisy datasets with many
features relative to rows, and problems that are purely dense interaction
surfaces, where a dedicated GBDT still wins.

## Relation to prior work

HydraBoost combines existing ideas rather than inventing a new one. The
specific combination does not appear to be published, but the components are
well established and the credit belongs upstream:

- Gradient and Newton boosting: Friedman (2001); Chen and Guestrin (2016)
- Componentwise boosting with base-learner selection per iteration, the
  direct ancestor of the arm pool: Bühlmann and Yu (2003); the `mboost` family
- Bandits inside boosting, though for feature selection rather than learner
  families: Busa-Fekete and Kégl (2009, 2010)
- Heterogeneous base learners across iterations as an alternative to
  hyperparameter tuning, and cross-validated early stopping for small data:
  the `OGBoost` package (2025)
- Pairwise interaction terms in an additive model: the EBM / `interpret`
  line of work
- Histogram tree construction: LightGBM (Ke et al., 2017), used directly here

## Contributing

Benchmark results on new datasets, especially failures, are the most valuable
thing you can send. See [CONTRIBUTING.md](CONTRIBUTING.md).

## License

MIT
