Metadata-Version: 2.4
Name: allocation
Version: 0.0.2
Summary: Online (streaming) portfolio construction: scikit-learn / skfolio-compatible estimators over a keyed dynamic universe
Author-email: Peter Cotton <peter.cotton@microprediction.com>
License: MIT
Project-URL: Homepage, https://allocation.microprediction.org
Project-URL: Source, https://github.com/microprediction/allocation
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy>=1.23
Requires-Dist: winning>=1.5.0
Provides-Extra: fast
Requires-Dist: winning[fast]>=1.5.0; extra == "fast"
Provides-Extra: scale
Requires-Dist: scipy>=1.10; extra == "scale"
Provides-Extra: skfolio
Requires-Dist: skfolio>=0.5; extra == "skfolio"
Provides-Extra: test
Requires-Dist: pytest>=7.0; extra == "test"
Requires-Dist: scipy>=1.10; extra == "test"
Dynamic: license-file

# allocation

**Online (streaming) portfolio construction.** A home for portfolio methods as
scikit-learn / [skfolio](https://skfolio.org)-compatible estimators
(`fit` / `partial_fit` / `predict` / `weights_`), on top of a keyed
dynamic-universe layer so they survive reconstituting universes.

📖 **[allocation.microprediction.org](https://allocation.microprediction.org)**
&nbsp;·&nbsp; 🤖 **[SKILL.md](SKILL.md)** — when to reach for `allocation` (for LLMs / code review)

```python
from allocation import ThurstonePortfolio

est = ThurstonePortfolio(calib="market", phi=1.0)
est.fit(returns)            # returns: (n_obs, n_assets)
est.weights_                # long-only weights on the simplex

for batch in stream:        # streaming / rebalancing
    est.partial_fit(batch)  # transports a fixed seed ensemble -> low turnover
    w = est.weights_
```

`score(X)` returns the portfolio's Sharpe ratio (skfolio's scoring convention),
and `to_portfolio(X)` wraps the fitted weights as a skfolio `Portfolio`
(`pip install allocation[skfolio]`) so the output drops into skfolio's metrics,
`Population`, and cross-validation.

## Why a separate package

Actually Hugo (mostly) and I did move a *batch* version of schur into skfolio and maybe more things here 
will be upstreamed into skfolio or river (Max Halford). Hugo made some improvements that are *not*
replicated in the streaming version of schur here, and the streaming versions of schur and HRP here also
deal with changing universes and weight stability. 

To be clear 
 - `allocation` is the **online** complement in that sense — estimators with `partial_fit`
 - It can also use a keyed dynamic universe (in the
spirit of [`precise`](https://github.com/microprediction/precise) but also river-ml) while staying API-compatible to both.

This is all MIT-licensed and anyone is welcome to take anything here. 

## Methods

| Estimator | Status | Notes |
|-----------|--------|-------|
| `ThurstonePortfolio` | working | Ability tilt: weights are winning probabilities of a correlated race; calibrate to a benchmark under a reference correlation, tilt under the estimate; smooth common-seed transport for `partial_fit`. Calibration and racing run through [`winning`](https://github.com/microprediction/winning). |
| `SchurComplementary` | working | The **collapsed encoding** of the HRP bridge, matching skfolio to machine precision: HRP at `gamma=0`, moving *toward* min-variance as `gamma→1` when the min-variance portfolio is long-only, without reaching it (`SchurBridge` is the exact one). Over a smooth **Fiedler seriation** instead of a dendrogram, so `partial_fit` is low-turnover. |
| `SchurBridge` | working | **Bridges named by their endpoints, one engine underneath**: `hrp_to_min_variance(gamma)`, `hmv_to_min_variance`, `herc_to_min_variance(gamma, eta, n_clusters)`, `nco_to_min_variance(gamma, n_clusters)`, `inverse_variance_to_min_variance(eta)`, each exact at both ends with the default sibling conditioning; `companion='vol'`/`'mean'` lands on maximum diversification / tangency instead. See below. |
| `HierarchicalRiskParity` | working | Dynamic HRP — the `gamma=0` special case of the Schur construction (recursive-bisection risk parity over the Fiedler order); named for recognisability. |
| `RiskParity` | working | Equal-risk-contribution (ERC); interior convex solution, solved by coordinate descent **warm-started from the previous weights** so updates stay smooth. |
| `EqualWeight`, `InverseVariance` | working | Smooth baselines for benchmarking (`1/n`; `w ∝ 1/σ²`). |
| `MinimumVariance`, `MaximumDiversification` | working | Closed-form `Σ⁻¹1` / `Σ⁻¹σ` with optional `shrinkage` for conditioning. Unconstrained (signed) so they stay smooth — a long-only QP would kink. |

Each has a river-style streaming twin for a *changing* universe — `StreamingThurstone`, `StreamingSchur`, `StreamingSchurBridge`, `StreamingHRP`, `StreamingRiskParity`, `StreamingEqualWeight`, `StreamingInverseVariance`, `StreamingMinimumVariance`, `StreamingMaximumDiversification`, `StreamingMaximumDecorrelation`, `StreamingMeanVariance` — with `learn_one({id: ret})` / `predict_one() → {id: weight}`.

**On smoothness.** Each method is written so that `partial_fit` over a drifting covariance moves weights only as much as the covariance moved. It helps to see `weights = allocator(cov_estimator(data))` as a product of two factors:

- For the **closed-form linear allocators** (`MinimumVariance`, `MaximumDiversification`, `InverseVariance`) the allocator factor is already a smooth (rational) function of `Σ`, so weight-smoothness *is* covariance-smoothness — pair them with a smooth online covariance (the default EWMA, a shrinkage estimator, or a `precise` skater) and you're done. Caveats: keep `Σ` well-conditioned (use `shrinkage`, since `Σ⁻¹` swings near a vanishing eigenvalue) and avoid hard long-only QPs (they kink at the zero bound — the smooth long-only min-variance is `SchurComplementary` as `gamma→1`).
- The package's distinctive work is the **other** family, where the allocator factor itself is rough no matter how smooth `Σ` is. Three sources, three primitives: sampling noise → common-seed transport (Thurstone); combinatorial ordering → Fiedler seriation (Schur/HRP); active-set kinks → keep the solution interior (ERC is interior by construction).

### Bridges, named by their endpoints

Schur conditioning is a technique; what matters is what it connects. Block
inversion writes the global direction `Σ⁻¹u` cluster by cluster: each cluster's
block is a minimum-variance direction on its covariance *conditional on everything
else*, with budget the inverse of that conditional variance. Every hierarchical
or clustered heuristic drops conditioning somewhere. `SchurBridge` puts it back by
degrees with two dials, `gamma` (cluster on the outside) and `eta` (asset on its
cluster mates), and a bridge earns its name only if **both ends are exact**:

```python
from allocation import SchurBridge

SchurBridge.hrp_to_min_variance(gamma=0.5)                      # HRP -> min-variance, one dial
SchurBridge.hmv_to_min_variance(gamma=0.5)                      # hierarchical min-var (Cotton 2024) -> min-variance
SchurBridge.herc_to_min_variance(gamma=0.5, eta=0.5, n_clusters=8)   # HERC -> min-variance, the square
SchurBridge.nco_to_min_variance(gamma=0.5, n_clusters=8)        # NCO -> min-variance
SchurBridge.inverse_variance_to_min_variance(eta=0.5)           # inverse variance -> min-variance (Stevens' path)
SchurBridge.hrp_to_min_variance(0.5, companion="vol")           # ... -> maximum diversification
SchurBridge.hrp_to_min_variance(0.5, companion="mean")          # ... -> tangency
est.endpoints_                                                  # e.g. ('HRP', 'minimum variance')
```

The HRP bridge needs a split rule that moves with the dial: the variance of the
child's `eta`-damped naive portfolio on its conditioned pair (`split='dial'`). HRP's
own naive split (`split='hrp'`, the rule inside `SchurComplementary` and skfolio)
starts at HRP but does not reach minimum variance; the min-variance split
(`split='minvar'`) reaches it but starts at hierarchical minimum variance. Which
published "Schur" is which is tabulated on the
[taxonomy page](https://schur.microprediction.org/taxonomy.html).

Two facts make the bridge the streaming choice. Everything after the partition is a
closed-form, continuous function of the covariance. And at the far end the weights
do not depend on the partition at all, so the turnover caused by a cluster
membership change shrinks to zero as the dials approach it (checked in
`experiments/bridge_churn_and_scale.py`): the dials that damp estimation noise damp
reclustering churn. The partition is the Fiedler order, which changes only when two
coordinates cross, bisected
(`n_clusters=None`) or cut into contiguous blocks (`n_clusters=k`), or a fixed label
vector (`clusters=`) such as sectors.

`conditioning` picks the path: `'siblings'` composes down the bisection tree
(exact, one half-size solve at the root), `'all'` conditions on every other asset
(exact, expensive), and `'factor'` conditions on one factor-mimicking portfolio per
other cluster, exact when cross-cluster dependence runs through one latent factor
per cluster and costing only cluster-sized solves; on a generic covariance it is
approximate at the far end, so the named constructors default to `'siblings'`. `long_only=True` caps `eta` at
each cluster's closed-form long-only frontier, and raises rather than return a
short position when the companion or outer layer would still produce one. The exact-arithmetic theory behind
the corners is in the papers at [schur.microprediction.org](https://schur.microprediction.org).

### Very large universes (e.g. Russell 3000)

Where the covariance is rank-deficient, every inversion-based allocator (min-variance,
max-diversification, …) is *undefined* (`is_singular()` flags it; `strict=True` refuses).
The robust large-universe methods are the ones that never invert `Σ`: inverse-variance,
HRP/Schur, and the **Thurstone tilt** — benchmark-anchored, no inversion, low turnover.
`ThurstonePortfolio(factors=k)` runs the tilt with a `k`-factor (low-rank) correlation and
an `O(M·n·k)` race instead of the dense `O(M·n²)+O(n³)` one, so it scales to thousands of
names (a few seconds per rebalance at n=3000 with the default path budget; fewer paths
trade accuracy for speed).

To make *minimum-variance itself* well-posed at scale, `FactorMinimumVariance` /
`FactorMaximumDiversification` invert a factor covariance `Σ = BBᵀ + diag(ψ)` via the
**Woodbury identity** (`O(n·k²)`, condition number bounded by an idiosyncratic floor, so the
inverse always exists). Supply a factor covariance (`loadings_` / `idiosyncratic_`, e.g. an
adapted `precise.FactorCovariance`) for the `O(n·k)` path, or let it factor a dense covariance.
And `SchurComplementary(ridge=…)` regularizes the cross-block solve so `gamma>0` stays
well-posed (and self-tempers toward HRP) when blocks are rank-deficient.

### Trading costs

`TurnoverPenalty(estimator, cost=λ)` wraps any estimator with a quadratic trading
cost (market-impact model). Minimising `‖w − w*‖² + λ‖w − w_prev‖²` gives the smooth,
budget-preserving blend `w_t = α w* + (1−α) w_prev` with `α = 1/(1+λ)`, so each step's
turnover is scaled exactly by `α`. This explicit damping composes with the implicit
turnover control (smooth target) and has a streaming twin `StreamingTurnoverPenalty`.

### Linear constraints

`BoxConstrained(estimator, lower=, upper=, groups=, group_caps=)` imposes per-asset
bounds and (disjoint) group caps on any estimator via a **log-barrier**, not a QP.
The barrier's domain *is* the constraint set, so the result is feasible for
any `tau>0` (strictly, except for coordinates that a bound or cap pins exactly) —
feasibility never depends on tuning. Infeasible or malformed constraints raise. The operator stays C¹ (no
active-set kinks), so turnover stays low. Streaming twin: `StreamingBoxConstrained`.

## Comparing methods

`allocation.backtest` is a numpy-only walk-forward harness: it trades each
estimator forward (form weights → earn next period → update) and tabulates
out-of-sample Sharpe, **turnover**, concentration, and Sharpe net of a
proportional cost. Turnover and cost are charged on the trades needed to return
drifted holdings to each new target.

```python
from allocation import SchurComplementary, TurnoverPenalty, MinimumVariance
from allocation.backtest import compare, format_table, make_panel

panel = make_panel()                       # synthetic; or any (n_obs, n_assets) array
print(format_table(compare({
    "schur":   lambda: SchurComplementary(gamma=0.5),
    "schur+λ": lambda: TurnoverPenalty(SchurComplementary(gamma=0.5), cost=3.0),
    "minvar":  lambda: MinimumVariance(shrinkage=0.1),
}, panel)))
```

No data is shipped (to avoid bloat); `compare()` takes any array, and
`load_returns_csv(url)` can pull a raw CSV from a data repo at runtime.

For a robust, universe-independent comparison, `compare_random_subsets(...)` runs
a Monte-Carlo over random name-subsets and windows, reporting each method's mean
± sd net Sharpe, mean turnover, and **win rate** (fraction of trials it had the
best net Sharpe) — averaging out the single-backtest dependence on which names
and period you happened to pick.

## Design

```
allocation/
  base.py        # BaseOnlinePortfolio: fit / partial_fit / predict / weights_
  moments.py     # EwmaCovariance (default); any partial_fit/covariance_ estimator plugs in
  baselines.py   # EqualWeight / InverseVariance / RiskParity (ERC)
  convex.py      # MinimumVariance / MaximumDiversification (closed-form, signed)
  thurstone.py   # ThurstonePortfolio
  _thurstone/    # calibration + transport (thin over winning)
  schur.py       # SchurComplementary / HierarchicalRiskParity
  bridge.py      # SchurBridge: the pair-form engine the other allocators are corners of
  _schur/        # Fiedler seriation + Schur coupling engine + the bridge (pairs, dials, knots)
  keyed.py       # river-style streaming twins over a changing universe
  universe.py    # keyed dynamic-universe state for the batch estimators (planned)
```

Covariance is pluggable: the default is a light EWMA, but any online estimator
exposing `partial_fit` and `covariance_` (e.g. a `precise` skater) can be passed
via `covariance=`.

## Papers

The two novel methods are written up as working papers.

- **Thurstone Portfolios: Allocation as Winning Probability** — long-only
  allocation as the winning probabilities of a correlated race: free of the
  duplication paradox, tail-consistent when the race is driven by a
  downside-dependent simulation, with an implied regularized objective and a
  smoothness/turnover bound. Draft in
  [`papers/thurstone-portfolios/`](papers/thurstone-portfolios), built on
  [`thurstone`](https://github.com/microprediction/thurstone).
- **Schur-complementary allocation** — robust, inversion-light allocation along a
  spectral (Fiedler) seriation; background and bibliography at
  [schur.microprediction.org](https://schur.microprediction.org).

## Status

Early. The Thurstone, Schur/HRP, and risk-parity estimators all work and are
tested, batch and streaming; the keyed dynamic-universe layer for the *batch*
estimators (so `partial_fit` survives a changing asset count, as the streaming
twins already do) is next. The theory behind the Thurstone method (feasibility,
redundancy consistency, the implied regularized objective, smoothness/turnover)
is written up in the accompanying paper.

## License

MIT © Peter Cotton
