# Changelog

All notable changes to this project will be documented in this file.

The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).


## [0.24.0] 16.08.2026

### Added
- Add `score_method=hellinger`: the Hellinger distance
  `0.5 * sum((sqrt(p) - sqrt(q)) ** 2)` as a dimension-neutral split score,
  less sensitive to noise in sparsely populated bins than total variation.
- Add `smoothing` training setting: Laplace smoothing alpha applied to the
  proportion estimates used by the `tv`/`hellinger` scoring. The
  left/neutral/right classification always uses the raw proportions, so the
  model structure is unaffected.
- Add `min_split_improvement` training setting: multi-dimensional candidates
  (dim >= 2) must beat the current best split by a minimum relative amount,
  preventing sparse combinations from winning purely due to score noise.
- Extend the `PILZ_DEBUG_NEUTRAL_DIR` debug dump with per-node sparse
  statistics (possible/observed/unobserved/sparse cells, neutral mass and
  rows, branch sizes) and per-repetition score statistics of the best
  candidate (`rep_stats`).

### Changes
- `CombinedCategorizedFeature` keeps the raw count column, enabling the
  smoothed scoring for multi-dimensional candidates.
- Extend `FEATURE.md` with the implemented measures (smoothing,
  min_split_improvement, Hellinger) and updated recommendations.

## [0.23.0] 16.08.2026

### Added
- Add `FEATURE.md` describing split scoring methods, sparse combinations and
  a unified candidate scoring proposal.
- Add `score_method` training setting with the dimension-neutral total
  variation (`tv`, default) as the split scoring method; the previous
  `abs(diff) * sqrt(abs(diff))` heuristic stays available as `entro`.
- Add dimension-neutrality tests: equal distributions with different bin
  counts score identically, and 2D combinations with an independent feature
  match the 1D score.
- Add `score_method` regression coverage in settings and training tests.

### Changes
- `calc_diff()` now computes the total variation `0.5 * sum(|diff|)` over all
  observed bin combinations by default, replacing the entropy-weighted
  difference sum.
- Complete truncated sentences in `FEATURE.md` and rename it from
  `FEATRUE.md`.
- Correct stale code snippets, line references and factual errors across the
  documentation (settings defaults, CLI requirements, model loading API,
  example settings).

## [0.22.1] 15.08.2026

### Fixes
- Remove the dimension-dependent `pl.len()` factor from `calc_diff`.
  Multi-dimensional candidates are now scored using `abs(diff) ** 1.5`
  instead of receiving an automatic boost based on the number of bin rows.
  This makes 1D, 2D and 3D candidate scores comparable and prevents sparse
  3D combinations from being selected solely because they contain more bins.
- Update the `calc_diff` regression expectation for the corrected score scale.

### Debugging
- Add opt-in debug dumps for candidate diff tables, count/group samples and
  neutral cubes via `PILZ_DEBUG_NEUTRAL_DIR`.

## [0.22.0] 15.08.2026

### Fixes
- Multi-dimensional splits now evaluate every feature combination once per
  repetition, keeping all features aligned to the same count/group split
  instead of combining candidates from different repetitions.
- Preserve the median selection for single-feature and combined candidates
  while retaining the per-repetition data needed for correctly aligned joint
  count tables.

### Changes
- Document the repetition behavior across `n_dims`, its interaction with
  `calcs_per_dim`, and the associated training-time trade-offs.
- Add regression coverage for multi-dimensional combinations with multiple
  repetitions.

## [0.21.4] 13.08.2026

### Fixes
- `CatsMixin.get_sympy_from_bits`: the "all but one bit" optimization now only
  emits `Not(<bin>)` when the excluded bin is the missing bin (which renders
  as `IS NOT NULL`) or when no missing label is configured. Negating a plain
  bin is unsound under SQL three-valued logic: `NOT(bin)` evaluates to `NULL`
  (not `TRUE`) for `NULL` rows, so a cube slice whose intent includes the
  missing bin silently dropped rows with missing values, which then matched no
  spore and scored exactly 0


## [0.21.3] 12.08.2026

### Fixes
- `cat_cater_impl`: the missing sentinel now gets a dedicated label (`__missing__<bin>`) when it shares a quantile bin with regular categories, so the `is null` filter no longer silently drops the other categories in that bin
- `CatCats.get_bits`/`get_single_filter`: bit positions and single-bin filters are resolved by the label's position within `all_labels` instead of its numeric value, decoupling them from the (non-deterministic) `set` ordering
- `CatCats`/`ContCats.all_labels`: the missing label is only appended when it does not already occur among the regular bin labels, preventing duplicate labels
- `ContCats.cut`: the binned series is cast to plain strings before assigning the missing label, so missing values are always mapped even when `pl.cut` returns an `Enum` series (polars >= 1.41) where `fill_null` would silently do nothing
- `CategorizedFeatureMixin.get_left_right_filter`: the missing label is `fill_null`-ed when joining the target/non-target count tables, so the missing bin is scored normally instead of falling into the neutral branch


## [0.21.2] 09.08.2026

### Fixes
- `TrainDataframes.score`: a node with a single row (`sum == 1`) now uses a correction factor of `0.5` instead of the log formula, which would otherwise evaluate to the maximum factor `1.0` (`log(1) == 0`); single-row leaves are now weakened the most


## [0.21.1] 09.08.2026

### Fixes
- Revert the deterministic tie-breaking in the categorical binning sort (`cat_cater_impl`): categories are sorted again only by weight (descending) and by target rate (ascending). The two-key sorts added in 0.21.0 are removed because training is inherently non-deterministic (random data selection), so the extra sort keys only changed binning order without improving reproducibility


## [0.21.0] 09.08.2026

### Changes
- Categorical features (`cat_cater_impl`) treat null rows as a regular category in the binning via an internal null sentinel (`MISSING_SENTINEL`) instead of excluding them from the group-by; the missing bin can now be merged with other categories by the quantile binning
- Mapping keys of `CatCats` are always strings: the feature column is cast to `Utf8` in `CatCats.cut` before applying the mapping, so numeric categorical features (e.g. MNIST pixels) stay consistent
- Deterministic tie-breaking when sorting categories by weight / target weight
- `ContCats.get_single_filter`: a feature whose group set is entirely null (no cuts learned) now renders its value bin as `is not null` instead of raising an `IndexError`; combined with the existing `is null` missing bin this yields a clean "has value vs missing" two-way split

### Fixes
- The binning (and therefore `calc_diff`) is now invariant to whether missing rows are represented as `NULL` or imputed with `missing_value`: identical rows produce identical category groupings and scores
- `CatCats.cut` no longer fails on null-dtype columns; the `Utf8` cast resolves the invalid `i64 -> null` conversion for all-null count sets
- `dataframes.set_diff_df`: use `how="full"` instead of the deprecated `how="outer"` join argument (polars >= 0.20.29)


## [0.20.2] 08.08.2026

### Fixes
- `darkwing.get_df_from_files`: read multiple CSV/parquet files with `pl.concat` instead of `pl.read_csv(",".join(files))` (polars does not accept a comma-joined string)
- `train.cont_cater_impl`: normalize cumulative weights by the non-null weight sum so quantile cuts work for features where null rows carry significant weight (previously `IndexError: index 0 is out of bounds`)
- `dataframes.get_left_right_filter`: treat bins with `diff == 0` as neutral instead of non-target when `neutral_faktor` is 0 (matches the documented three-way split semantics)
- `train.counter`: return `(None, None, None)` when no feature survives categorization instead of raising `IndexError`, so the node becomes a leaf
- `pilz.get_where_sql`: render spores without cuts as `WHEN TRUE` so a root leaf with an empty cut list produces valid SQL


## [0.20.1] 08.08.2026

### Changes
- Add docstrings to the public modules (cli, datacard, settings, dataframes, filter, pilz, darkwing, datacardhelper, eval, thresholds, train, mermaid_fence)
- Split the monolithic `tests/test_models.py` into per-module test files with shared fixtures, and add tests for the CLI, training, eval, darkwing, thresholds and datacardhelper


## [0.20.0] 08.08.2026

### Added
- Missing values in features without `missing_value` are now treated as a separate "missing" category during categorization (detected per subsample), producing `IS NULL`/`IS NOT NULL` conditions in the trained spores
- `IS NULL`/`IS NOT NULL` SQL support in the filter layer

### Changes
- Features with `missing_value` set are still imputed at load time; features without it keep nulls and learn an explicit missing category instead


## [0.19.0] 06.08.2026

### Changes
- Move `train_files`/`test_files` from the DataCard into the settings: `train_files` (and optional `eval_files`) in `TrainSettings`, `test_files` in `EvalSettings`
- Thresholds during training are computed on `eval_files` (falls back to `train_files` if empty)


## [0.18.0] 04.08.2026

### Added
- Compute threshold files (youden and f1) at the end of the training step on the train data

### Changes
- Eval only reads the threshold files from the training folder (selected via different_target_pilz_comb_method), thresholds are not computed in eval anymore
- Add max back as different_target_pilz_comb_method (highest score, no threshold file)

## [0.17.4] 04.08.2026

### Added
- Combine different target pilze with max and youden as optional argument

## [0.17.3] 02.08.2026

### Added
- Combine same target pilze with max as optional argument

### Fixes
- Bug for eval with colum names conatining - 

## [0.17.2] 30.07.2026

### Changes
- Change weakening factor to log scale

## [0.17.1] 30.07.2026

### Added
- Add weakening factor for score for low popopulated spores

## [0.17.0] 29.07.2026

### Added
- Add n_rep which repeats the cut calulation to avoid statisccal fluctations


## [0.16.2] 07.06.2026

### Updates
- rm use_neutral_state cli arg
- disable simplfy usage
- Calc Youden index for multiclass result

## [0.16.1] 12.05.2026

### Updates
- Add missing files

## [0.16.0] 12.05.2026

### Updates
- Docs now in a good shape


## [0.15.3] 10.05.2026

### Updates
- update docs

## [0.15.2] 27.04.2026

### Added
- update docs

## [0.15.1] 26.04.2026

### Added
- KI generated documentation
- Fix links

## [0.15.0] 22.04.2026

### Added
- add churn example
- neutral state calulations


## [0.14.0] 12.03.2026

### Added
- implement infer (same as evel except no label is given in data)
- implemnet output file for eval/infer 
- added keep extra columns for evel/infer
- no hist plots (they need more work)


## [0.13.1] 09.03.2026

### Fixed
- Fix Homepage Path

## [0.13.0] 04.03.2026

- Rename to Pilz


## [0.12.0] 25.01.2026

- cli for creating data card
- add support for parquete files
- handle missing values 


## [0.11.0] 05.01.2026

### Added

- add neutral state for low populated states (optionla feature)
- add espresso minmizer for smaller queries
- introduce batched eval, since larger where clauses are not good for duckdb


## [0.10.0] 10.12.2025

### Changed

- use Sympy to handle binary logic (very slow but works, could also simplify numerical expressions but then even more slow)
- Bank Term Deposit example (many cat features)

### Added
- Works with Categorical features


## [0.9.1] 07.10.2025

### Added
- First version of CICD pipeline

### Changed

- make eval folde to list

## [0.9.0] 08.09.2025

### Added

- Initial version of project
- Works with continus features only
- Example version runs with mnist 
- No docmention so far
