Metadata-Version: 2.5
Name: tickbloom
Version: 0.7.3
Summary: Market-data integrity scoring and look-ahead detection for quantitative Python.
Project-URL: Homepage, https://tickbloom.com
Project-URL: Documentation, https://tickbloom.com
Project-URL: Source, https://github.com/tickbloom/tickbloom
Project-URL: Issues, https://github.com/tickbloom/tickbloom/issues
Author: Tickbloom
License-Expression: MIT
License-File: LICENSE
Keywords: audit,backtesting,data-quality,look-ahead,market-data,quant,quantitative-finance,survivorship-bias,trading
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Financial and Insurance Industry
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Office/Business :: Financial :: Investment
Classifier: Topic :: Software Development :: Quality Assurance
Classifier: Typing :: Typed
Requires-Python: >=3.10
Requires-Dist: pandas>=2.0
Provides-Extra: all
Requires-Dist: databento>=0.34; extra == 'all'
Requires-Dist: fpdf2>=2.7; extra == 'all'
Requires-Dist: pyarrow>=14.0; extra == 'all'
Provides-Extra: databento
Requires-Dist: databento>=0.34; extra == 'databento'
Provides-Extra: dev
Requires-Dist: build; extra == 'dev'
Requires-Dist: fpdf2>=2.7; extra == 'dev'
Requires-Dist: pyarrow>=14.0; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: twine; extra == 'dev'
Provides-Extra: parquet
Requires-Dist: pyarrow>=14.0; extra == 'parquet'
Provides-Extra: pdf
Requires-Dist: fpdf2>=2.7; extra == 'pdf'
Description-Content-Type: text/markdown

# tickbloom

Data integrity you can put in a document.

Runs entirely in your own process. Your data, your strategy code, and your fills never leave the machine.

## Quickstart

```python
import tickbloom as tb

df  = tb.load("ticks.csv", "2024-Q3")       # or source="databento"
rep = tb.audit(df, code_path="strategies/") # data + look-ahead in one call
tb.export(rep, "report.html")               # or fmt="pdf" / "json"
```

```python
import pandas as pd
import tickbloom as tb

df = pd.read_csv("es_2024q3.csv")           # bring your own frame if you prefer

rep = tb.audit(df, source="databento:ES.c.0")
print(rep)                    # <AuditReport score=92.6 verdict=review fail=0 flag=2>
print(rep.breakdown.table())  # where every deducted point went
rep.to_json("audit.json")     # reproducible, byte-stable
```

```
check          weight      rate      tol    -pts
------------------------------------------------
gaps               25   0.5000%    2.00%    5.53
duplicates         10   0.1000%    0.50%    1.81
order              15   0.0000%    0.10%    0.00
lookahead          25structural        —    0.00
survivorship       10structural        —    0.00
session            10   0.0000%    0.50%    0.00
zero_volume         5   0.0000%    1.00%    0.00
------------------------------------------------
SCORE                                           92.6
```

## What the score promises

Three properties, each with a test in `tests/test_scoring.py`:

1. **Deterministic** — same frame, same config, same float. No sampling, no wall-clock, no dependence on column or dict ordering.
2. **Decomposable** — `score = 100 - sum(penalties)`. Every point is attributable to one check. When an allocator asks where 7.4 points went, you print the table.
3. **Explainable in one sentence per check** — if a weight can't be justified to a trader in one sentence, the weight is wrong.

Weights are declared in `scoring.WEIGHTS`, recorded in every manifest, and overridable. That's deliberate: a fund should be able to say *"we used these thresholds"*, not *"the vendor decided."*

## Weights and why

| Check | Weight | One-sentence rationale |
|---|---|---|
| `lookahead` | 25 | Directly fabricates returns; a single leaked bar can turn a losing strategy into a plausible winner. |
| `gaps` | 25 | Missing sessions silently change the sample — a gap across a crash removes exactly the periods that set your tail risk. |
| `order` | 15 | Out-of-order timestamps break the causality assumption every feature is built on. |
| `survivorship` | 10 | Delisted names removed from the universe inflate equity returns 1–4% annually. |
| `duplicates` | 10 | Double-counted prints bias volume-weighted features and can double-fill in replay. |
| `session` | 10 | Overnight prints leaking into a regular-hours frame contaminate open/close logic. |
| `zero_volume` | 5 | Phantom prints move indicators without being tradeable, but rarely dominate a result. |

## The penalty curve

```
penalty = weight * (1 - exp(-defect_rate / tolerance))
```

Not `weight * min(1, rate/tolerance)`. The linear clip was the first implementation and it was wrong: in a 1,000-row sample a **single** out-of-order tick is a 0.1% rate, which equals the tolerance exactly and deducted the entire 15-point weight for one bad tick. Small samples were being destroyed by single defects.

Published reference points, asserted in tests:

| Defect rate | Share of weight deducted |
|---|---|
| 0.25 × tolerance | 22% |
| 1.0 × tolerance | 63% |
| 3.0 × tolerance | 95% |
| 5.0 × tolerance | 99% (treated as full) |

Structural checks (`lookahead`, `survivorship`) stay binary. A "small" look-ahead leak is not a thing.

## Look-ahead scanning

```bash
python -m tickbloom scan strategies/          # exit 1 if a certain leak exists
python -m tickbloom audit data.csv --code strategies/ -o report.html
```

A static AST pass over your source. Eight rules, each declaring a **confidence**, because a static analyser dies from false positives rather than missed detections:

| Confidence | Severity | Meaning |
|---|---|---|
| `certain` | `fail` | A leak by definition. `close.shift(-1)` is tomorrow's close; there is no other reading. |
| `likely` | `flag` | Usually a leak, but has legitimate uses — label construction, offline research. |

Only `certain` rules can fail an audit. A false `fail` costs you the user; a false `flag` costs them four seconds.

Forward-looking code is *correct* when building labels, so suppress per line:

```python
df["target_1d"] = close.shift(-1)   # tickbloom: allow
df["target_1d"] = close.shift(-1)   # tickbloom: allow TB-101
```

Without this, every supervised-learning codebase reports dozens of findings on its target construction and the tool gets uninstalled on day one.

**Limits, stated plainly.** This is a syntactic pass. It cannot see through a variable holding a shift amount, a leak inside a library you call, or one assembled at runtime. It catches the common written-down forms. TB-108 does one-pass tracking of names bound to `.shift(n>0)` so it stays quiet on code that lagged correctly upstream — but it is reported as `likely`, not `certain`, precisely because that analysis is shallow.

## Exchange calendars

```python
tb.audit(df, venue="XNYS")                            # or XNAS, ARCX, GLBX
tb.trading_days("2024-01-01", "2024-12-31")
```

**Pass `venue=` or your score will be wrong.** Without it the gap check falls back to
business days, and every public holiday is a weekday — a perfect July 2024 dataset
scored 77.8 because the market being shut on 4 July read as a missing session.

Calendars are computed from rules, not a bundled table: holidays are defined as "the
third Monday in January", so nothing goes stale and future years resolve. Verified
against published NYSE dates for 2024-2026 and the real trading-day counts. Saturday
holidays observe Friday, Sunday observes Monday, Good Friday tracks Easter.

Rules can't know about *unscheduled* closures, so Hurricane Sandy and the days of
mourning sit in an exception table and `trading_days()` accepts `extra_closures=`.

## Splits and contract rolls

```python
tb.check_splits(df)                    # runs inside audit() by default
tb.adjust_splits(df, split_table)
```

An unadjusted 2-for-1 split halves the price overnight. Every return-based feature
reads a -50% day that never happened — momentum flips, volatility blows up, stops fire
on a move nobody experienced.

No corporate-actions feed needed, because a split leaves a signature no market move
imitates: it lands on an **exact** ratio. A stock falling on bad news closes at 0.4972
of the prior close, essentially never at 0.5000. The detector looks for overnight moves
above 15% landing within 0.5% of a small rational number.

That band is tight on purpose. Half the tests assert clean results on ordinary
volatility, a real 48% crash, and a 1.2% near-miss.

`adjust_splits()` uses a table **you** supply and will not consume `check_splits()`
output — detection is a suspicion, not a fact, and applying a guessed ratio rewrites
your prices to something equally wrong.

Contract rolls are reported as a **flag**, never a failure: on price alone a genuine
gap event is indistinguishable from a badly-stitched roll.

## Survivorship

```python
tb.check_survivorship(df)        # runs automatically on multi-symbol frames
```

Point-in-time constituents are expensive and most people don't have them. You don't
need them to know something is wrong: in any real multi-year universe some symbols stop
trading, roughly 3% a year. If every symbol in a ten-year sample runs to the final date,
the universe was filtered on membership *today* — the backtest is buying companies
already known not to have failed.

`reconstruct_universe(df, delistings)` merges a table you supply. It does **not** invent
names or prices; a symbol with no final price is reported, not fabricated. Detection is
not reconstruction, and that difference is the honest part.

## Loading

```python
tb.load("ticks.csv")                          # CSV / TSV, columns auto-mapped
tb.load("ticks.csv", "2024-Q3")               # filtered to the period
tb.load("ES.c.0", "2024-Q3", source="databento")
```

Everything lands on one normalized schema — `ts_event` (UTC), `symbol`, `price`, `size` — because every check downstream assumes exactly that shape. Extra columns you supplied are kept, never dropped.

Timestamps that mix precisions in one column — `14:00:00` alongside `14:00:00.250000`, which is how tick data serialises to CSV — are parsed correctly. pandas infers a single format from the first row, which silently rejected 75% of a real tick file before this was fixed.

Column names are auto-mapped from what real exports actually contain (`Datetime`, `Px`, `Qty`, `Ticker`). Unparseable timestamps **raise** rather than coerce to `NaT`: silently dropping them would produce a clean-looking audit that verified nothing.

Periods accept `2024-Q3`, `2024-H1`, `2024`, `2024-07`, `2024-07-01`, or `2024-07-01:2024-09-30`. End bounds are inclusive of the whole final day.

Add a provider in about thirty lines — no subclassing, no imports from us:

```python
class MyLoader:
    name = "myprovider"
    def fetch(self, symbol, start, end, **kw):
        return tb.normalize(fetch_somehow(symbol, start, end), symbol=symbol)

tb.register_loader(MyLoader())
tb.load("ES", "2024-Q3", source="myprovider")
```

## Install extras

```bash
pip install tickbloom                 # CSV + the full integrity suite
pip install 'tickbloom[parquet]'      # adds pyarrow
pip install 'tickbloom[pdf]'          # adds fpdf2 for PDF reports
pip install 'tickbloom[databento]'    # adds the Databento client
pip install 'tickbloom[all]'
```

`pyarrow` is ~40MB and only Parquet needs it, so it's an extra rather than a tax on every install.

## Proving causality at runtime

The scanner reads your code. This runs it.

```python
def build_features(df):
    df["sma"] = df["price"].shift(1).rolling(20).mean()
    return df

rep = tb.verify_causal(df, build_features)
print(rep.summary())
```

A feature is causal if and only if its value at time `t` doesn't change when every
row after `t` is deleted. So that's what this does — recompute on truncated data and
compare. **A mismatch is proof of a leak**, not a suspected pattern.

It catches what the AST pass explicitly cannot:

```python
n = -1
df["leak"] = df["price"].shift(n)   # invisible to static analysis
```

```
tested 5 cut points on 300 rows
  causal:  sma
  LEAK:    leak — value at row 90 changed when future rows were removed (5545.5 -> nan)
```

`replay(df)` gives the structural version — each window physically cannot contain
future rows, so a leak is impossible by construction.

**A clean result is not proof of causality.** Truncation is data-dependent:
`price / price.max()` is unambiguously a leak, but if the maximum falls early in the
sample — as in any mean-reverting series — deleting later rows changes nothing and
the leak stays invisible. The static scanner flags it structurally regardless. Run both;
neither is sufficient alone, and there's an explicit test in the suite asserting this
false negative rather than pretending it away.

There is deliberately **no** "wrap the frame and intercept reads" guard. For vectorised
pandas, `close.rolling(20).mean()` is one whole-array operation — there's no moment at
which the library is "computing row t" for a wrapper to observe. Anything claiming
otherwise is either doing nothing or making you rewrite row-by-row.

## Proposing fixes

```python
from tickbloom import agent

items = agent.propose(tb.scan("strategies/"))
for p in items:
    agent.regress(p, df)          # re-verify with the patch applied
print(agent.summary(items))
```

```
[refused] TB-103: a global aggregate must become a rolling or expanding window,
          and choosing the window length is a modelling decision, not a mechanical fix
[P-002] strategies/signal.py - 2 line(s), from TB-101, TB-102
  regression: STILL LEAKING - norm
```

Deterministic, not model-generated. A look-ahead leak has one exact repair —
`shift(-1)` becomes `shift(1)` — and handing that to a sampler would put
non-determinism inside a tool selling reproducible evidence.

Three constraints:

- **Dry run.** `propose()` never writes. `agent.apply(p)` is a separate call, backs
  up to `.tickbloom.bak`, and refuses a stale diff if the file changed underneath it.
- **Regression-backed, including bad news.** If the patch leaves a leak it says
  `STILL LEAKING` and names the column. An agent that only surfaces flattering
  changes is a liability.
- **Scope refusal.** Findings needing a modelling decision — window length, train/test
  split, bar convention — are declined with a stated reason rather than guessed at.

## Trade documentation

```python
rep = tb.audit(ticks, trades="orders.csv")
tb.export(rep, "bundle.html")     # or fmt="pdf" / "csv" / "json"
```

Order log, slippage attribution, realised P&L, equity curve with drawdown, and an
outlier register showing which fills still lack a written reason.

**The reference price is required, not defaulted.** "3.25 ticks of slippage" means
different things measured against the mid, the touch, or the price when your signal
fired. Most tools quietly pick one. This one makes you choose, records it in the
manifest, and prints it on the document — so a reviewer can disagree with the
convention instead of silently misreading the number. An order log with no reference
column is **refused**, because reporting zero slippage would be a false clean result.

Position accounting handles reversals: a `SELL 5` against a long 2 closes two and opens
a short three. Getting that wrong silently corrupts the equity curve.

## Performance

| Stage | Before | Now |
|---|---|---|
| **Full audit, 2.41M ticks** | 4.14s | **1.07s** |
| `check_gaps` | 1.99s | 0.12s |
| `check_duplicates` | 1.05s | 0.42s |
| corporate detection | 2.31s | 0.31s |

Measured on a 187-session ES tick frame. With categorical symbols (the `load()`
default) the full audit is **0.73s**.

### Memory

| | Frame | Audit peak |
|---|---|---|
| string symbols | 180.7 MB | 139.8 MB |
| **categorical (default)** | **60.2 MB** | 139.8 MB |

`load()` stores `symbol` as a categorical. It's low-cardinality by nature, so a
string dtype keeps a pointer per row — on a tick frame that one column is two thirds
of the total. Projected to a 4-year ES dataset (~50M rows): **6.6 GB → 2.8 GB peak**,
which is the difference between OOM and fine on a 16 GB laptop.

Opt out with `load(..., categorical_symbols=False)`.

Converting inside the checks instead was measured and was *worse* — hashing millions
of strings costs more than the copy it avoids. Pay it once, at load.

The common cause was `.dt.date` — a per-row Python object constructor that dominates
whatever else the function does at tick scale. `.dt.floor("D")` is vectorised, and
`.unique()` collapses to the few values actually needed.

Duplicates use a hash prefilter, but the hash only narrows candidates — **equality
still decides**. A 64-bit hash over millions of rows collides at roughly 1e-7, and a
false duplicate report from an integrity tool isn't worth 0.2s.

Regression tests assert the *number of session aggregations*, not only elapsed time:
a wall-clock threshold is unreliable on shared CI, but "did this run twice" is exact.

## CI

```bash
tickbloom ci data/ticks.csv --venue XNYS --code strategies/ \
  --fail-under 90 --baseline .tickbloom-baseline.json --no-regress \
  --junit junit.xml --summary "$GITHUB_STEP_SUMMARY"
```

| Exit | Meaning |
|---|---|
| 0 | passed |
| 1 | below `--fail-under`, or regressed against the baseline |
| **2** | **a new blocking finding since the baseline** |
| 3 | usage error |

2 is separate from 1 on purpose: "this PR introduced a look-ahead leak" and "the
score drifted down" need different responses from whoever is reading the failure.

**Baseline mode is what makes this survivable.** A repo starting at 82 shouldn't fail
forever; it should fail when it gets worse. And a failing run never writes the
baseline — storing a failing score as the new bar lowers the standard every time
someone breaks it.

Copy `.github/workflows/tickbloom.yml` into any repo that backtests. It caches the
baseline per branch so a PR is compared against main, and only main is allowed to move
the bar.

## Configuration

`tickbloom.toml`, found by walking upward like git:

```toml
[scoring.weights]
lookahead = 30
gaps      = 20

[trades]
reference  = "arrival"
sigma      = 3.0
multiplier = 50
```

The config path is recorded in the manifest. That's what makes *"we used these
thresholds"* verifiable rather than a slogan. Weights must still sum to 100 — the score
is "100 minus deductions", so any other total makes it uninterpretable and incomparable
between runs. Setting a tolerance on a structural check is refused: a leak either exists
or it does not.

## Sending a report

```bash
python -m tickbloom audit their-file.csv -o report.html --source "ES · 2024-H1"
```

Produces a self-contained HTML file — no external assets, no scripts, no network — that opens from an email attachment on a locked-down laptop. The column mapper handles `Datetime`/`Px`/`Qty`/`Ticker` and the usual variants, so nobody has to rename anything first.

```python
rep = tb.audit(df)
tb.export(rep, "report.html")               # sendable document
tb.export(rep, "report.json", fmt="json")   # machine-readable
```

Reports are **byte-reproducible**: same input, same bytes out. No timestamp is stamped unless you pass `generated=` explicitly. A document whose bytes change when nothing changed is not evidence, and `test_html_is_byte_reproducible` fails the build if that regresses.

`--fail-under 95` exits 1 below a threshold, so the CLI composes into CI today.

## Status

Implemented: provider loaders, the five integrity checks, the scoring model, the static look-ahead scanner, runtime causality verification, PDF export, the robustness agent, the report object, the reproducibility manifest, HTML/JSON export, and the CLI.

Not yet: model-backed agent proposals for open-ended changes. Not claimed on the site.

## Tests

```
python -m pytest tests/ -q     # 69 passed
```

## A note on scope

This does not replace QuantConnect or Lean. It sits upstream of whatever backtester you already use, and it does not assert compliance with, certification by, or approval from any trading firm. It produces evidence. A human draws the conclusion.
