Metadata-Version: 2.4
Name: hmft
Version: 0.4.0
Summary: Estimate how long an LLM call will take before you send it, plus a live ETA while it streams. Fully client-side.
License-Expression: Apache-2.0
Project-URL: Homepage, https://hmft.dev
Project-URL: Repository, https://github.com/hmft-dev/hmft
Project-URL: Changelog, https://github.com/hmft-dev/hmft/blob/main/CHANGELOG.md
Project-URL: Issues, https://github.com/hmft-dev/hmft/issues
Keywords: llm,latency,estimate,eta,streaming,tokens,tokenizer,openai,anthropic,benchmark
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: tiktoken>=0.7
Requires-Dist: httpx>=0.25
Provides-Extra: openai
Requires-Dist: openai>=1.30; extra == "openai"
Provides-Extra: anthropic
Requires-Dist: anthropic>=0.30; extra == "anthropic"
Provides-Extra: dev
Requires-Dist: pytest>=8; extra == "dev"
Requires-Dist: openai>=1.30; extra == "dev"
Requires-Dist: anthropic>=0.30; extra == "dev"
Dynamic: license-file

# hmft

Estimate how long an LLM call will take **before you send it**, and get a live ETA while it
streams. Runs entirely on your machine: no prompt or response text ever leaves the process.

```bash
pip install hmft                      # from PyPI
pip install "hmft[openai,anthropic]"  # with the SDK wrappers
pip install -e ".[dev]"               # from a clone, with the test extras
```

## Quickstart

No API key, no network call: the estimate is computed locally from the prompt.

```python
import hmft

est = hmft.estimate(
    [{"role": "user", "content": "Explain how bicycles stay upright in two paragraphs."}],
    model="gpt-4.1-mini", provider="openai",
)
print(est)                                      # ~2.3s (p90 4.6s): ttft 0.69s, ~220 tokens [measured]
print(est.total_s.p50, est.total_s.p90)         # 2.27 4.58
print(est.output_tokens.p50, est.output_tokens.p90)  # 220 352
```

Every estimate is a **band, not a point**: `p50` is the typical case, `p90` is the slow-but-likely
case you should plan around.

## Accuracy on held-out prompts

186 rows from prompt sets A to D, four models, re-scored under the current code with
`scripts/compare_runs.py`. "Coverage" is the share of rows whose actual landed at or under
the estimated p90; the target is about 90%.

**What "held out" means here, precisely.** These 186 rows are held out from **v2** constant
fitting: `compare_runs.py` marks sets A to D held-out and fits only on the later sets (E, F).
They are *not* untouched by v1. Set A is the v1 calibration set, and four v1.1 constants were
fitted or corrected using rows that are in this table — `TOKENS_PER_LIST_ITEM` (set B
actuals), `TOKENS_PER_ONE_LINE_ITEM_P50/P90` (A, B and C), the spoken-duration cue
`490 + 59 × minutes` (C and D), and the `code_large` band (the `long_code` actuals in A, B
and C). Read the table below as a standing regression score, not as a clean out-of-sample
result. The cleanest out-of-sample evidence is set E, measured on different subjects, where
the duration cue covered 29 of 30 rows.

| model | held-out rows | token p90 coverage | time p90 coverage |
|---|---|---|---|
| gpt-4.1-mini | 46 | 100% | 98% |
| claude-haiku-4-5 | 47 | 96% | 100% |
| claude-sonnet-4-5 | 47 | 100% | 94% |
| gpt-5@low | 46 | 100% † | 100% † |
| **all non-reasoning** | **140** | **99%** | **97%** |

† **gpt-5@low covers every row, but its token estimate runs low.** Under the 0.3.0 row,
fitted on set E and set F runs, the median actual / estimate is 1.01 for time and 1.40 for
tokens: the hidden-token median comes from set F's short prompts, and the p90 side absorbs
the gap. Under the 0.2.x row the same 100% was an over-estimate of time (0.55) cancelling
an under-estimate of tokens. See `RESULTS.md`.

All four time-coverage breaches across the 140 non-reasoning rows were caused by time to
first token, not by the length heuristics; the token estimate was inside p90 in every one.

### The current models (0.4.0)

Six more specs were measured on 2026-09-19, on the same held-out sets A to D, 46 rows each.
These rows are held out from the fit in the strict sense: every constant behind them, the
verbosity multiplier included, comes from sets E and F only.

| spec | held-out rows | token p90 coverage | time p90 coverage | first token p90 coverage |
|---|---|---|---|---|
| `claude-sonnet-5@high` | 46 | 98% | 98% | 80% |
| `claude-sonnet-5@medium` | 46 | 98% | 98% | 89% |
| `gpt-5.6-sol@medium` | 46 | 96% | 89% | 78% |
| `gpt-5.6-sol@low` | 46 | 96% | 96% | 87% |
| `claude-opus-5@high` | 46 | 87% | 91% | 70% |
| `claude-opus-5@medium` | 46 | 93% | 98% | 85% |

**Two things to know before you use these.**

**`claude-opus-5@high` clears the bar and still runs about 30% short at p50.** Its median
actual / estimate is 1.30 on tokens and 1.28 on time: a typical call takes about 30% longer and
writes about 30% more than the p50 says, and its token coverage clears the 85% bar by two
points, at 87%. The cause is thinking depth: the first-token band is built on a hidden-token
median of 92 from the fit prompts, where the held-out prompts sit at 184. Use its p90, not its
p50, and see `RESULTS.md` for the per-task breakdown.

### Time to first token is below target on the reasoning models

First-token coverage, held out, every measured spec in one place:

| spec | first token p90 coverage | on its own fit rows |
|---|---|---|
| `gpt-4.1-mini` | 93% | |
| `claude-haiku-4-5` | 79% | |
| `claude-sonnet-4-5` | 81% | |
| `gpt-5@low` | 100% | |
| `claude-sonnet-5@high` | 80% | 89% |
| `claude-sonnet-5@medium` | 89% | 91% |
| `gpt-5.6-sol@medium` | 78% | 88% |
| `gpt-5.6-sol@low` | 87% | 88% |
| `claude-opus-5@high` | **70%** | 88% |
| `claude-opus-5@medium` | 85% | 88% |

**Three of the 0.4.0 specs are below the 85% target: `claude-opus-5@high` at 70%,
`gpt-5.6-sol@medium` at 78% and `claude-sonnet-5@high` at 80%.** The older non-reasoning models
sit at 79 to 93% on the same measure, so this is not new, but on a reasoning model the cause is
different and the gap is larger.

**The gap is structural, not an unlucky sample.** All six 0.4.0 specs land at 88 to 91% on their
own fit rows, where the p50 is that set's own median. Held-out prompts make these models think
longer than the fit prompts did, and a longer think pushes the first token past p90 while the
total still lands, because the thinking time comes out of the generation that follows.

**Nothing was tuned against the held-out rows to lift it.** The rows are fitted on sets E and F
only and first token is reported, never fitted to; lifting it by widening the band until the
held-out rows fit would be fitting on the set that exists to check the fit. The fix is a fit set
whose thinking depth spans what real prompts do, which is recorded as a backlog item in
`docs/v2-per-model-constants.md`.

Plan around the total, not around the first token.

## Supported models

| model | provider | status | basis |
|---|---|---|---|
| `gpt-4.1-mini` | openai | measured | 10 benchmark samples; 46 held-out rows |
| `claude-haiku-4-5` | anthropic | measured | 10 benchmark samples; 47 held-out rows |
| `claude-sonnet-4-5` | anthropic | measured | 40 benchmark samples; 47 held-out rows |
| `gpt-5@low` | openai | experimental | 52 fit rows (set E runs 2-3, set F runs 1-2); 46 held-out rows, token estimate runs low |
| `gpt-5@medium` | openai | experimental | 10 benchmark samples; no held-out rows |
| `gpt-5-mini` | openai | experimental | 10 benchmark samples; no held-out rows |
| `gpt-4.1` | openai | experimental | 10 benchmark samples; no held-out rows |
| `claude-sonnet-5@high` | anthropic | measured | 44 fit rows (set E, set F twice); 46 held-out rows |
| `claude-sonnet-5@medium` | anthropic | measured | 44 fit rows (set E, set F twice); 46 held-out rows |
| `gpt-5.6-sol@medium` | openai | measured | 42 fit rows (set E, set F twice); 46 held-out rows |
| `gpt-5.6-sol@low` | openai | measured | 42 fit rows (set E, set F twice); 46 held-out rows |
| `claude-opus-5@high` | anthropic | measured | 42 fit rows (set E, set F twice); 46 held-out rows, token and time estimates run about 30% short |
| `claude-opus-5@medium` | anthropic | measured | 42 fit rows (set E, set F twice); 46 held-out rows |
| `openai/gpt-4.1-mini`, `openai/gpt-4.1`, `openai/gpt-5-mini`, `anthropic/claude-sonnet-4.5`, `anthropic/claude-haiku-4.5` | openrouter | experimental | 6 to 10 benchmark samples each; no held-out rows |
| any other model | openai, anthropic | provider default | `basis="provider_default"`: token band, and a time from the provider's pooled row |
| any other model | any other provider | not measured | `basis="unmeasured"`: token band only, no timing |

**measured** means the throughput row comes from a real benchmark run *and* the model has
held-out coverage above. **experimental** means the throughput row is real but the estimate
has not been scored against held-out prompts, or was scored and did not pass.

**provider default** is what a model with no row of its own gets on OpenAI or Anthropic. Its
time comes from `openai/_default` or `anthropic/_default`, which the benchmark computes from
that provider's measured non-reasoning models (median p50, slowest slow side), and the estimate
says so: `basis="provider_default"`, `basis_key` naming the `_default` row. That row contains no
thinking time, so for an unbenchmarked reasoning model it is likely to run fast.

**not measured** is every other provider: hmft refuses to invent a number. You get an
output-token band and `basis="unmeasured"`, never a time.

A bare `gpt-5` call with no effort set maps to the `gpt-5@medium` row, the provider default,
and `basis_key` says so. OpenRouter has no provider-wide fallback row, because it routes
across model families; an unbenchmarked slug is `unmeasured`.

## Privacy

**No prompt text and no response text is ever stored, logged, printed or transmitted.**
Nothing is uploaded anywhere, by any code path, in v1. There is no server.

After each wrapped call hmft appends one JSON line to `~/.hmft/telemetry.jsonl`, created
`0600`, so you can score your own accuracy offline. The row is an allow-list of integers,
floats and short enum words — exactly these 28 fields and nothing else:

`ts`, `hmft_version`, `model`, `provider`, `prompt_tokens`, `output_tokens`, `ttft_ms`,
`total_ms`, `est_output_p50`, `est_output_p90`, `est_ttft_p50_s`, `est_ttft_p90_s`,
`est_total_p50_s`, `est_total_p90_s`, `basis`, `reasoning_effort`, `effort_source`,
`reasoning_tokens`, `finish_reason`, `visible_chunks`, `first_second_visible_chunks`,
`last_visible_ms`, `first_event_ms`, `max_gap_ms`, `est_verbosity`, `client_timeout_s`,
`client_max_retries`, `est_prompt_tokens`.

No prompt, no response, no request ids, no user ids. `reasoning_effort`, `effort_source`,
`basis` and `finish_reason` are validated against fixed sets, so free text is dropped rather
than written. `est_verbosity` is null when no fitted verbosity multiplier applied, so it is
never confused with a fitted 1.0. `client_timeout_s` and `client_max_retries` are your SDK
client's read timeout and retry count, read from the client object: a total near the timeout
was cut off, and a retried call's total includes every attempt. `est_prompt_tokens` is hmft's
own count of the prompt, next to the provider's `prompt_tokens`. The write is enforced by an
assertion against the allow-list in `hmft/telemetry.py`, and `tests/test_telemetry.py`
checks it.

Opt out with one environment variable:

```bash
HMFT_TELEMETRY=off                     # write nothing
HMFT_TELEMETRY=/path/to/file.jsonl     # or move it somewhere else
```

## The name

HMFT is short for **"how much fucking time"** — the question you actually ask when a model has
been streaming for forty seconds and you have no idea whether it is nearly done or has barely
started. The package, the import and the docs are all just `hmft`; the long form lives here and
nowhere else.

## How estimates work

```
total_time ≈ TTFT(prompt_tokens, model) + output_tokens / tokens_per_second(model)
```

1. **Output length comes from cues in the prompt** (`hmft/length.py`), in priority order:
   an explicit count ("in one sentence", "about 500 words", "8 questions", "one per line"),
   a stated speaking duration ("a 10-minute keynote" → `490 + 59 × minutes` tokens, validated
   2 to 45 minutes), a forced format (JSON mode, forced tool call, `max_tokens`), and finally
   a task type guessed from keywords (code, list, JSON, yes/no, translate, summarise, essay,
   explain, chat), each with its own base band.
2. **TTFT and tokens/second are measured constants**, per model and provider, in
   `hmft/data/throughput.json`. They are only ever written by `scripts/benchmark.py` from real
   runs — never estimated, never hand-edited. A model with no row falls back to its provider's
   pooled `_default` row (`basis="provider_default"`) on OpenAI and Anthropic; with no row at
   all, the estimator reports `basis="unmeasured"` instead of guessing.
3. **The band comes from both.** The p50 time uses the throughput `p50`; the p90 time uses the
   throughput **`p10`**, because for a rate the slow side is the low percentile. The length
   heuristic contributes its own p50/p90, and the two compose into `total_s`.

Measured throughput, as of the `updated_at` in the table:

| model | TTFT p50 / p90 | tokens/s p50 / p10 | samples |
|---|---|---|---|
| gpt-4.1-mini | 0.69 s / 1.622 s | 139.2 / 119.1 | 10 |
| gpt-4.1 | 0.76 s / 2.229 s | 111.3 / 90.5 | 10 |
| claude-haiku-4-5 | 0.67 s / 0.686 s | 83.4 / 79.5 | 10 |
| claude-sonnet-4-5 | 0.86 s / 1.355 s | 36.8 / 34.6 | 40 |
| gpt-5@low | 3.93 s / 18.11 s | 74.3 / 36.2 | 52 |
| claude-sonnet-5@high | 0.916 s / 1.599 s | 91.5 / 69.6 | 44 |
| claude-sonnet-5@medium | 0.914 s / 1.371 s | 92.9 / 73.0 | 44 |
| gpt-5.6-sol@medium | 1.475 s / 4.641 s | 48.5 / 32.7 | 42 |
| gpt-5.6-sol@low | 1.474 s / 3.46 s | 35.9 / 30.8 | 42 |
| claude-opus-5@high | 2.211 s / 8.306 s | 75.5 / 59.5 | 42 |
| claude-opus-5@medium | 2.049 s / 7.072 s | 75.8 / 59.8 | 42 |

On reasoning models the TTFT column absorbs hidden thinking time, which is why it is large and
why the `gpt-5@low` row is experimental.

The six 0.4.0 rows are measured differently from the rows above them: their samples are fit
rows from real prompt sets, not benchmark calls. The benchmark's own prompt shape made these
models think far harder than a real prompt does (it read a 27.8 s first token for
`gpt-5.6-sol@medium`, where real prompts give 1.5 s), so a row fitted on it over-estimated by
six times. Their tokens/second is an effective visible rate: thinking that happens mid-stream
is charged to it, which is why Sol decodes at 48.5 rather than the 93 the benchmark saw.

For gpt-5 at low effort that means: typical first token about 4 seconds, 9 in 10 under 18
seconds, with provider stalls of 10-43 seconds in roughly 1 call in 10. The hidden-token
median behind it is fitted on set F, whose prompts are short, so on longer, more varied
prompts expect the token estimate to run low.

## Streaming ETA

```python
from openai import OpenAI
import hmft

client = hmft.wrap_openai(OpenAI())
stream = client.chat.completions.create(
    model="gpt-4.1-mini", stream=True,
    messages=[{"role": "user", "content": "Explain how bicycles stay upright in two paragraphs."}],
)
print(stream.estimate)             # pre-call band
for chunk in stream:
    print(stream.eta(), end="\r")  # live: blends the observed rate in after a few tokens
print(stream.result)               # actual vs estimated
```

`hmft.wrap_anthropic(Anthropic())` is the same shape around `client.messages.stream(...)`.
Wrappers are sync-only in v1; async is a v1.x item.

## Filling the throughput table

```bash
export OPENAI_API_KEY=...       # and/or ANTHROPIC_API_KEY, OPENROUTER_API_KEY
python scripts/benchmark.py --all --dry-run    # print the plan, make no calls
python scripts/benchmark.py --provider anthropic --models claude-haiku-4-5 --runs 3
```

Each run sends two synthetic prompts and records only token counts and timings.
`name@effort` (for example `gpt-5@low`) sets `reasoning_effort` and gets its own row.

## Known limits

- Output-length heuristics are rules, not a model. Expect the p50 to be off by 2x on
  open-ended prompts; the p90 band is what to plan around.
- Reasoning models spend most of their wall time on hidden tokens the heuristics cannot see.
- Non-OpenAI token counts use the o200k tokenizer as a proxy, typically within 10 to 20%.
- Image and tool-result content blocks are not counted.
- Talk-duration cues are validated 2 to 45 minutes; above 45 is extrapolation. The p90 held on
  29 of 30 measured talks, but the p50 is biased per model: gpt-4.1-mini writes far less than
  the estimate on long talks, Claude models somewhat more.
- Provider latency drifts hour to hour; `measured_at` tells you how stale a row is.

## Tests

```bash
pip install -e ".[dev]"
python -m pytest
```

All tests use synthetic prompts and fake clients, and the suite makes no network calls —
`tests/test_no_network.py` enforces that. See `CONTRIBUTING.md` for the held-out rule and the
version-bump rule. `BASELINE.md` is the frozen v1 reference and `RESULTS.md` records every
measurement since.

## License

Apache-2.0. See `LICENSE`.
