Metadata-Version: 2.4
Name: turbofastapply
Version: 0.1.1
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: Software Development :: Libraries
Requires-Dist: pandas>=1.5
Requires-Dist: numpy>=1.23
Requires-Dist: pytest>=7.0 ; extra == 'test'
Provides-Extra: test
License-File: LICENSE
Summary: A drop-in, accelerated replacement for pandas .apply()
Keywords: pandas,apply,performance,dataframe,accelerated
Author-email: Ponugoti Sudhakar <ponugotisudhakar3@gmail.com>
Requires-Python: >=3.9
Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
Project-URL: Homepage, https://github.com/P-Sudhakar-tech/FastApply
Project-URL: Repository, https://github.com/P-Sudhakar-tech/FastApply

# TurboFastApply

A drop-in, accelerated replacement for `pandas.apply()`.

Status: **feature-complete** (all 8 roadmap phases) — numeric transforms
detected as linear/abs (`x * 2 + 1`) and row-wise DataFrame functions
that are linear combinations of columns (`row['a'] + row['b']`) run
through genuine native fast paths, ~6.2x and ~4.6x faster than plain
`pandas.apply()` on 1,000 rows respectively (`examples/benchmark.py`).
A dedicated small-data optimization pass (dedup'd array/Index
construction, replacing numpy scalar calls with plain Python arithmetic
in the hot verification path, avoiding redundant column-selection/NaN
scans) also closed a real regression that used to exist from 50 rows
onward — see "Benchmarks: previous vs now" below for exact numbers. A
follow-up pass replaced the row-wise path's real per-row pandas Series
construction (for its correctness-required probe/verification rows) with
a lightweight dict-based stand-in (`_RowView` in `decide_row.py`), cutting
that fixed cost by 3-4x and lowering its `engine="auto"` threshold
(`MIN_ROWS`) from an interim 300 back down to 100 — now a genuine win from
50 rows up, not just parity. See "Benchmarks: previous vs now" for the
full before/after.
Arbitrary callables that don't qualify get a sampling-based threaded
fallback that only engages when actually measured faster (helps I/O-bound
work, correctly declines pure-CPU-bound Python). Everything else falls
back to native `pandas.apply()`, so `.turbofastapply` is always
correctness-equivalent to it. See `claude.md` for the full roadmap,
including three real correctness bugs found and fixed during hardening.

String ops (`str.upper`/`.lower`/`.strip`, `.turbofastapply.str.contains()`/
`.replace()`) are implemented and correct but **not** faster in practice
— benchmarked from 500 to 1,000,000 rows, consistently ~0.6–0.8x plain
pandas — so they're excluded from the default `engine="auto"` and only
reachable via explicit `engine="native"`. See "Options" below.

## Development

The native extension is built with [PyO3](https://pyo3.rs) +
[maturin](https://www.maturin.rs).

```bash
python -m venv .venv
.venv/Scripts/activate        # Windows
pip install maturin pytest pandas numpy
maturin develop                # builds the extension and installs it in the venv
pytest -v
```

If `maturin develop` fails to launch on your machine (e.g. `maturin.exe`
can't load `api-ms-win-crt-*.dll` because only the GNU Rust toolchain is
installed and the MSVC Visual C++ Redistributable is missing), use the
fallback build script instead. It calls `cargo build` directly, drops the
resulting DLL into the package as a `.pyd`, and links `python/` into the
venv via a `.pth` file (same end result as `maturin develop`, so
`import turbofastapply` works from anywhere without setting `PYTHONPATH`):

```powershell
./build.ps1
.venv/Scripts/python.exe -m pytest -v
```

CI (GitHub Actions, `.github/workflows/ci.yml`) runs on Ubuntu with a
properly provisioned toolchain, so `maturin develop` works there regardless.

## Usage

The accessor is directly callable — `s.turbofastapply(func)` is the primary API,
not `s.turbofastapply.apply(func)` (kept only as an alias):

```python
import pandas as pd
import turbofastapply  # registers the .turbofastapply accessor

s = pd.Series([1, 2, 3])
s.turbofastapply(lambda x: x * 2)   # identical result to s.apply(lambda x: x * 2)

df = pd.DataFrame({"a": [1, 2, 3], "b": [4, 5, 6]})
df.turbofastapply(lambda row: row["a"] + row["b"], axis=1)

# Whitelisted string methods and regex ops (.turbofastapply.str mirrors
# pandas' own .str accessor) are implemented and correct, but not
# faster than plain pandas in practice — see the Status note above.
# They use plain pandas by default; engine="native" forces the compiled
# path if you want it anyway:
names = pd.Series(["  Ada  ", "GRACE", "margaret"] * 50)
names.turbofastapply(str.upper, engine="native")
names.turbofastapply.str.contains(r"^GRACE$")

# .groupby(...).turbofastapply(func) also works — a drop-in for
# .groupby(...).apply(func) with the same engine/verbose/progress_bar
# options as everything else. engine="auto" tries a threaded parallel
# fallback first (measured, not assumed — same approach as the
# GIL-releasing-callable case above, just chunked by group), for
# GroupBy objects with at least 800 groups — a real, ~2x+ speedup for
# GIL-releasing callables, verified never to regress a CPU-bound one
# even at that scale (see claude.md's P9 section for why the threshold
# is that high). Below that it falls back to plain pandas immediately,
# no measurement overhead paid. There's no *native* (Rust) GroupBy path
# though, so engine="native" raises rather than pretending to
# accelerate something that doesn't exist yet.
df.groupby("a").turbofastapply(lambda g: g["b"].sum())
```

### Options

```python
s.turbofastapply(func, engine="auto")     # default: try the native fast path, fall back silently
s.turbofastapply(func, engine="pandas")   # always plain pandas .apply(), skips the eligibility check
s.turbofastapply(func, engine="native")   # require the fast path; raises ValueError with the
                                     # specific reason if func isn't eligible, instead of
                                     # silently falling back

s.turbofastapply(func, verbose=True)      # prints which engine was used and why, to stderr

s.turbofastapply(func, progress_bar=True) # progress bar for the pandas-fallback path only —
                                     # the native path is one vectorized call, nothing to
                                     # report progress on there
```

## Benchmarks

`examples/benchmark.py` compares plain `pandas.apply()` against `.turbofastapply()`
on 1,000 rows (median of 50 runs, 5 warmup): the numeric-transform case
(`x * 2 + 1`) lands at **~6.2x** and the row-wise case
(`row['a'] + row['b']`, notoriously slow in vanilla pandas since it
constructs a Series per row) at **~4.6x**. It also runs the string case
both ways — default `engine="auto"` (matches plain pandas exactly, no
native path picked) and forced `engine="native"` — to show directly why
`"auto"` never picks it.

### Benchmarks: previous (PyPI 0.1.0) vs now

The 0.1.0 release published to PyPI had a real regression from small
data up through a few hundred rows — turbofastapply was *slower* than
plain pandas in that range, not just less of a win. A follow-up
optimization pass (not yet released) fixed the redundant work causing
it: deduplicated array/Index construction that used to run twice per
call, `np.isclose()`/`np.round()` scalar calls replaced with plain
Python arithmetic in the hot verification path (measured at ~38% of
`decide()`'s total time by itself), and eliminating a redundant
DataFrame column-selection + a redundant `.isna()` column scan in the
row-wise path. Median of 100 runs (60 for row-wise), 15 warmup, timed
against `turbofastapply==0.1.0` installed from PyPI in a clean venv vs.
the same numbers from this optimization applied locally:

**Numeric path** (`s.turbofastapply(lambda x: x * 2 + 1)`)

| rows | previous (0.1.0) | now | previous speedup | now speedup |
|-----:|------------------:|----:|------------------:|------------:|
| 50 | 0.162ms | 0.053ms | 0.34x (regression) | **1.29x** |
| 100 | 0.161ms | 0.053ms | 0.42x (regression) | **1.29x** |
| 200 | 0.191ms | 0.054ms | 0.53x (regression) | **1.83x** |
| 500 | 0.165ms | 0.055ms | 1.15x | **3.46x** |
| 1,000 | 0.187ms | 0.056ms | 2.06x | **6.20x** |
| 5,000 | 0.194ms | 0.060ms | 9.77x | **25.75x** |

**Row-wise path** (`df.turbofastapply(lambda row: row['a'] + row['b'], axis=1)`)

Three passes. Pass 1 (dedup'd Index construction, reused used-column
arrays) narrowed the regression but never closed it. Pass 2 raised
`MIN_ROWS` from 50 to 300 so `engine="auto"` would decline rather than
engage a losing native call below that size — parity, not a win. Pass 3
replaced the actual cost driver — a real pandas Series built per probe and
verification row — with `_RowView`, a plain-dict stand-in supporting
`row['col']`/`row.col` (everything an affine row function needs) without
pandas' `Series.__init__` overhead; a function needing more of the real
Series API (`row.sum()`, `.values`, ...) safely declines instead of
breaking. That cut the fixed cost 3-4x and let `MIN_ROWS` drop to 100 —
the first size where every trial in an 8-trial sweep cleared 1x:

| rows | previous (0.1.0) | pass 1 | pass 2 | pass 3 (current) |
|-----:|------------------:|----:|----:|----:|
| 50 | 0.41x (regression) | 0.30x (regression) | 1.0x (declines) | **1.0x (declines, matches pandas)** |
| 100 | 0.47x (regression) | 0.82x (regression) | 1.0x (declines) | **~2x (native engages from here)** |
| 200 | 0.86x (regression) | 1.29x | 1.0x (declines) | **~1.5-3x** |
| 500 | 1.96x | 2.03x | ~3x | **~5-7x** |
| 1,000 | 4.25x | 4.58x | ~7x | **~10-20x** |
| 5,000 | 22.25x | 22.84x | ~30x | **~90-100x** |

`engine="native"` still works below 100 rows for anyone who wants the
fast path regardless of profitability (`enforce_min_rows=False`
bypasses the threshold on an explicit request, same as the numeric path).
See `claude.md` for the full investigation.

`cargo bench` (`benches/native_benches.rs`) benchmarks the pure Rust
compute cores directly (1,000 / 50,000 / 200,000 elements), independent
of Python/PyO3 marshaling overhead — useful for catching a regression in
the Rust layer itself.

`examples/benchmark_vs_competitor.py` adds a comparison against another
accelerated-`.apply()` library. It's a separate opt-in script (not a
`turbofastapply` dependency), since that library pulls in dask, tqdm, etc. As of
this writing its 1.4.0 release doesn't run cleanly against recent
pandas/Python — see that script's error message for the specific
compatibility gap if you hit it.

