Metadata-Version: 2.4
Name: mock-pyspark
Version: 0.0.1.dev20260819
Summary: Drop-in mock for Apache PySpark 4.x — Arrow-backed, no JVM required
Author-email: Peter Dowdy <peter.dowdy@gmail.com>
License: MIT
Project-URL: Homepage, https://github.com/PeterDowdy/mock-pyspark
Project-URL: Repository, https://github.com/PeterDowdy/mock-pyspark
Project-URL: Issues, https://github.com/PeterDowdy/mock-pyspark/issues
Keywords: pyspark,mock,testing,arrow,duckdb
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Topic :: Software Development :: Testing
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Requires-Python: >=3.9
Description-Content-Type: text/markdown
Requires-Dist: pyarrow>=14
Requires-Dist: duckdb>=1.0
Requires-Dist: pandas>=2.0
Requires-Dist: numpy>=1.24
Provides-Extra: avro
Requires-Dist: fastavro>=1.9; extra == "avro"
Provides-Extra: delta
Requires-Dist: deltalake>=1.0; extra == "delta"
Provides-Extra: all
Requires-Dist: mock-pyspark[avro,delta]; extra == "all"

# mockpyspark

A drop-in mock for Apache PySpark 4.x for unit and integration testing. Runs entirely in Python — no JVM, no Hadoop, no cluster — using PyArrow as the compute engine and DuckDB for SQL.

```bash
pip install mock-pyspark
```

`mockpyspark` (the PyPI distribution is `mock-pyspark`; you still `import mockpyspark`) installs alongside real `pyspark`; there is no namespace collision. Tests opt in via a one-liner so that `from pyspark.sql import ...` in your production code transparently resolves to the mock.

---

## Quick start

```python
# conftest.py
import mockpyspark
mockpyspark.activate()
```

```python
# tests/test_my_pipeline.py
from pyspark.sql import SparkSession, functions as F

def test_normalize(spark=None):
    spark = SparkSession.builder.getOrCreate()
    df = spark.createDataFrame([
        {"id": 1, "name": "  alice ", "score": 10},
        {"id": 2, "name": "BOB",      "score": 20},
    ])
    out = (
        df.withColumn("name", F.initcap(F.trim(F.col("name"))))
          .filter(F.col("score") > 15)
          .select("id", "name", "score")
          .collect()
    )
    assert out == [(2, "Bob", 20)]
```

No changes to your Spark logic. The alias just makes `pyspark.*` imports resolve to `mockpyspark.*` for the test process.

---

## Version targeting

`mockpyspark` supports both PySpark 4.0.x and 4.1.x. APIs that only exist in 4.1 (e.g. `current_time`, `TimeType`) raise `AttributeError` when running in 4.0 mode.

It picks a target version automatically, in order:

1. `PYSPARK_VERSION` env var
2. The `pyspark` pin in `pyproject.toml`, `requirements*.txt`, `poetry.lock`, `uv.lock`, `Pipfile.lock`, or `setup.cfg` (walking up from the working directory)
3. `SparkSession.builder.config("spark.mock.targetVersion", "4.0").getOrCreate()` — per-session override
4. If nothing is found, defaults to the latest supported version and emits a `UserWarning`. Set `MOCKPYSPARK_STRICT_VERSION=1` to raise instead.

So if your project pins `pyspark==4.0.1` in `pyproject.toml`, the mock will refuse 4.1-only APIs without any configuration on your part.

---

## Usage modes

| Mode                         | What to do                                                       |
| ---------------------------- | ---------------------------------------------------------------- |
| **Mock for tests (default)** | `mockpyspark.activate()` in conftest.                            |
| **Pytest plugin**            | `pytest_plugins = ["mockpyspark.pytest_plugin"]` in conftest.    |
| **Real PySpark for a run**   | Pass `--no-mock-pyspark` to pytest, or don't call `activate()`.  |

The pytest plugin also accepts `--no-mock-pyspark` to bypass activation for a single invocation, useful for running the same suite against real PySpark for integration coverage.

See **[docs/usage.md](docs/usage.md)** for the full pattern, including switching between mock and real PySpark in the same suite, and known limitations.

---

## Detecting the mock at runtime

```python
import pyspark
if getattr(pyspark, "__mock__", False):
    print("running against mockpyspark")
```

---

## Contributing

**Run unit tests** (no Java, no Docker):

```bash
git clone https://github.com/PeterDowdy/mock-pyspark.git && cd mock-pyspark
pip install pyarrow duckdb pandas numpy pytest
PYTHONPATH=. pytest -q -m unit
```

**Run integration tests** against real PySpark (requires Java 17):

```bash
# Via Docker (easiest)
./start.sh 4.1 --integration-test

# Or locally with a PySpark venv
python -m venv /opt/spark-venv && /opt/spark-venv/bin/pip install pyspark==4.1.0
PYTHONPATH=. PYSPARK_VERSION=4.1 PYSPARK_PYTHON=/opt/spark-venv/bin/python pytest -q -m integration
```

See **[docs/contributing.md](docs/contributing.md)** for the full setup guide.

---

## API coverage

Supports PySpark 4.0.x and 4.1.x — 439+ functions, full DataFrame/Column/Window/Catalog API, CSV/JSON/Parquet IO.

- [STATUS-4.1.md](STATUS-4.1.md) — PySpark 4.1 coverage
- [STATUS-4.0.md](STATUS-4.0.md) — PySpark 4.0 coverage
