Metadata-Version: 2.4
Name: spark-data-profiler
Version: 0.1.0
Summary: PySpark DataFrame profiler — ydata-profiling-style HTML/JSON reports, alerts, and stats for Spark DataFrames
Author: dsnaveen
License: MIT License
        
        Copyright (c) 2026 dsnaveen
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
        
Project-URL: Homepage, https://github.com/dsnaveen/spark-data-profiling
Project-URL: Repository, https://github.com/dsnaveen/spark-data-profiling
Project-URL: Issues, https://github.com/dsnaveen/spark-data-profiling/issues
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pyspark>=3.0
Provides-Extra: extras
Requires-Dist: numpy; extra == "extras"
Dynamic: license-file

# Spark Data Profiler

A tool that looks at a big table of data living in [Apache Spark](https://spark.apache.org/) and hands you back an easy-to-read report: what's missing, what looks wrong, and what the data actually contains — without you having to write a single line of analysis code.

Think of it like `df.describe()`, but far more thorough, and delivered as a report you can actually read.

## Why you'd want this

- **Spot missing data instantly** — see exactly which columns have gaps, and how big those gaps are.
- **Catch problems before they cause bugs** — columns that never change (constant), columns that are almost all unique (like an ID column mistakenly used as a feature), lopsided categories, unusually skewed numbers.
- **Understand relationships between columns** — a correlation heatmap shows which columns move together.
- **No setup, no extra tools** — one Python file, works anywhere PySpark already runs (including Databricks notebooks).
- **Scales to big data** — you can profile a small random sample instead of the whole table, so it stays fast even on huge datasets.

## What the report looks like

Running the profiler produces a single self-contained HTML page (or an inline view if you're in a notebook) with:

- **Overview cards** — row count, column count, memory footprint, at a glance.
- **A colored alert badge on every column** that has something worth flagging (missing values, all-zero, all-unique, skewed, imbalanced, etc.).
- **Per-column detail panels** — statistics, most frequent values or extreme values, and a distribution chart, one tab per column.
- **A correlation heatmap** between numeric columns.
- **A missing-values summary table** across the whole dataset.
- **A sample of actual rows**, so you can eyeball the real data alongside the statistics.

## Requirements

- [PySpark](https://spark.apache.org/docs/latest/api/python/) — this is the only hard requirement.
- `numpy` and Spark's `pyspark.ml` module are used if they're available (for correlations and extra stats), but the profiler works fine without them — it detects what's installed automatically.

## Get started in 2 minutes

New to this? Follow [**QUICKSTART.md**](QUICKSTART.md) — a plain-language, step-by-step walkthrough from "I have a Spark DataFrame" to "I have a report," including how to read what it tells you.

If you're already comfortable with PySpark, here's the short version:

```python
from spark_data_profiler import SparkDataProfiler

profiler = SparkDataProfiler(df, sample_fraction=0.1, title="My Report")
profiler.profile()                  # prints a console summary
profiler.to_html("report.html")     # saves a full HTML report
profiler.display()                  # renders inline in Databricks / Jupyter
profiler.to_json("report.json")     # machine-readable export
report = profiler.to_dict()         # plain Python dict, for custom use
```

## Credits

Created and maintained by **[dsnaveen](https://github.com/dsnaveen)**.

The imbalance-score design (an entropy-based measure of how lopsided a categorical column is) was inspired by ideas from [ydata-profiling](https://github.com/ydataai/ydata-profiling), an excellent pandas-based profiling library. This project reimplements the same kind of profiling report natively on Spark DataFrames, so it scales to datasets that don't fit on a single machine.

## License

Released under the [MIT License](LICENSE) — free to use, modify, and distribute, for personal or commercial projects, with attribution.
