Metadata-Version: 2.4
Name: noweda
Version: 0.1.3
Summary: Automated EDA with insights, scoring, and security-aware detection — built as a native pandas extension.
Author-email: Daniel Peng <danielpeng95@gmail.com>
License: MIT
Project-URL: Homepage, https://github.com/codewithdaniel1/NowEDA
Project-URL: Documentation, https://codewithdaniel1.github.io/NowEDA/
Project-URL: Repository, https://github.com/codewithdaniel1/NowEDA
Project-URL: Bug Tracker, https://github.com/codewithdaniel1/NowEDA/issues
Keywords: eda,data-analysis,pandas,data-quality,pii-detection,automated-eda,profiling
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pandas>=1.3
Requires-Dist: numpy>=1.21
Requires-Dist: pyspark>=3.4
Requires-Dist: openpyxl>=3.1
Requires-Dist: lxml>=4.6
Provides-Extra: excel
Requires-Dist: xlrd>=2.0.1; extra == "excel"
Requires-Dist: pyxlsb>=1.0.10; extra == "excel"
Requires-Dist: odfpy>=1.4.1; extra == "excel"
Provides-Extra: viz
Requires-Dist: matplotlib>=3.5; extra == "viz"
Requires-Dist: scipy>=1.7; extra == "viz"
Provides-Extra: ml
Requires-Dist: statsmodels>=0.13; extra == "ml"
Requires-Dist: scikit-learn>=1.0; extra == "ml"
Provides-Extra: test
Requires-Dist: pytest>=7; extra == "test"
Requires-Dist: pyarrow>=8.0; extra == "test"
Requires-Dist: odfpy>=1.4.1; extra == "test"
Requires-Dist: matplotlib>=3.5; extra == "test"
Requires-Dist: scipy>=1.7; extra == "test"
Provides-Extra: parquet
Requires-Dist: pyarrow>=8.0; extra == "parquet"
Provides-Extra: hdf
Requires-Dist: tables>=3.7; extra == "hdf"
Provides-Extra: spss
Requires-Dist: pyreadstat>=1.1; extra == "spss"
Provides-Extra: full
Requires-Dist: xlrd>=2.0.1; extra == "full"
Requires-Dist: pyxlsb>=1.0.10; extra == "full"
Requires-Dist: odfpy>=1.4.1; extra == "full"
Requires-Dist: matplotlib>=3.5; extra == "full"
Requires-Dist: scipy>=1.7; extra == "full"
Requires-Dist: statsmodels>=0.13; extra == "full"
Requires-Dist: scikit-learn>=1.0; extra == "full"
Requires-Dist: pyarrow>=8.0; extra == "full"
Requires-Dist: tables>=3.7; extra == "full"
Requires-Dist: pyreadstat>=1.1; extra == "full"
Dynamic: license-file

<div align="left">
  <img src="https://raw.githubusercontent.com/codewithdaniel1/NowEDA/main/assets/noweda-wordmark-logo.svg" alt="NowEDA" width="700" />
</div>

[![PyPI version](https://img.shields.io/pypi/v/noweda)](https://pypi.org/project/noweda/)

# NowEDA

Exploratory data analysis through a native pandas accessor. Profile columns,
find missing data and duplicates, flag potential PII, explore relationships,
and generate reports with `df.eda` or its equivalent alias `df.noweda`.

[Documentation](https://codewithdaniel1.github.io/NowEDA/) ·
[API reference](https://codewithdaniel1.github.io/NowEDA/api-reference/) ·
[Changelog](https://codewithdaniel1.github.io/NowEDA/changelog/) ·
[Report an issue](https://github.com/codewithdaniel1/NowEDA/issues)

## Install

Requires Python 3.8 or later. pip selects dependency versions compatible with
your Python version.

```bash
pip install noweda
pip install "noweda[viz]"  # Add charts and density overlays
```

## Quick start

This example runs without downloading a dataset:

```python
import pandas as pd
import noweda as eda

# Or load your own file: df = eda.read("data.csv")
df = pd.DataFrame({
    "age": [25, 31, None, 42],
    "income": [42000, 55000, 47000, 68000],
    "segment": ["A", "B", "A", "B"],
})

print(df.eda.scores_df())
print(df.eda.missing_df())
df.eda.statsall()
```

## Analysis methods

### 1. `df.eda.statsall()` — Statistical profile

Prints scores, column roles, numeric and categorical statistics, missingness,
outliers, and preprocessing suggestions. Optional `noweda[ml]` dependencies
provide VIF and time-series diagnostics; unavailable diagnostics are omitted.

### 2. `df.eda.mlall()` — ML recommendations and preprocessing guidance

Heuristic recommendations help choose starting points for experimentation.
Ratings are not measured model performance or guarantees of model readiness.

**Supervised Learning (Classification & Regression)**

- Baseline and alternative algorithms with reasons and preprocessing guidance.
- For classification, use `df.eda.mlall(target="segment")` to check class balance.
  The target is excluded from feature recommendations. Class imbalance is flagged
  when the most frequent observed class has more than twice the count of the least
  frequent observed class. Without a target, class balance is not assessed.
- Numeric targets are treated as class labels when supplied; omit `target` for
  regression guidance without a class-balance check.

**Unsupervised Learning (Clustering & Dimensionality Reduction)**

- Suggestions for clustering, dimensionality reduction, and anomaly detection.

**Multicollinearity Guidance**

- High-correlation warnings and suggestions for feature selection or regularization.

**Data Preprocessing Pipeline**

- Suggested steps for missing values, encoding, scaling, and outliers.
- Example snippets are guidance; adapt them to your train/test split and task.

### 3. `df.eda.vizall()` — Automatic charts

Install `noweda[viz]` to draw distributions, correlations, categorical associations,
missingness, and applicable temporal plots. Unsupported or undefined categorical
associations appear as `N/A`, not zero association.

```python
df.eda.vizall()              # Samples 10,000 rows when the input exceeds 50,000
# df.eda.vizall(sample=5000)  # Choose a sample size
# df.eda.vizall(sample=False) # Plot the complete dataset
```

Statistical reports use the complete DataFrame; visualization sampling affects
only charts.

### 4. `df.eda.profile_column("age")` — One-column profile

Inspect a column's distribution, missingness, outliers, and suggested transforms.

### 5. `df.eda.compare(other_df)` — Compare reports

Compare dimensions, inferred column roles, scores, and detected risks between
two DataFrames. This is not a formal statistical test for distribution drift.

## Tables and programmatic reports

| Method | Output |
|---|---|
| `scores_df()` | `Value` column indexed by score name |
| `insights_df()` | `Insight` column |
| `schema_df()` | `Column`, `dtype`, `role`, `confidence`, `unique`, `uniqueness_ratio` |
| `stats_df()` | Per-column statistics; numeric fields include `mean`, `std`, `q25`, `median`, `q75`, `skewness`, `kurtosis` |
| `missing_df()` | Missing percentages; use `format="number"` for counts |
| `duplicates_df()` | Duplicate rows and constant columns |
| `correlation_df()` | Numeric Pearson correlation matrix |
| `outliers_df()` | IQR outlier counts; use `format="percentage"` for percentages |
| `pii_df()` | `Column`, `PII_Type`, `Count` |
| `encoding_df()` | `Column`, `Encoding_Type` (possible Base64 signals) |
| `summary()` | Dictionary of raw plugin outputs |
| `report()` | Dictionary containing `results`, `scores`, and `insights` |

Reports are cached. Each request fingerprints the DataFrame's values and schema,
recomputing analysis when they change. This check scans the data; cached access is
not constant-time. Use `df.eda.refresh()` to explicitly force a new report.

PII detection uses patterns and can miss sensitive values or flag false positives.
A risk score of zero means no configured pattern was detected, not that data is
safe to share. Quality and readiness scores are heuristics.

## File formats and optional dependencies

```python
df = eda.read("data.csv", dtype={"customer_id": str})
# eda.read("data.jsonl") defaults to lines=True
```

| Formats / feature | Installation |
|---|---|
| CSV, TSV, TXT, JSON/JSONL, XLSX/XLSM, XML, HTML, Stata, SAS, Pickle | `pip install noweda` |
| XLS, XLSB, ODS/ODF/ODT | `pip install "noweda[excel]"` |
| Parquet, Feather, ORC | `pip install "noweda[parquet]"` |
| HDF5 | `pip install "noweda[hdf]"` |
| SPSS | `pip install "noweda[spss]"` |
| Charts and KDE overlays | `pip install "noweda[viz]"` |
| VIF, stationarity and seasonality dependencies | `pip install "noweda[ml]"` |
| All optional analysis and format dependencies | `pip install "noweda[full]"` |

Only read Pickle files from trusted sources, since loading them can execute code.

## Large files

CSV, JSON, and all chunked reads use pandas consistently regardless of file size.
PySpark remains a standard dependency in 0.1.3. Large Parquet/ORC files (at least
128 MB) without reader options may use Spark, with a pandas fallback on failure.
Spark requires a compatible Java installation. Reads with options use pandas.
Spark loading still collects the final pandas DataFrame into local memory and
is not guaranteed to be faster.

For data that does not fit in memory, iterate over CSV or line-delimited JSON:

```python
for chunk in eda.read_chunked("large.csv", chunksize=10000, concat=False):
    print(chunk.eda.missing_df())
```

The default `concat=True` retains all chunks and allocates a combined DataFrame;
it is appropriate only when the data and concatenation overhead fit in RAM.
Per-chunk scores describe each chunk, not the whole dataset.

## Export and CLI

```python
from noweda.report.html import generate_html_report

generate_html_report(df.eda.report(), "report.html")
```

```bash
noweda data.csv --html report.html --json report.json
```

## Development

```bash
git clone https://github.com/codewithdaniel1/NowEDA.git
cd NowEDA
pip install -e ".[test]"
python -m pytest tests/
```

NowEDA is alpha software, licensed under MIT. See the
[documentation](https://codewithdaniel1.github.io/NowEDA/) for plugins and examples.
