Metadata-Version: 2.4
Name: data_advisor
Version: 0.1.0
Summary: A production-grade diagnostic and code generation advisor for timed data analysis exams and rapid exploratory data science.
Author: Principal Data Science Architect
License-Expression: MIT
Keywords: data-analysis,eda,data-science,hypothesis-testing,code-generation,machine-learning,automated-eda,pandas,scipy,scikit-learn
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Education
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Classifier: Topic :: Software Development :: Code Generators
Requires-Python: >=3.9
Description-Content-Type: text/markdown
Requires-Dist: pandas>=2.0.0
Requires-Dist: numpy>=1.24.0
Requires-Dist: scipy>=1.10.0
Requires-Dist: scikit-learn>=1.2.0
Requires-Dist: matplotlib>=3.7.0
Requires-Dist: seaborn>=0.12.0
Provides-Extra: dev
Requires-Dist: pytest>=7.0.0; extra == "dev"
Requires-Dist: pytest-cov>=4.0.0; extra == "dev"

# DataAdvisor (data_advisor)

> **Autonomous Data Science Diagnostic Engine & Exam Code Generator**  
> Engineered for students, practitioners, and data scientists to audit unknown datasets, formulate statistical hypotheses, and generate copy-pasteable, production-ready Python pipelines during timed data analysis examinations.

---

## 🚀 Key Highlights

1. **Deterministic Auditing Engine**: Deeply profiles any `pd.DataFrame` (or CSV, TSV, Excel, Parquet file) in seconds.
2. **Automated Hypothesis Testing**:
   - Numeric vs. Numeric: **Pearson** & **Spearman** correlations with $p$-values.
   - Categorical vs. Categorical: **Chi-Square Test of Independence** with Cramér's V effect size.
   - Numeric vs. Binary Target: **Welch's Two-Sample t-test** & **Mann-Whitney U test**.
   - Numeric vs. Multi-class Target: **One-Way ANOVA** ($F$-test) & **Kruskal-Wallis $H$-test**.
3. **Outlier & Skewness Diagnostics**: Combines **IQR fences (1.5× and 3.0×)** and **Z-score criteria ($|z| > 3$)**; automatically recommends `np.log1p`, Yeo-Johnson power transformations, or winsorization.
4. **Target & Model Strategy Selector**: Automatically identifies whether the task is **Regression** or **Classification** (Binary or Multiclass), detects critical class imbalance, and selects baseline models (`LogisticRegression`, `Ridge`, `RandomForest`).
5. **Copy-Pasteable Code Generator**: Emits clean, PEP 8 compliant, runnable Python scripts containing complete Pandas, Scipy, and Scikit-Learn code tailored to the audited dataset.

---

## 📦 Installation

To install `data_advisor` in editable mode directly from your local repository:

```bash
# Navigate to the project root directory
cd "path/to/Custom Library"

# Install in editable mode with dependencies
pip install -e .
```

### Verified Dependencies
- Python `>= 3.9`
- `pandas >= 2.0.0`
- `numpy >= 1.24.0`
- `scipy >= 1.10.0`
- `scikit-learn >= 1.2.0`
- `matplotlib >= 3.7.0`
- `seaborn >= 0.12.0`

---

## ⚡ Quickstart: The 1-Line Exam Workflow

When handed an unfamiliar exam dataset (`exam_dataset.csv`), run the top-level `data_advisor` function in your Jupyter Notebook or Python terminal:

```python
from data_advisor import data_advisor

# 1-Line Diagnostic & Code Generation:
# Outputs:
# 1. Full diagnostic report stating what all analysis to perform
# 2. Executable, copy-pasteable Python code to execute all steps
data_advisor("exam_dataset.csv")
```

Or with an optional target column or output script:
```python
data_advisor("exam_dataset.csv", target="churn", output_script="solution.py")
```

You can also use the object-oriented API:
```python
from data_advisor import DataAdvisor

advisor = DataAdvisor("exam_dataset.csv", target="churn", id_cols=["customer_id"])
advisor.report()                          # Diagnostic report
advisor.generate_script("solution.py")    # Full Python script
clean_code = advisor.get_code_block("cleaning")  # Specific stage code
```

Or directly from your command-line terminal:
```bash
# Diagnostic report + code prompt
data-advisor exam_dataset.csv

# Print complete executable Python code directly to the terminal
data-advisor exam_dataset.csv --code

# Save complete pipeline to a Python file
data-advisor exam_dataset.csv --output solution.py
```

---

## 🔍 Instant Code Snippet Search: `search(key)`

`data_advisor` includes a search engine that searches **any keyword** across your own local code folder (`code_bank/`).

### How It Works:
1. **Your Folder**: Simply paste your custom Python scripts (`.py`), Jupyter Notebooks (`.ipynb`), Arduino/ESP32 sketches (`.ino`, `.cpp`), or text files into `code_bank/`.
2. **Dynamic Scanning**: The search engine scans `code_bank/` in real time on every search call—no indexing or restarts required. You can add, edit, or delete files anytime.
3. **Any Keyword**: Search for single terms, phrases, or keyword lists.

```python
from data_advisor import search

# 1. List all available snippets in your code_bank
search()

# 2. Search for any keyword (e.g., machine learning, IoT, algorithms)
search("linear regression")
search("dht22")
search("kmeans")
search("wokwi")

# 3. Search with multiple keywords or a list
search(["esp32", "temperature"])
search("decision tree, entropy")

# 4. Search within a custom directory
search("apriori", folder="my_other_codes")
```

Or search directly via the CLI:
```bash
# List all files currently in code_bank
data-advisor --search "list"

# Search for any keyword
data-advisor --search "linear regression"
data-advisor --search "wokwi"
```

---

## 📄 Complete File Viewing, Retrieval & Execution: `get_file()`, `read_file()`, `run_file()`

To retrieve or execute the **entire complete code file** from `code_bank/` (not just search snippets):

### 1. View / Print Complete File
```python
from data_advisor import get_file, view_file

# By question number:
get_file("q1")                     # Complete code for Question 1
get_file(24)                       # Complete code for Question 24
get_file("q32", line_numbers=True) # With line numbers

# By filename or keyword:
get_file("sketch.ino")             # Arduino / ESP32 sketch
get_file("subscriber.py")
```

### 2. Retrieve Complete Raw Code String (No Printing)
```python
from data_advisor import read_file

# Return complete raw code as a Python string
code_string = read_file("q24")
print(f"Loaded {len(code_string)} characters")
```

### 3. Run Any File Directly
```python
from data_advisor import run_file

# Execute a Python file directly via subprocess
run_file("q24")
run_file("iss")
```

### 4. Direct CLI Commands
```bash
# Print complete file to console
data-advisor --get q1
data-advisor -g q24 --line-numbers
data-advisor --get sketch.ino

# Execute script directly
data-advisor --run q24
data-advisor -r 14

# List all available code files
data-advisor --list
```



---

## 🧠 Decision Rules Matrix

### 1. Data Cleaning & Integrity Rules
| Condition | Severity | Recommended Action | Generated Code |
| :--- | :--- | :--- | :--- |
| **Duplicates Detected** (`> 0`) | `WARNING` / `CRITICAL` | Drop duplicate rows | `df = df.drop_duplicates().reset_index(drop=True)` |
| **High Missingness** (`>= 60%`) | `CRITICAL` | Drop column entirely | `df = df.drop(columns=[col])` |
| **Low Missingness** (`< 5%`) | `INFO` | Impute median (num) / mode (cat) | `df[col] = df[col].fillna(df[col].median())` |
| **Moderate Missingness** (`5% - 60%`) | `WARNING` | Skewed: Median; Symmetric: Mean | `SimpleImputer(strategy='median')` |
| **Constant Column** (variance = 0) | `CRITICAL` | Drop column | `df = df.drop(columns=[col])` |
| **Identifier Column** (IDs, hashes) | `WARNING` | Exclude from feature matrix $X$ | `X = df.drop(columns=[id_col])` |
| **Date String Column** | `RECOMMENDED` | Parse date & extract temporal features | `.dt.year`, `.dt.month`, `.dt.dayofweek` |

### 2. Distribution & Outlier Rules
| Metric | Condition | Transformation Strategy | Rationale |
| :--- | :--- | :--- | :--- |
| **Skewness** | $> 1.2$ (Non-negative) | `np.log1p(df[col])` | Stabilizes variance, compresses long positive tail |
| **Skewness** | $> 1.2$ (Contains negatives) | `PowerTransformer(method='yeo-johnson')` | Normalizes non-Gaussian data with non-positive values |
| **Skewness** | $< -1.2$ (Left-skewed) | `PowerTransformer(method='yeo-johnson')` | Corrects severe left tail |
| **IQR Outliers** | $> 5\%$ of samples | IQR clipping / `RobustScaler()` | Caps extreme leverage points |

### 3. Categorical Processing Rules
| Cardinality | Encoding Strategy | Implementation |
| :--- | :--- | :--- |
| **Binary** ($2$ levels) | Direct Binary Mapping ($0/1$) | `df[col].map({'No': 0, 'Yes': 1})` |
| **Low-to-Medium** ($3$ to $10$ levels) | One-Hot Encoding (`drop_first=True`) | `pd.get_dummies(..., drop_first=True, dtype=int)` |
| **High** ($> 10$ levels) | Frequency Encoding / Target Encoding | `df[col].map(df[col].value_counts(normalize=True))` |

### 4. Hypothesis Testing Decision Engine
| Variable Types | Primary Test | Non-Parametric Alternative | Effect Size Metric |
| :--- | :--- | :--- | :--- |
| **2 Numeric Features** | Pearson Correlation ($r, p$) | Spearman Rank Correlation ($\rho, p$) | Correlation Coefficient ($r$) |
| **2 Categorical Features** | Chi-Square Test of Independence ($\chi^2$) | Fisher's Exact Test (for $2\times 2$) | Cramér's $V$ |
| **1 Numeric + 1 Binary Cat** | Welch's Independent $t$-test | Mann-Whitney $U$ test | Cohen's $d$ / Rank-biserial |
| **1 Numeric + 1 Multi-class Cat** | One-Way ANOVA ($F, p$) | Kruskal-Wallis $H$-test | Eta-squared ($\eta^2$) |

### 5. Target Variable & Model Baseline Matrix
| Target Type | Detected Problem | Imbalance Threshold | Recommended Baselines | Evaluation Metrics |
| :--- | :--- | :--- | :--- | :--- |
| **Continuous Numeric** | Regression | N/A | Linear Regression, Ridge ($L_2$), Random Forest Regressor | RMSE, MAE, $R^2$ Score |
| **Binary / Discrete** | Classification | Minority $< 20\%$ (Ratio $\ge 3:1$) | Logistic Regression (`class_weight='balanced'`), Random Forest Classifier | F1-Score, PR-AUC, ROC-AUC, Confusion Matrix |

---

## 📂 Project Architecture

```
Custom Library/
├── pyproject.toml                     # Modern setuptools packaging & dependency metadata
├── README.md                          # Comprehensive library documentation & exam cheat sheet
├── src/
│   └── data_advisor/
│       ├── __init__.py                # Package exports (DataAdvisor, DataScanner, RuleEngine)
│       ├── advisor.py                 # Primary facade class & CLI reporter
│       ├── scanner.py                 # Core profiling engine & statistical test runner
│       ├── rules.py                   # Heuristic recommendation engine & severity prioritizer
│       ├── codegen.py                 # Sequential Python script & code block generator
│       ├── visualizer.py              # Publication-ready Matplotlib & Seaborn code generator
│       └── cli.py                     # Command-line entry point (`data-advisor`)
├── tests/
│   └── test_advisor.py                # Automated pytest suite covering regression & classification
└── examples/
    ├── demo_exam_workflow.py          # End-to-end exam simulation walkthrough
    ├── student_exam_data.csv          # Sample dataset with outliers, nulls, and duplicates
    └── generated_analysis_pipeline.py # Fully generated, executable production script
```

---

## 🧪 Running Tests

Run the test suite using `pytest`:

```bash
python -m pytest -v
```

All tests inspect and validate:
- Missing value audit & column dropping (>60%).
- Identifier and constant column detection.
- Skewness and IQR/Z-score outlier calculations.
- Parametric and non-parametric hypothesis testing.
- Target classification and regression auto-detection.
- Full end-to-end Python script code generation and execution.

---

## 📜 License
MIT License. Built for rigorous data science education, competitive coding, and exam excellence.
