Metadata-Version: 2.4
Name: sparklens
Version: 0.1.0
Summary: Spark Physical Plan Analyzer and Visualizer
Requires-Python: >=3.12
Description-Content-Type: text/markdown
Requires-Dist: altair
Requires-Dist: matplotlib
Requires-Dist: networkx
Requires-Dist: pydot
Requires-Dist: pyvis
Requires-Dist: google-genai
Requires-Dist: python-dotenv

# SparkLens

## Deterministic Spark Execution Plan Analyzer with AI-Assisted Explanations

SparkLens is a tool for analyzing Apache Spark Physical Execution Plans.

It transforms raw Spark execution plans into:

* Structured execution DAGs
* Deterministic performance findings
* Visual execution graphs
* Markdown and HTML reports
* AI-generated explanations powered by Gemini

SparkLens is designed around a simple principle:

> AI should explain analysis, not replace analysis.

Unlike tools that send raw execution plans directly to an LLM, SparkLens first performs deterministic rule-based analysis and then uses AI to explain the findings.

---

# Why SparkLens?

Spark execution plans are powerful but difficult to interpret.

Even moderately complex plans may contain:

* Broadcast joins
* Shuffle exchanges
* Multi-stage aggregations
* Sorting operations
* Adaptive Query Execution (AQE)
* Partitioning strategies
* Stage boundaries

Understanding:

* Why a query is slow
* Where data is shuffled
* Which join strategy Spark selected
* Where parallelism is lost
* How execution stages are formed

often requires significant Spark expertise.

SparkLens automates this process and presents the results in a structured, explainable format.

---

# Core Features

## Execution Plan Parsing

Parse Spark Physical Execution Plans into structured operator representations.

---

## Execution DAG Construction

Convert execution plans into a directed acyclic graph representing execution flow.

---

## Deterministic Rule Engine

Detects Spark performance patterns and execution characteristics including:

* Broadcast Hash Joins
* Shuffle Operations
* Stage Boundaries
* Single Partition Bottlenecks
* Aggregation Pipelines
* Sorting Operations
* AQE Usage
* Partitioning Strategies
* Performance Anti-Patterns

---

## AI-Assisted Explanations

Generate:

* Executive Summaries
* Execution Walkthroughs
* Performance Impact Analysis
* Optimization Recommendations

using Google Gemini.

The AI layer receives structured findings generated by SparkLens and is instructed to explain those findings rather than independently analyze the plan.

---

## Visualization

Generate execution DAG visualizations as PNG images.

Visualization highlights operators associated with analysis findings.

---

## Reporting

Generate:

* Markdown Reports
* HTML Reports

containing both deterministic analysis and AI-generated explanations.

---

# Analysis Philosophy

SparkLens follows a deterministic-first architecture.

Instead of sending a raw execution plan directly to an LLM:

```text
Spark Plan
    ↓
LLM
    ↓
Explanation
```

SparkLens performs structured analysis first:

```text
Spark Plan
    ↓
Parser
    ↓
Execution DAG
    ↓
Rule Engine
    ↓
Analysis Report
    ↓
Recommendation Engine
    ↓
AI Explanation
```

Benefits:

* Reduced hallucinations
* Consistent findings
* Explainable analysis
* Reproducible results
* Better separation between analysis and explanation

---

# Evaluation

SparkLens findings are scored against a fixed ground-truth dataset. The evaluation runs fully offline (no API calls, no model downloads) and is deterministic: the same inputs always give the same numbers.

## What is measured

* `evaluation/dataset_expected.csv` — 50 real Spark physical plans (49 rendered with Spark 3.5 `explain("formatted")`, 1 Databricks Photon plan), each with the analysis a correct analyzer should produce, written as plain sentences (316 atomic claims)
* `evaluation/dataset_predicted.csv` — what SparkLens reports for each plan (`sparklens analyze <plan> --eval`: rule engine only, AI disabled)

Both paragraphs are split into atomic claims and compared claim by claim:

* **Precision** — share of SparkLens claims backed by the ground truth (false alarms lower it)
* **Recall** — share of ground-truth claims that SparkLens states (misses lower it)
* **F1** — balance of the two
* **Final score (F0.5)** — weights precision above recall, because false alarms erode developer trust

## How claims are matched

```text
SparkLens output / ground truth
    ↓
Atomic claims (sentences split at "so / because / but"; pronouns resolved)
    ↓
Spark concept normalisation ("SMJ" = "SortMergeJoin" = "sort-merge join")
    ↓
TF-IDF vectors (Spark-concept channel + wording channel)
    ↓
Directional matching (precision: backed by the ground truth; recall: stated by SparkLens)
    ↓
Consistency rules (stage counts, numbers, polarity, exclusive alternatives, advice vs observation)
```

## Validating the evaluator

`python evaluation/evaluate_engine.py validate` measures the evaluator itself on 1,377 test cases whose correct answer is known by construction:

| Test type | Example | Correct verdict |
|---|---|---|
| Paraphrase | the same claim in different words | match |
| Number change | "3 stages" → "4 stages" | no match |
| Polarity flip | enabled ↔ disabled, required ↔ avoidable | no match |
| Concept swap | sort-merge ↔ broadcast hash join, hash ↔ range partitioning | no match |
| Cross-plan claim | a claim about an operator the plan text does not contain | no match |
| Role swap | same words, roles swapped | no match |

It also checks invariance (reordering, duplicating or re-spacing findings never changes a score) and bounds (the ground truth as prediction scores 1.0, an empty answer 0.0, another plan's output far lower).

## Current baseline

Rule engine `c7daf3c`, 4 Oct 2026:

| Metric | Value |
|---|---|
| SparkLens precision / recall / F1 | 0.637 / 0.478 / 0.546 |
| SparkLens final score (F0.5) | **0.597** |
| Evaluator accuracy, 1,377 known-answer cases | **92.2%** (dev split 94.4%, test split 90.2%) |
| Evaluator false accepts / false rejects | 11.5% / 3.5% |
| Invariance checks / bounds | 200 of 200 / pass |

Known limitation: claims that use the same words with swapped roles ("orders is broadcast, customers is shuffled" vs the reverse) are not yet told apart.

## Running the evaluation

```bash
pip install -r evaluation/requirements-eval.txt

python evaluation/evaluate_engine.py selftest    # unit checks of the matcher and metrics
python evaluation/evaluate_engine.py validate    # evaluator accuracy on known-answer cases
python evaluation/evaluate_engine.py predict     # SparkLens (--eval) on all 50 plans -> dataset_predicted.csv
python evaluation/evaluate_engine.py evaluate    # precision / recall / F1 / final score, change vs baseline
python evaluation/evaluate_engine.py baseline    # freeze the current result as eval_baseline.json
```

`predict` runs the SparkLens CLI, so SparkLens itself must be installed (`pip install -e .`). After a rule change, run `predict --force` and then `evaluate`; improved and regressed plans are listed.

To fill `dataset_predicted.csv` by hand, run `python evaluation/evaluate_engine.py export` and paste the output of `sparklens analyze evaluation/eval_plans/<id>.txt --eval` into each row.

All evaluation files live in `evaluation/`; run the commands from the repository root.

| File | Purpose |
|---|---|
| `evaluate_engine.py` | the evaluator (commands above) |
| `dataset_expected.csv` | ground truth |
| `dataset_predicted.csv` | SparkLens output being scored |
| `eval_validation.csv` | paraphrases and role swaps used by `validate` |
| `eval_baseline.json` | frozen reference result, including evaluator accuracy |
| `eval_review.csv` | per-claim verdicts of the last run (generated, not tracked) |

---

# Architecture

```text
Spark Physical Plan
          │
          ▼
   Plan Normalizer
          │
          ▼
      Plan Parser
          │
          ▼
      DAG Builder
          │
          ▼
    Execution DAG
          │
          ▼
      Rule Engine
          │
          ▼
   Analysis Report
          │
          ▼
 Recommendation Engine
          │
          ▼
     Prompt Builder
          │
          ▼
      Gemini AI
          │
          ▼
    AI Explanation


Execution DAG
      │
      ▼
 Visualization
      │
      ▼
    dag.png


Analysis Report
      │
      ▼
 Report Generator
      │
      ▼
report.md
report.html
```

---

# Installation

Clone the repository:

```bash
git clone <repository-url>
cd SparkLens
```

Install SparkLens locally:

```bash
pip install -e .
```

---

# AI Setup

Create a `.env` file in the project root:

```env
GEMINI_API_KEY=your_api_key_here
```

SparkLens uses Google Gemini for natural-language explanations.

---

# Usage

## Deterministic Analysis

```bash
sparklens analyze sample_plans/broadcast_join.txt
```

Outputs:

```text
dag.png
report.md
report.html
```

---

## AI-Enhanced Analysis

```bash
sparklens analyze sample_plans/broadcast_join.txt --ai
```

Outputs:

```text
dag.png
report.md
report.html
```

with AI-generated:

* Executive Summary
* Execution Explanation
* Recommendations

## Evaluation Output

```bash
sparklens analyze evaluation/eval_plans/<id>.txt --eval
```

This prints the deterministic rule-engine findings as a single line without making AI API calls.

---

# Example Findings

```text
Single Partition Bottleneck
Severity: High

The execution plan contains a SinglePartition exchange,
which may reduce parallelism by processing data in a
single partition.
```

---

# Example AI Summary

```text
The execution plan uses a Broadcast Hash Join and Adaptive
Query Execution but contains a critical Single Partition
bottleneck that limits parallelism and may impact scalability.
```

---

# Project Structure

```text
SparkLens/
│
├── ai/
├── analyzer/
├── cli/
├── evaluation/
├── models/
├── report/
├── rules/
├── sample_plans/
├── tests/
├── visualization/
│
├── pyproject.toml
├── README.md
└── .env
```

---

# Running Tests

Execute the full test suite:

```bash
pytest
```

---

# Current Capabilities (v0.1.0)

* Spark Physical Plan Parsing
* Execution DAG Construction
* Rule-Based Analysis
* Recommendation Generation
* AI-Assisted Explanations
* DAG Visualization
* Markdown Reports
* HTML Reports
* CLI Interface
* Local Package Installation

---

# Roadmap

## v0.2

* Databricks Integration
* Rich HTML Dashboard
* Embedded DAG Visualizations
* Additional Rule Packs
* Enhanced Recommendation Engine

## v0.3

* Runtime Metrics Analysis
* Spark UI Integration
* Query Plan Comparison
* Execution Replay Visualizations
* Cost-Based Optimization Insights

---

# Version

Current Release:

```text
v0.1.0
```

---

Built to make Spark execution plans understandable, explainable, and actionable.
