Metadata-Version: 2.4
Name: sparklens
Version: 0.2.0
Summary: Spark Physical Plan Analyzer and Visualizer
Requires-Python: >=3.12
Description-Content-Type: text/markdown
Requires-Dist: altair
Requires-Dist: matplotlib
Requires-Dist: networkx
Requires-Dist: pydot
Requires-Dist: pyvis
Requires-Dist: google-genai
Requires-Dist: python-dotenv

<div align="center">

# SparkLens

### Deterministic Spark Execution Plan Analyzer with AI-Assisted Explanations

**Understand Spark execution. Identify bottlenecks. Make better optimization decisions.**

<br>

[![Version](https://img.shields.io/badge/version-0.1.1-6366f1?style=flat-square)](#version)
[![Python](https://img.shields.io/badge/Python-3.12%2B-3776AB?style=flat-square&logo=python&logoColor=white)](#installation)
[![Google Gemini](https://img.shields.io/badge/AI-Google_Gemini-8E75B2?style=flat-square&logo=googlegemini&logoColor=white)](#ai-setup)

</div>

---

## Overview

SparkLens is a tool for analyzing **Apache Spark Physical Execution Plans**.

It transforms raw execution plans into structured execution DAGs, deterministic performance findings, visual execution graphs, and comprehensive reports with optional AI-generated explanations.

| Capability | Description |
|:---|:---|
| Plan Parsing | Converts physical execution plans into structured operator representations |
| DAG Construction | Represents execution flow as a directed acyclic graph |
| Performance Analysis | Identifies execution characteristics and potential bottlenecks |
| Visualization | Generates execution DAGs as PNG images |
| Reporting | Produces Markdown and HTML reports |
| AI Explanations | Generates natural-language insights using Google Gemini |

> **The core principle: AI should explain analysis, not replace analysis.**

Unlike approaches that send raw execution plans directly to an LLM, SparkLens performs deterministic, rule-based analysis before using AI to explain the findings.

---

## Why SparkLens?

Spark execution plans are powerful, but interpreting them isn't always straightforward.

Even moderately complex plans can contain:

- Broadcast joins
- Shuffle exchanges
- Multi-stage aggregations
- Sorting operations
- Adaptive Query Execution (AQE)
- Partitioning strategies
- Stage boundaries

Understanding these plans often requires significant Spark expertise, especially when investigating:

- Why a query is slow
- Where data is shuffled
- Which join strategy Spark selected
- Where parallelism is lost
- How execution stages are formed

SparkLens automates much of this interpretation, presenting execution characteristics and potential performance issues in a structured, explainable format.

---

## Core Features

### Execution Plan Parsing

Parses Spark Physical Execution Plans into structured operator representations for further analysis.

### Execution DAG Construction

Converts execution plans into a directed acyclic graph (DAG) representing execution flow.

### Deterministic Rule Engine

Identifies Spark performance patterns and execution characteristics, including:

- Broadcast Hash Joins
- Shuffle Operations
- Stage Boundaries
- Single Partition Bottlenecks
- Aggregation Pipelines
- Sorting Operations
- AQE Usage
- Partitioning Strategies
- Performance Anti-Patterns

### AI-Assisted Explanations

Uses Google Gemini to generate:

- Executive Summaries
- Execution Walkthroughs
- Performance Impact Analysis
- Optimization Recommendations

The AI layer receives structured findings generated by SparkLens and is instructed to explain those findings rather than independently analyze the plan.

### Visualization

Generates execution DAG visualizations as PNG images.

Visualizations highlight operators associated with analysis findings, making it easier to connect potential performance issues with the execution graph.

### Reporting

Generates reports in two formats:

- Markdown (`report.md`)
- HTML (`report.html`)

Reports contain deterministic analysis and, when AI-assisted analysis is enabled, AI-generated explanations.

---

## Analysis Philosophy

### Deterministic First. AI Second.

Sending an entire execution plan to an LLM and asking it to identify performance issues makes the analysis dependent on the model's interpretation.

SparkLens separates the two responsibilities.

**Traditional approach**

`Spark Plan → LLM → Explanation`

**SparkLens approach**

1. Parse the Spark Physical Execution Plan.
2. Construct the execution DAG.
3. Apply deterministic analysis rules.
4. Generate structured analysis findings.
5. Generate recommendations.
6. Use AI to explain the findings.

This approach provides:

- **Reduced hallucinations** — AI explanations are grounded in structured findings.
- **Consistent findings** — Analysis is driven by deterministic rules.
- **Explainability** — Findings originate from identifiable analysis rules.
- **Reproducibility** — Deterministic analysis does not depend on AI-generated interpretations.
- **Separation of concerns** — Analysis and explanation remain distinct stages.

---

## Architecture

SparkLens separates plan processing, deterministic analysis, AI-assisted explanations, and report generation into distinct components.

| Component | Responsibility |
|:---|:---|
| Plan Normalizer | Prepares the Spark Physical Plan for parsing |
| Plan Parser | Extracts structured operator representations |
| DAG Builder | Constructs the execution DAG |
| Rule Engine | Identifies execution characteristics and potential performance issues |
| Analysis Report | Organizes the results of deterministic analysis |
| Recommendation Engine | Generates recommendations based on analysis findings |
| Prompt Builder | Prepares structured information for AI-assisted explanations |
| Gemini AI | Generates natural-language explanations |
| Visualization | Generates the execution DAG as `dag.png` |
| Report Generator | Produces `report.md` and `report.html` |

The execution DAG is processed by the visualization component, while the analysis results are passed to the report generator. When AI is enabled, the explanation is incorporated into the generated reports.

---

## Installation

### Requirements

- Python 3.12 or newer

### Install from PyPI

```bash
pip install sparklens
```

---

## Input Format

SparkLens reads Spark physical plans generated using **formatted explain mode**, where every operator is numbered and described in its own block.

**PySpark**

```python
df.explain("formatted")
```

**Spark SQL / Databricks**

```sql
EXPLAIN FORMATTED SELECT ...
```

### Preparing the Input

1. Generate the physical execution plan using formatted explain mode.
2. Copy the printed plan into a text file, such as `plan.txt`.
3. Pass the file to SparkLens.

> **Note:** Plans printed by plain `df.explain()` or `df.explain(True)` are not supported yet.

---

## AI Setup

SparkLens uses Google Gemini for natural-language explanations.

Set the `GEMINI_API_KEY` environment variable or create a `.env` file in the directory where you run SparkLens.

```env
GEMINI_API_KEY=your_api_key_here
```

**Note:** The API key is required only when using the `--ai` flag. Deterministic analysis runs fully offline.

---

## Usage

### Deterministic Analysis

Analyze a Spark physical execution plan without AI-generated explanations.

```bash
sparklens analyze plan.txt
```

**Generated files**

```text
dag.png
report.md
report.html
```

### AI-Enhanced Analysis

Enable AI-assisted explanations and recommendations.

```bash
sparklens analyze plan.txt --ai
```

**Generated files**

```text
dag.png
report.md
report.html
```

The reports additionally include AI-generated:

- Executive Summary
- Execution Explanation
- Recommendations

---

## Example Findings

SparkLens generates structured findings that identify execution characteristics and their potential impact.

**Single Partition Bottleneck**

```text
Single Partition Bottleneck
Severity: High

The execution plan contains a SinglePartition exchange,
which may reduce parallelism by processing data in a
single partition.
```

---

## Example AI Summary

An example of an AI-generated explanation based on the analysis findings:

> The execution plan uses a Broadcast Hash Join and Adaptive Query Execution but contains a critical Single Partition bottleneck that limits parallelism and may impact scalability.

---

## Project Structure

```text
SparkLens/
│
├── ai/
├── analyzer/
├── cli/
├── models/
├── report/
├── rules/
├── sample_plans/
├── tests/
├── visualization/
│
├── pyproject.toml
├── README.md
└── .env
```

---

## Running Tests

From a source checkout:

```bash
pip install -e . pytest
pytest
```

---

## Current Capabilities

**Current release: `v0.1.1`**

- Spark Physical Plan Parsing
- Execution DAG Construction
- Rule-Based Analysis
- Recommendation Generation
- AI-Assisted Explanations
- DAG Visualization
- Markdown Reports
- HTML Reports
- CLI Interface
- Local Package Installation

---

## Roadmap

### v0.2

- Databricks Integration
- Rich HTML Dashboard
- Embedded DAG Visualizations
- Additional Rule Packs
- Enhanced Recommendation Engine

### v0.3

- Runtime Metrics Analysis
- Spark UI Integration
- Query Plan Comparison
- Execution Replay Visualizations
- Cost-Based Optimization Insights

---

## Version

| Property | Value |
|:---|:---|
| Current Release | `v0.1.1` |

---

## Authors

- [Arya Mulay](https://github.com/aryasm13)
- [Aryan Vinod Deshpande](https://github.com/dshryn)
- [Yashwant Sahoo](https://github.com/Yashwant1105)

---

<div align="center">

**Built to make Spark execution plans understandable, explainable, and actionable.**

</div>
