Metadata-Version: 2.4
Name: commentlens
Version: 0.1.1
Summary: NLP-based code comment classifier for Python codebases
Author-email: Pavan Prakash Lella <lellapavanprakashnaidu@gmail.com>
Keywords: nlp,code-quality,comment-analysis,static-analysis,text-classification
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Classifier: Topic :: Software Development :: Quality Assurance
Classifier: Intended Audience :: Developers
Requires-Python: >=3.9
Description-Content-Type: text/markdown
Requires-Dist: scikit-learn>=1.3
Requires-Dist: pandas>=2.0
Requires-Dist: numpy>=1.24
Requires-Dist: spacy>=3.7
Requires-Dist: nltk>=3.8
Requires-Dist: textstat>=0.7
Requires-Dist: scipy>=1.11
Requires-Dist: matplotlib>=3.7
Requires-Dist: jinja2>=3.1
Requires-Dist: click>=8.1

# CodeAudit

A classical NLP pipeline that scans Python codebases, extracts comment blocks,
and classifies each one by quality — with no LLMs, no API calls, and no GPU required.

---

## What It Does

You point it at any Python project. It reads every file, finds all comments,
groups them into logical blocks, classifies each block, and generates a ranked report.

Each comment block gets one of four labels:

| Label | Meaning |
|---|---|
| `informative` | Explains why or what — adds real value |
| `low_quality` | Vague, redundant, or restates the code |
| `directive` | TODO, FIXME, HACK, NOTE, WARNING |
| `outdated` | References removed APIs, old versions, legacy code |

---

## Why Classical NLP and Not an LLM

| | This tool | LLM-based approach |
|---|---|---|
| Inference per block | < 2ms | 2000–10000ms |
| 50K file audit | ~ 50 seconds | ~ 14 hours |
| Cost | $0 | $5–$50 per run |
| Works offline | ✅ | ❌ |
| Reproducible | ✅ 100% | ❌ varies |
| Install size | ~ 50MB | 2GB+ |

For a measurement problem across thousands of files, classical NLP
is the correct architecture — not a compromise.

---

## Installation

```bash
pip install codeaudit
python -m spacy download en_core_web_sm
```

---

## Usage

```bash
codeaudit scan ./my_project --output ./results
```

Open `results/codeaudit_report.html` in your browser for the full visual report.

---

## Output

Three files are generated in your output folder:

```
results/
├── codeaudit_report.html     ← visual dashboard, open in browser
├── block_report.csv          ← every comment block with label + confidence
└── file_report.csv           ← per-file quality summary
```

### block_report.csv

```
file_path        | block_id | start_line | end_line | text                        | label       | confidence
flask/app.py     | 1        | 12         | 14       | Retries up to 3 times...    | informative | 0.91
flask/app.py     | 2        | 23         | 23       | TODO: fix auth flow         | directive   | 0.96
flask/helpers.py | 1        | 8          | 9        | this does the thing         | low_quality | 0.88
```

### file_report.csv

```
file_path        | total_blocks | informative_pct | low_quality_pct | directive_pct | outdated_pct | dominant_label
flask/app.py     | 42           | 61.9            | 21.4            | 14.3          | 2.4          | informative
flask/helpers.py | 18           | 33.3            | 55.6            | 11.1          | 0.0          | low_quality
```

---

## Model Performance

Trained on 800 manually labeled comment blocks from 5 major open-source
Python repositories — Flask, Requests, Django, pytest, click.

Inter-rater reliability validated at Cohen's Kappa = 0.74 before training.
Evaluated on 160 held-out blocks the model never saw during training.

| Model | Accuracy | F1 (weighted) |
|---|---|---|
| **Logistic Regression** | **0.956** | **0.957** |
| Linear SVM | 0.950 | 0.949 |
| Naive Bayes | 0.825 | 0.813 |

Default model is Logistic Regression.

### A note on class distribution

The dataset reflects real-world comment distributions:

```
low_quality    69%
informative    19%
directive      10%
outdated        1%
```

Overall F1 is influenced by this imbalance. Per-class F1 from the
classification report is the more meaningful metric — particularly
for minority classes like outdated. All models were trained with
class_weight=balanced to compensate.

---

## NLP Pipeline

```
raw comment text
      ↓
camelCase / snake_case splitting
      ↓
lowercase + noise removal
      ↓
tokenization
      ↓
stopword removal  (NLTK)
      ↓
lemmatization     (NLTK WordNet)
      ↓
TF-IDF vectorization  (ngram 1–3, tuned vocabulary size)
      ↓
hand-engineered features  (readability, keyword flags, structural)
      ↓
Logistic Regression classifier
      ↓
label + confidence score
```

---

## Feature Engineering

Two types of features are combined into one matrix:

**TF-IDF features**
- Unigrams, bigrams, trigrams on cleaned comment text
- Sublinear TF normalization
- Vocabulary size tuned via grid search

**Hand-engineered features**
- Flesch-Kincaid readability score
- Gunning Fog index
- Word count
- Keyword flags — todo, fixme, hack, note, warning, deprecated, bug
- Block line count
- Code token ratio — how much code leaked into the comment
- Average token length

---

## Dataset

No existing labeled dataset existed for code comment quality classification.
It was built entirely from scratch:

- Cloned 5 major open-source Python repositories
- Extracted comment blocks using Python's tokenizer module
- Manually labeled 800 blocks across 4 categories
- Validated inter-rater reliability with a second annotator
- Cohen's Kappa = 0.74 confirmed before training began
- Only blocks where both annotators agreed were kept

Repositories used for training: Flask, Requests, Django, pytest, click

---

## Validation

- 80/20 stratified train/test split
- Evaluated on 160 held-out blocks never seen during training
- 5-fold cross-validation during hyperparameter tuning
- Calibration curves confirm confidence scores are meaningful
- Tested on held-out repositories not used in training or labeling

---

## Extending This Tool

**Add more languages**
Swap the Python tokenizer for `tree-sitter` to support Java, Go, Rust.

**CI/CD integration**
Run on every pull request. Flag files where low_quality percentage
rises above a configured threshold.

**Drift detection**
Run on the same repo at two points in time. Track whether comment
quality improves or degrades across releases.

---

## Requirements

```
Python     >= 3.9
scikit-learn >= 1.3
pandas       >= 2.0
spacy        >= 3.7
nltk         >= 3.8
textstat     >= 0.7
scipy        >= 1.11
matplotlib   >= 3.7
jinja2       >= 3.1
click        >= 8.1
```

---

## License

MIT License.
