Metadata-Version: 2.4
Name: dsqus
Version: 0.0.9
Summary: dsqusss Python package
Author: Sanjog Sigdel
Project-URL: Homepage, https://github.com/sigdelsanjog/dsqus
Project-URL: Repository, https://github.com/sigdelsanjog/dsqus
Requires-Python: >=3.10
Description-Content-Type: text/markdown
Requires-Dist: pandas>=2.2
Requires-Dist: openpyxl>=3.1

# 🧩 Dataset Quality Scoring Engine — System Framework (Markdown)

## Local Development Split

The repository now separates responsibilities:

- `dsqus/engine`: core Python package code (no FastAPI).
- `backend`: FastAPI service that imports and calls the core package.
- `ui`: browser frontend that uploads files to the backend.

### Run Upload Flow End-to-End

From repository root:

```bash
python -m venv .venv
source .venv/bin/activate
pip install -e .
pip install -r backend/requirements.txt
uvicorn backend.app:app --reload
```

From `ui` in another terminal:

```bash
npm install
npm run dev
```

Then open the Vite URL (typically `http://localhost:5173`) and upload a CSV/XLS/XLSX file.

#️⃣ 1. Overview

The Dataset Quality Scoring Engine (DQS) evaluates the quality of any dataset using automated, model-agnostic metrics.
The system processes user-uploaded datasets, computes embeddings, analyzes statistical and semantic properties, and outputs a standardized quality score (0–100) along with detailed submetrics.

## 2. High-Level Workflow

User Upload → Preprocessing → Embedding → Metric Computation → Scoring → Report Generation → Cleanup

## 3. Input Specifications

The system accepts:

jsonl
json
txt
csv
folder of text/code files
PDFs (extracted into text)

## 4. Preprocessing Pipeline

Validate file format
Convert to normalized internal format (list[str or dict])
Clean text:
remove control chars
normalize whitespace
optional: strip HTML/markup
Segment long documents into meaningful chunks
Remove empty or invalid samples

Output: clean, structured dataset

## 5. Embedding Generation

Two embedding flows:

5.1 Local Embeddings (Per Upload)

Used for:

redundancy
coherence
diversity
factual contradictions
clustering/domain analysis

These embeddings exist only for the request and are deleted afterward.

5.2 Global Reference Embeddings (Static)

Used only for novelty detection.

Pre-built FAISS/Vector DB containing ~1M representative samples:

Wikipedia
Common Crawl samples
C4 slices
StackOverflow
Books corpus
Public domain corpora

This is static, never modified by user uploads.

## 6. Metric Computation

DQS computes 10 core quality metrics:

6.1 Redundancy Score
compute embedding similarity within dataset
clustering density = redundancy
score = inverse redundancy
6.2 Malware / Toxicity Score
run samples through pre-trained toxicity classifier
aggregate severity
6.3 Diversity Score
linguistic diversity (entropy, vocab richness)
semantic diversity (embedding variance)
6.4 Readability Score
Flesch–Kincaid
sentence complexity
coherency heuristics
6.5 Semantic Coherence
embedding flow consistency
perplexity using a small reference LLM
6.6 Novelty Score
compare against global reference corpus
nearest neighbor distance = novelty measure
6.7 Structure Quality

Applicable to:

JSON
code
SQL
XML
YAML

Checks:

syntax validity
AST parsing success
6.8 Factual Conflict Score
sample random pairs
pass to NLI contradiction model
aggregate contradictions
6.9 Domain Balance Score
cluster dataset embeddings
measure cluster distribution via entropy
6.10 Length Distribution Score
detect outliers
analyze token distribution

## 7. Composite Score Calculation

All metrics normalized 0–100.

Weighted aggregation formula:

overall_score =
0.15*redundancy +
0.10*toxicity +
0.10*diversity +
0.10*readability +
0.10*coherence +
0.10*novelty +
0.10*structure +
0.10*factual_conflict +
0.075*domain_balance +
0.075*length_distribution

## 8. Report Generation

Output includes:

8.1 JSON Report

Contains:

overall_score
all sub-scores
dataset metadata
top detected issues
summary of duplicates
domain distribution histogram
8.2 Human-Readable Text Report
simple explanations
listed issues
recommendations
optional PDF

## 9. System Architecture

Components
API Layer
file upload
async processing
report delivery
Compute Engine
embeddings
scoring logic
batching
concurrency optimized
Reference Store
FAISS/Qdrant global novelty index
static
Models Folder
toxicity classifier
contradiction/NLI model
small LLM for perplexity

## 10. Execution Flow Diagram

[Upload]
↓
[Preprocess]
↓
[Generate Local Embeddings]
↓
[Compute All Self-Contained Metrics]
↓
[Compare with Global Reference Embeddings]
↓
[Aggregate Scores]
↓
[Generate JSON + Text Report]
↓
[Return to User]
↓
[Delete all temp embeddings + data]

## 11. Privacy Model

No dataset stored after processing
No embeddings stored
Only the report is saved (optional)
Global reference embeddings NEVER contain user data
Fully GDPR-safe

## 12. MVP Boundary (Important)

Not included in v1:

dataset cleaning
dataset repair
dataset marketplace
collaborative annotation
data augmentation
agentic workflows

You stay laser-focused on:
analysis → scoring → reporting.
