Metadata-Version: 2.4
Name: viseda
Version: 1.0.1
Summary: Visual Exploratory Data Analysis for image, video, hyperspectral, point-cloud, and text/NLP datasets
Author: Isaac Osei Agyemang, Daniel Acheampong, Adu Asare Baffour, Isaac Adjei-Mensah
License-Expression: MIT
Keywords: EDA,exploratory data analysis,image,video,hyperspectral,point cloud,LiDAR,text,NLP,computer vision,remote sensing
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Science/Research
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Image Processing
Classifier: Topic :: Scientific/Engineering :: Visualization
Classifier: Topic :: Scientific/Engineering :: GIS
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy<2,>=1.23
Requires-Dist: matplotlib<4,>=3.6
Requires-Dist: scikit-learn<2,>=1.2
Requires-Dist: scikit-image<1,>=0.20
Requires-Dist: opencv-python<5,>=4.7
Provides-Extra: hyperspectral
Requires-Dist: scipy>=1.10; extra == "hyperspectral"
Requires-Dist: spectral>=0.23; extra == "hyperspectral"
Requires-Dist: rasterio>=1.3; extra == "hyperspectral"
Provides-Extra: pointcloud
Requires-Dist: laspy[lazrs]>=2.3; extra == "pointcloud"
Provides-Extra: all
Requires-Dist: scipy>=1.10; extra == "all"
Requires-Dist: spectral>=0.23; extra == "all"
Requires-Dist: rasterio>=1.3; extra == "all"
Requires-Dist: laspy[lazrs]>=2.3; extra == "all"
Provides-Extra: dev
Requires-Dist: pytest>=7.4; extra == "dev"
Requires-Dist: pytest-cov>=4.1; extra == "dev"
Requires-Dist: build>=1.0; extra == "dev"
Requires-Dist: twine>=5.0; extra == "dev"
Dynamic: license-file

<div align="center">

<img src="VisEDA.png" alt="VisEDA — Comprehensive Documentation" width="100%">

# VisEDA

### Visual Exploratory Data Analysis for multimodal datasets

**ImageEDA · VideoEDA · HyperspectralEDA · PointCloudEDA · TextEDA**

[![PyPI](https://img.shields.io/pypi/v/viseda.svg)](https://pypi.org/project/viseda/)
[![Python](https://img.shields.io/pypi/pyversions/viseda.svg)](https://pypi.org/project/viseda/)
[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)

**One toolkit for inspecting, understanding, and validating visual and text datasets before modelling.**

</div>

---

## Overview

**VisEDA** is a Python toolkit for exploratory data analysis across five major data modalities:

| Module | Designed for | Core analyses |
|---|---|---|
| **ImageEDA** | Image datasets | size, aspect ratio, brightness, contrast, sharpness, noise, exposure, colour, texture, frequency, duplicates, class balance |
| **VideoEDA** | Video datasets | frame count, duration, FPS, temporal brightness/contrast, sharpness, blur, motion, scene changes, similarity |
| **HyperspectralEDA** | Hyperspectral cubes | band statistics, SNR, noise, dropout bands, vegetation/water indices, PCA, false colour, spectral diversity |
| **PointCloudEDA** | 3D point clouds | geometry, density, height, duplicates, outliers, nearest neighbours, PCA shape descriptors, cloud distances |
| **TextEDA** | Text/NLP datasets | document length, vocabulary, lexical diversity, readability, symbols, scripts, duplicates, TF-IDF distances, label analysis |

VisEDA is intended for **data scientists, researchers, students, and practitioners** who want a consistent EDA workflow before training machine-learning or deep-learning models.

---

## Why VisEDA?

Exploratory analysis is standard practice for tabular data, but visual and multimodal datasets often require separate tools and custom scripts.

VisEDA provides a unified interface for:

- **dataset inventory and integrity checks**
- **quality and distribution analysis**
- **class-balance inspection**
- **duplicate and similarity analysis**
- **per-sample and dataset-level visualisations**
- **self-contained HTML reports**
- **command-line workflows**
- **Python API workflows**
- **reproducible pre-model data auditing**

The goal is simple:

> **Understand the dataset before trusting the model.**

---

## Installation

Install the base package from PyPI:

```bash
pip install viseda
```

### Optional extras

For extended hyperspectral file support:

```bash
pip install "viseda[hyperspectral]"
```

For LAS/LAZ point-cloud support:

```bash
pip install "viseda[pointcloud]"
```

Install all optional dependencies:

```bash
pip install "viseda[all]"
```

Python **3.9+** is recommended.

---

# Quick Start


> **Important:** ImageEDA uses `plot()` for its comprehensive dataset dashboard.  
> VideoEDA, HyperspectralEDA, PointCloudEDA, and TextEDA expose `plot_dataset()` for dataset-level dashboards.

## ImageEDA

```python
from viseda import ImageEDA

eda = ImageEDA(verbose=True)

eda.load(
    "path/to/images",
    label_from_parent=True
)

summary = eda.summary()

print(summary["inventory"])
print(summary["quality"])

# ImageEDA's full dataset dashboard is plot(), not plot_dataset()
eda.plot(
    save_path="image_dashboard.png"
)

eda.report(
    "viseda_image_report.html"
)
```

For in-memory images:

```python
import numpy as np
from viseda import ImageEDA

images = [
    np.random.default_rng(0).integers(
        0, 256, size=(128, 128, 3), dtype=np.uint8
    ),
    np.random.default_rng(1).integers(
        0, 256, size=(128, 128, 3), dtype=np.uint8
    ),
]

eda = ImageEDA(
    verbose=False,
    compute_glcm=False,
    compute_freq=False,
)

eda.load_arrays(
    images,
    labels=["class_a", "class_b"]
)

print(eda.summary()["inventory"])

eda.plot(
    save_path="image_dashboard.png"
)
```

---

## VideoEDA

```python
from viseda import VideoEDA

eda = VideoEDA(
    verbose=True,
    frame_sample_rate=5
)

eda.load(
    "path/to/videos",
    label_from_parent=True
)

summary = eda.summary()

print(summary["inventory"])
print(summary["temporal"])
print(summary["motion"])

eda.plot_dataset(
    save_path="video_dashboard.png"
)

eda.report(
    "viseda_video_report.html"
)
```

For one video rather than the whole dataset:

```python
eda.plot(
    video_index=0,
    save_path="single_video.png"
)
```

---

## HyperspectralEDA

```python
import numpy as np
from viseda import HyperspectralEDA

# One wavelength value per spectral band
wavelengths = np.load(
    "path/to/wavelengths.npy"
)

eda = HyperspectralEDA(
    wavelengths=wavelengths,
    compute_glcm=False,
    compute_pca=True,
)

# File-backed loading is recommended when using compute_index()
# or pca_scores().
eda.load(
    "path/to/cube.npy"
)

summary = eda.summary()

print(summary["inventory"])
print(summary["spectral_quality"])

ndvi = eda.compute_index(
    cube_index=0,
    index_name="ndvi"
)

scores, variance_ratio = eda.pca_scores(
    cube_index=0,
    n_components=3
)

eda.plot(
    cube_index=0,
    save_path="hyperspectral_cube.png"
)

eda.report(
    "viseda_hyper_report.html"
)
```

For a directory of hyperspectral cubes:

```python
from viseda import HyperspectralEDA

eda = HyperspectralEDA()

eda.load(
    "path/to/hyperspectral_dataset",
    label_from_parent=True
)

eda.plot_dataset(
    save_path="hyperspectral_dataset.png"
)
```

> **Current 1.0.0 note:** `load_arrays()` is suitable for summary and plotting workflows, but `compute_index()` and `pca_scores()` reload the selected cube from its source path. Use `load()` with a file-backed cube when calling those two methods.

---

## PointCloudEDA

```python
import numpy as np
from viseda import PointCloudEDA

points = np.random.default_rng(0).random(
    (10000, 3)
).astype("float32")

eda = PointCloudEDA(
    max_points_per_cloud=200000,
    compute_neighbors=True,
    compute_geometry=True,
)

eda.load_arrays(
    [points],
    labels=["sample"]
)

summary = eda.summary()

print(summary["inventory"])
print(summary["geometry"])
print(summary["quality"])

eda.plot_dataset(
    save_path="pointcloud_dashboard.png"
)

eda.report(
    "viseda_pointcloud_report.html"
)
```

For an individual cloud:

```python
eda.plot(
    cloud_index=0,
    save_path="single_cloud.png"
)
```

---

## TextEDA

```python
from viseda import TextEDA

eda = TextEDA()

eda.load_texts(
    [
        "Exploratory data analysis should come before model training.",
        "TextEDA analyses vocabulary, length, readability and duplicates.",
    ],
    labels=["eda", "eda"],
)

summary = eda.summary()

print(summary["length"])
print(summary["lexical"])

print(
    eda.vocabulary(top_n=10)
)

eda.plot_dataset(
    save_path="text_dashboard.png"
)

eda.report(
    "viseda_text_report.html"
)
```

TextEDA also supports directory datasets and structured files such as CSV, TSV, JSON, JSONL, Markdown, HTML, and plain text.

---

# Command-Line Interface

VisEDA installs a unified command:

```bash
viseda --help
```

Available subcommands:

```text
viseda image ...
viseda video ...
viseda hyper ...
viseda cloud ...
viseda text ...
```

### Image dataset

```bash
viseda image "C:/datasets/images" \
    --label-from-parent \
    --plot \
    --report viseda_image_report.html
```

### Video dataset

```bash
viseda video "C:/datasets/videos" \
    --label-from-parent \
    --plot \
    --report viseda_video_report.html
```

### Hyperspectral dataset

```bash
viseda hyper "C:/datasets/hyper" \
    --label-from-parent \
    --plot \
    --dataset-plot \
    --report viseda_hyper_report.html
```

### Point-cloud dataset

```bash
viseda cloud "C:/datasets/pointclouds" \
    --label-from-parent \
    --plot \
    --report viseda_pointcloud_report.html
```

### Text dataset

```bash
viseda text "C:/datasets/text" \
    --label-from-parent \
    --plot \
    --report viseda_text_report.html
```

For command-specific options:

```bash
viseda image --help
viseda video --help
viseda hyper --help
viseda cloud --help
viseda text --help
```

---

# HTML Reports

Every VisEDA module can generate a standalone HTML report.

```python
eda.report("report.html")
```

Reports are designed to provide a portable summary of dataset properties and quality indicators that can be opened directly in a browser.

Typical report sections include:

- dataset inventory
- labels/classes
- spatial or structural statistics
- quality metrics
- modality-specific statistics
- duplicate/similarity information where applicable
- summary distributions
- analysis metadata

---

# Visualisation Workflows

The dataset-level dashboard method is module-specific:

```python
# ImageEDA
image_eda.plot()

# VideoEDA
video_eda.plot_dataset()

# HyperspectralEDA
hyper_eda.plot_dataset()

# PointCloudEDA
cloud_eda.plot_dataset()

# TextEDA
text_eda.plot_dataset()
```

Specialised plotting methods are also available for modality-specific tasks such as:

- sample previews
- colour analysis
- quality analysis
- temporal analysis
- spectral analysis
- PCA analysis
- point-cloud geometry
- vocabulary and n-gram frequency
- pairwise similarity/distance matrices

---

# Supported Input Formats

| Module | Main supported inputs |
|---|---|
| **ImageEDA** | JPG/JPEG, PNG, BMP, TIFF and other supported image formats |
| **VideoEDA** | MP4, AVI, MOV, MKV, WEBM, MPEG/MPG, M4V |
| **HyperspectralEDA** | NPY, NPZ, MAT, ENVI HDR/BIL/BIP/BSQ/ENVI, TIF/TIFF |
| **PointCloudEDA** | NPY, NPZ, TXT, CSV, XYZ, PTS, ASCII PLY, LAS/LAZ |
| **TextEDA** | TXT/TEXT, MD, RST, LOG, HTML/HTM, CSV/TSV, JSON, JSONL/NDJSON |

Some formats require optional dependencies.

---

# Typical Workflow

```text
Raw Dataset
    │
    ▼
Load with VisEDA
    │
    ▼
Dataset Inventory
    │
    ▼
Quality & Distribution Analysis
    │
    ▼
Duplicate / Similarity Checks
    │
    ▼
Visual Diagnostics
    │
    ▼
HTML Report
    │
    ▼
Preprocessing Decisions
    │
    ▼
Model Training
```

---

# Package Structure

```text
viseda/
├── image/
├── video/
├── hyperspectral/
├── pointcloud/
├── text/
├── report/
├── core/
├── utils/
├── cli.py
└── __init__.py
```

The main classes can be imported directly:

```python
from viseda import (
    ImageEDA,
    VideoEDA,
    HyperspectralEDA,
    PointCloudEDA,
    TextEDA,
)
```

---

# Development

Clone the repository and install in editable mode:

```bash
python -m pip install -e ".[dev]"
```

Run the tests:

```bash
python -m pytest
```

Build a distribution:

```bash
python -m pip install --upgrade build twine

python -m build
python -m twine check dist/*
```

---

# Documentation

The VisEDA documentation covers all five modules in detail, including:

- installation
- quick-start workflows
- complete metric explanations
- visualisation interpretation
- Python API
- command-line interface
- HTML reports
- test workflows
- advanced use cases
- troubleshooting
- API reference
- changelog and roadmap

---

# Citation

If you use **VisEDA** in your research, publication, dissertation, thesis, teaching, or software project, please cite the software as follows:

> **I.O.Agyemang, D. Acheampong, A.A.Baffour, & I. Adjei-Mensah**  
> *VisEDA: A Unified Visual Exploratory Data Analysis Toolkit for Image, Video, Hyperspectral, Point Cloud, and Text Data.*  
> GitHub repository: https://github.com/Isaac45/VisEDA

## Authors

**Isaac Osei Agyemang**  
Data Science and Big Data Technology, Stirling College, Chengdu University, Chengdu 610054, P.R. China

**Daniel Acheampong**  
Lutgert College, Florida Gulf Coast University, USA

**Adu Asare Baffour**  
School of Science and Engineering, University of Missouri-Kansas City, USA

**Isaac Adjei-Mensah**  
College of Artificial Intelligence, Yango University, Fuzhou 350015, P.R. China  
Fujian University Engineering Research Center of Spatial Data Mining and Applications, Yango University, Fuzhou 350015, P.R. China

## BibTeX

```bibtex
@software{agyemang_viseda,
  author = {
    Agyemang, Isaac Osei and
    Acheampong, Daniel and
    Baffour, Adu Asare and
    Adjei-Mensah, Isaac
  },
  title = {
    VisEDA: A Unified Exploratory Data Analysis Toolkit for
    Image, Video, Hyperspectral, Point Cloud, and Text Data
  },
  url = {https://github.com/Isaac45/VisEDA},
  note = {Python software library}
}
```

> If a DOI or peer-reviewed VisEDA publication becomes available, this section can be updated with the formal publication details.

---

# Contributing

Contributions are welcome.

Useful contributions include:

- bug reports
- additional file-format support
- new EDA metrics
- additional visualisations
- documentation improvements
- performance optimisations
- tests and reproducible examples

Before proposing major API changes, please open an issue describing the motivation and expected behaviour.

---

# License

VisEDA is released under the **MIT License**.

See `LICENSE` for details.

---

<div align="center">

### VisEDA

**Explore first. Model second.**

ImageEDA · VideoEDA · HyperspectralEDA · PointCloudEDA · TextEDA

</div>
