Metadata-Version: 2.5
Name: mat-summarize
Version: 1.0.0
Summary: A package of methods to summarize MATs
Project-URL: issues, https://github.com/RepresentantativeMAT/mat-summarization/issues
Project-URL: source, https://github.com/RepresentantativeMAT/mat-summarization
Author-email: Vanessa Lago Machado <vanessalagomachado@gmail.com>, Jonathan Roberto Fragoso Bonatto <jonathanrfbonatto@gmail.com>
Maintainer-email: Vanessa Lago Machado <vanessalagomachado@gmail.com>, Jonathan Roberto Fragoso Bonatto <jonathanrfbonatto@gmail.com>
License-Expression: GPL-3.0-only
License-File: LICENSE
Keywords: MAT,summarization,summarize,trajectory
Classifier: License :: OSI Approved :: GNU General Public License v3 (GPLv3)
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python
Requires-Python: >=3.9
Requires-Dist: numpy>=2.0
Requires-Dist: pandas>=2.2
Requires-Dist: scipy>=1.14
Provides-Extra: viz
Requires-Dist: folium>=0.16; extra == 'viz'
Requires-Dist: plotly>=5.18; extra == 'viz'
Provides-Extra: web
Requires-Dist: streamlit>=1.32; extra == 'web'
Description-Content-Type: text/markdown

# MAT-data: Multiple Aspect Trajectory Summarization Modeling (MAT-SG & MAT-SGT)

[![Python Version](https://img.shields.io/badge/python-%3E=3.9-blue.svg)](https://www.python.org/downloads/)
[![License: GPL v3](https://img.shields.io/badge/License-GPLv3-blue.svg)](https://www.gnu.org/licenses/gpl-3.0)
[![PyPI version](https://badge.fury.io/py/mat-summarize.svg)](https://badge.fury.io/py/mat-summarize)

This repository contains the Python implementation for **Multiple Aspect Trajectory Summarization** algorithms. Initially based on Java, the **MAT-SG** (Multiple-Aspect Trajectories based on a Spatial Grid) and **MAT-SGT** (Multi-Aspect Trajectories Based on Spatio-Temporal Segmentation) models were rewritten and optimized in object-oriented Python using the **Template Method** design pattern to process massive datasets of enriched trajectories.

## 📌 Project Overview

The central goal of this library is to reduce large volumes of trajectory points while maintaining spatial, temporal, and semantic fidelity. This is achieved through the **clustering and definition of representative points** that carry proportional semantics (e.g., Weather, POI locations, Prices, etc.).

The semantic clustering supports the summarization of both **Numerical** variables, which generate average scores via the median, and **Categorical** variables, preserving the weighted/proportional distribution of values within the consolidated clusters.

### 📍 Available Algorithm Patterns

There are two fundamental variations of spatial analysis applied in this model. For a complete theoretical foundation, refer to the [Approaches (MAT-SG vs MAT-SGT)](docs/approaches.md) guide.

1. **MAT-SG** (Multiple-Aspect Trajectories based on a Spatial Grid): Clustering elements based solely on a **Spatial** threshold and grid. It summarizes semantics to generate mathematically calculated representative centroid points while discarding excessive outlier points.
2. **MAT-SGT** (Multi-Aspect Trajectories Based on Spatio-Temporal Segmentation): Incorporates a **Temporal** threshold, clustering spatial cells that are not only spatially close but also co-exist within the same contiguous time interval of the day. This groups spatial cells that compose time and space (`STI` - Spatial-Temporal Intervals).

### 🔗 Feature-Based Extension (Semantic Dependency)
Both MAT-SG and MAT-SGT were extended to extract **Composed Features**. Instead of summarizing attributes independently (which loses context, like assuming a POI and a Price have no relation), the framework allows users to explicitly define dependencies (e.g., `(POI, Price)`), when user identifyied the need to do so, in order to preserve the contextual distribution (e.g., preserving that `{Restaurant, 20}` happened together). For details, see [Feature-Based Summarization](docs/Feature_Summarization_Method.md).

### 📝 Natural Language Narrative Generation (`NarrativeGenerator`)
In addition to outputting the structured Representative Trajectory (RT) in `.csv` format, MAT-Summarize automatically generates a **Natural Language Narrative** (`<output_file>_narrative.md` and console summary) via the deterministic `NarrativeGenerator` engine. It translates spatial clusters, temporal intervals (`STI`), and semantic probability distributions into a human-readable analyst report tailored to the active method (**Regions of Interest** for `MAT-SG` and **Chronological Routine / Day-Phase Condensed Timeline** for `MAT-SGT`). For details, see [Natural Language Narrative Generation](docs/Narrative_Generator.md).

---

## 🏗 Repository Structure and Architecture

The repository is structured following clean formal design patterns, specifically leveraging the **Template Method**. For a deep dive into the architectural design, refer to the [Architecture and Inheritance](docs/architecture.md) documentation.

### Class Architecture (Template Method & SOLID)

```mermaid
classDiagram
    %% Core Algorithms (Method)
    class MATSummarize {
        <<abstract>>
        # _trc: float
        # _trv: float
        # _representative_trajectory: MultipleAspectTrajectory
        + method_name: str
        + load()
        + compute_min_spatial_threshold()
        + identify_and_process_cells()
        + fuse_aspects()
        + summarize_numerical_aspects()
        + summarize_categorical_aspects()
        + write_representative_trajectory()
        + generate_natural_language_summary()
        + generate_visual_report()
        + execute() SummarizationResult
        # process_cell_points(points, cell_id) *abstract*
    }

    class ThresholdOptimizer {
        + is_auto_threshold(val) bool
        + estimate_initial_thresholds() tuple
        + compute_composite_score() float
        + optimize(summarizer) tuple
    }

    class NarrativeGenerator {
        + generate(full: bool) str
        - _build_regions(points)
        - _build_timeline(points)
        - _build_condensed_timeline(points)
    }

    class VisualReportGenerator {
        + generate_html() str
        + save(html_output_path) str
        + generate_comparison_html(comp) str
        + save_comparison_report(comp, path) str
    }

    class MATComparator {
        + execute() ComparisonResult
    }

    class MATSGT {
        + method_name: str
        + summarize_temporal_aspect(times, stis)
        # process_cell_points(points, cell_id) *override*
    }

    class MATSG {
        + method_name: str
        - _define_ranking_temporal()
        # process_cell_points(points, cell_id) *override*
    }

    %% Data Entities (Model)
    class MultipleAspectTrajectory {
        + int id
        + list point_list
    }

    class Centroid {
        + tuple cell_reference
        + STI sti
        + list point_list_source
    }

    class Point {
        + float x
        + float y
        + TemporalAspect time
        + dict list_feat_values
    }
    
    class TemporalAspect {
        + time start_time
        + time end_time
    }

    class SemanticAspect {
        + str name
        + SemanticType type
    }

    %% Inheritances and Relationships
    MATSummarize <|-- MATSGT
    MATSummarize <|-- MATSG
    MATSummarize --> ThresholdOptimizer : Optimizes (rc, rv, z)
    MATSummarize --> NarrativeGenerator : Generates Narrative (.md)
    MATSummarize --> VisualReportGenerator : Generates Report (.html)
    MATComparator --> MATSG : Runs
    MATComparator --> MATSGT : Runs
    MATComparator --> VisualReportGenerator : Generates Comparative HTML
    
    MATSummarize --> MultipleAspectTrajectory : Populates
    MultipleAspectTrajectory *-- Point : Contains 1..*
    Centroid --|> Point : Extends
    Point --> TemporalAspect : Has
    Point *-- SemanticAspect : Keys list_feat_values
```

### Directory Layout

```text
/
├── README.md                       # Entry point project documentation
├── docs/                           # Extended project documentation
│   ├── architecture.md             # Template Method & SOLID architecture guide
│   ├── approaches.md               # Theoretical overview (MAT-SG vs. MAT-SGT)
│   ├── Feature_Summarization_Method.md # Composed Features specification
│   ├── Narrative_Generator.md      # Natural Language Narrative Engine guide
│   ├── Modification_Log.md         # Bilingual changelog & dynamic threshold ($P_z$) guide
│   └── Refactoring_and_SOLID_Log.md # Code cleanup, bug fixes & SOLID refactoring log
├── data/                           # Local datasets manipulation
│   ├── input/                      # CSV spreadsheet repository for raw modeling
│   └── output/                     # Representative trajectories, narratives & visual reports
├── src/mat_summarize/              # Main package
│   ├── method/                     # Core clustering, optimization, narrative & HTML reports
│   │   ├── MATSummarize.py         # Base Abstract Class (Template Method Pattern)
│   │   ├── MATSG.py                # Subclass applying Spatial Grid heuristics
│   │   ├── MATSGT.py               # Subclass adding partitioning via Time STIs
│   │   ├── ThresholdOptimizer.py   # Statistical/Entropic (rc, rv) estimation & refinement
│   │   ├── NarrativeGenerator.py   # Deterministic natural-language narrative generator (.md)
│   │   ├── VisualReportGenerator.py# Zero-dependency interactive HTML5 report generator (.html)
│   │   ├── MATComparator.py        # Automated MAT-SG vs. MAT-SGT comparator & advisor
│   │   └── MUITAS.py               # Multiple-Aspect Trajectory Similarity measure
│   ├── model/                      # Structural and Geometric Models
│   │   ├── Centroid.py             # Representative stop/region entity
│   │   ├── ComparisonResult.py     # Structured output of MATComparator
│   │   ├── MultipleAspectTrajectory.py # Trajectory container entity
│   │   ├── Point.py                # Raw trajectory point with tuple-based list_feat_values
│   │   ├── STI.py                  # Significant Temporal Interval model
│   │   ├── SemanticAspect.py       # Semantic dimension descriptor
│   │   ├── SemanticType.py         # Enum (CATEGORICAL, NUMERICAL)
│   │   ├── SummarizationResult.py  # Structured output of MATSG / MATSGT
│   │   ├── TemporalAspect.py       # Time interval wrapper
│   │   └── Util.py                 # Geometric and temporal utility functions
│   └── viz/                        # Optional interactive Web application (`pip install "mat-summarize[web]"`)
│       └── web_app.py              # Streamlit visual analytics tool (`mat-summarize-web`)
```

---

## 🚀 How to Install and Run

### Prerequisites
*   Python 3.9 or later
*   NumPy
*   Pandas
*   SciPy

### 1. Installation

Install the package from PyPI using `pip`:

```bash
pip install mat-summarize
```

**For local development:**
Clone the repository and install it in editable mode:
```bash
git clone https://github.com/RepresentantativeMAT/mat-summarization.git
cd mat-summarization
pip install -e .
```

**With the optional Interactive Web Application (`Streamlit`):**
```bash
pip install -e ".[web]"
```

### 2. Running MAT-Summarize

#### Step 1: Import a Method

MAT-Summarize provides two core algorithms (`MATSG` and `MATSGT`) as well as an automated comparator (`MATComparator`):

```python
from mat_summarize import MATSG, MATSGT, MATComparator
```

#### Step 2: Initialize the Method

Create an instance of the selected method (passing explicit thresholds or `'auto'` for dynamic threshold optimization):

```python
model_sgt = MATSGT(
    trc=0.25,   # or 'auto' (default)
    trv=0.25,   # or 'auto' (default)
    path='test_input.csv'
)
```

The constructor accepts the following parameters:

| Parameter | Description                                                                                                                                         |
| --------- | --------------------------------------------------------------------------------------------------------------------------------------------------- |
| `trc`     | Threshold for relevant cells (`tauRelevantCell`). Pass a float (e.g. `0.25`) or `'auto'`/`None` (default) to dynamically estimate and refine `rc`.  |
| `trv`     | Threshold for the representativeness value (`tauRepresentativenessValue`). Pass a float (e.g. `0.25`) or `'auto'`/`None` (default) for dynamic mode. |
| `path`    | Path to the input dataset.                                                                                                                          |

#### Step 3: Execute the Method

Run the method using the `execute()` function, which returns a structured `SummarizationResult` object (`result.rt_df`, `result.input_df`, `result.metrics`, `result.narrative`, `result.csv_path`, `result.narrative_path`, `result.html_path`):

```python
result = model_sgt.execute(
    lst_categorical_pd=['price'],
    values_null=[-1, -999],
    dir='data/',
    file='test_output',
    ignore_columns=None,
    features=[('poi',), ('price',), ('weather', 'precip')]
)
```

The `execute()` function accepts the following parameters:

| Parameter            | Description                                                                                                                                                                                             |
| -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `lst_categorical_pd` | List of numerical columns that should always be treated as categorical. Default value is None, meaning that all columns will be treated as numerical values by default if is composed by numbers.                   |
| `values_null`        | List of values that represent missing data. Default value is None, indicating that there are no null values to be treated.                                                                              |
| `dir`                | Path to the output directory.                                                                                                                                                                                                                                            |
| `file`               | Name of the output file.                                                                                                                                                                                                                                               |
| `ignore_columns`     | List of dataset columns to exclude from processing. Use `None` to include all columns.                                                                                                                   |
| `features`           | List of tuples specifying the attributes to be used by the method.                                                                                       |

#### Step 4 (Optional): Compare Both Methods & Auto-Recommend (`MATComparator`)

To run both `MAT-SG` and `MAT-SGT` on the same dataset, diagnose intra-cell temporal recurrence, and receive an automated method recommendation (`ComparisonResult`):

```python
from mat_summarize import MATComparator

comp = MATComparator(trc='auto', trv='auto', path='test_input.csv')
comp_result = comp.execute(
    dir='data/',
    file='test_output',
    lst_categorical_pd=['price'],
    values_null=[-1, -999],
    ignore_columns=None,
    features=[('poi',), ('price',), ('weather', 'precip')]
)
print("Recommended method:", comp_result.recommended_method)
print(comp_result.comparison_df)
```

#### Step 5 (Optional): Launch the Interactive Web Application (`mat-summarize-web`)

If installed with `pip install -e ".[web]"`, launch the Streamlit visual analytics application:

```bash
mat-summarize-web
# or: PYTHONPATH=src streamlit run src/mat_summarize/viz/web_app.py
```

The Web application dynamically inspects any uploaded trajectory CSV containing `tid`, `lat_lon`, and `date_time` (or `time`), discovers all semantic attributes automatically, and allows the analyst to parameterize ignored columns, numerical-to-categorical conversions, single/composed `features`, threshold mode (`auto` vs. manual sliders), and method execution (`MAT-SG`, `MAT-SGT`, or both via `MATComparator`).

### 3. Output

After execution, MAT-Summarize produces three complementary, zero-dependency outputs in the target directory (creating the directory automatically if it does not exist), using a timestamped naming convention so previous runs are never overwritten:

1. **Representative Trajectory CSV (`{file}_rc{rc}_rv{rv}_{method}_{YYYYMMDD_HHMMSS}.csv`):**
   For the example above, the CSV file will be named:

   ```text
   test_output_rc25_rv25_MATSGT_20261104_110000.csv
   ```

   The resulting CSV contains a numerical and temporal condensation of the input data. Some attributes may be represented using dictionary-like notation, such as for the POI attribute:

   ```text
   [{HOME: 0.67; LIBRARY: 0.33}]
   ```

   The output header provides information about:
   * The method used (`MAT-SG` or `MAT-SGT`).
   * The coverage achieved.
   * The reduction applied to the original dataset (including the optimal `thresholdCellSize` $z$).

2. **Natural Language Narrative (`{file}_rc{rc}_rv{rv}_{method}_{YYYYMMDD_HHMMSS}_narrative.md` & Console Output):**
   Alongside the CSV, `MATSummarize` invokes `NarrativeGenerator` to print a human-readable analytical report to the console and save it as a Markdown file:

   ```text
   test_output_rc25_rv25_MATSGT_20261104_110000_narrative.md
   ```

   * **For `MAT-SG`:** Characterizes the identified **Regions of Interest** ordered by point coverage, detailing region profiles, temporal visit distributions, typical days, environmental context, price/unpriced distributions, and an overall synthesis.
   * **For `MAT-SGT`:** Describes the **Temporal Sequence of the Identified Routine** chronologically. When the RT exceeds 8 stops, the narrative automatically condenses the timeline into chronological **Phases of the Day** (*Early morning*, *Morning*, *Midday*, *Afternoon*, *Evening*, *Night*), highlights the **Most Relevant Stops** and **Day Pattern Contrasts**, and appends the full stop-by-stop sequence at the end of the `_narrative.md` file. See [Narrative Generator Documentation](docs/Narrative_Generator.md) for full details.

3. **Interactive Visual Report (`{file}_rc{rc}_rv{rv}_{method}_{YYYYMMDD_HHMMSS}_report.html`):**
   `VisualReportGenerator` automatically exports a self-contained HTML5 dashboard (requiring no external Python plotting packages) with:
   * **Visão 1 — Interactive Spatial Map (Leaflet Street Map & Plotly 2D Grid Plot):** Toggleable layers (raw trajectories, grid cells, coverage rays, Representative Trajectory). For `MAT-SG`, renders **zoom-adaptive region circles** (`Region 1`, `Region 2`...) that scale with map zoom to always enclose each region's covered points without directional arrows; for `MAT-SGT`, renders sequential chronological stops connected by directional arrows. Clicking any representative stop/region **highlights its originating input points** (`#facc15` markers and rays) and displays a rich tooltip/popup containing both **temporal information (`STI`/windows)** and **semantic information**.
   * **Visão 2 — Spatio-Temporal Timeline & Coverage Charts:** Plots 24-hour temporal intervals solely as **semi-transparent horizontal bars** so the original covered input points inside each interval remain clearly visible, alongside per-stop point/trajectory coverage bar charts and the full natural-language narrative.
   * **Comparative Dashboard (`MATComparator`):** Exports `{file}_comparison_{YYYYMMDD_HHMMSS}_report.html` and `..._narrative.md` with side-by-side `MAT-SG` vs. `MAT-SGT` maps, spatio-temporal interval timelines, metric comparisons, and narratives.

This allows you to evaluate mathematically, visually, and in natural language how the input data was condensed while preserving relevant information.

---

## 🛠️ Recent Refactoring Contributions (Changelog)

-   **Interactive Web Application (`viz/web_app.py`), Visual HTML Reports (`VisualReportGenerator`) & Method Comparator (`MATComparator`):** Added zero-dependency standalone HTML5 dashboards (`_report.html`), automated side-by-side `MAT-SG` vs. `MAT-SGT` comparison and recommendation (`MATComparator`), structured return models (`SummarizationResult`, `ComparisonResult`), and a 100% schema-driven Streamlit web application (`mat-summarize-web`) that dynamically discovers dataset columns and lets users configure composed features, categorical overrides, and thresholds interactively.
-   **Code Cleanup, Bug Fixes & SOLID Refactoring:** Removed obsolete legacy modules (`Feature.py`, `FeatureAggregator.py`), standardized the `process_cell_points()` LSP contract across `MATSG` and `MATSGT` (eliminating duplicate aspect fusion), activated the dynamic statistical temporal threshold (`+ threshold`) in `MATSGT.create_temporal_intervals()`, extracted `ThresholdOptimizer.py` (SRP), centralized zero-dependency HTML reporting in `method/VisualReportGenerator.py`, and fixed latent model/narrative bugs (including preserving unpriced/null share proportions in `NarrativeGenerator`). See [Code Cleanup & SOLID Refactoring Log (PT-BR / EN)](docs/Refactoring_and_SOLID_Log.md).
-   **Hybrid Dynamic Threshold Selection (`rc` & `rv`):** Added automatic estimation and local coordinate refinement of `trc` and `trv` (`trc='auto', trv='auto'`) based on spatial occupancy density, Shannon perplexity ($e^H$), and a composite `MUITAS` + coverage + conciseness score. See the bilingual [Modification Log (PT-BR / EN)](docs/Modification_Log.md).
-   **Natural Language Narrative Generation (`NarrativeGenerator`):** Deterministic generation of human-readable reports (`_narrative.md`) for both `MAT-SG` and `MAT-SGT`. See [Narrative Generator Documentation](docs/Narrative_Generator.md).
-   **Timestamped Output Naming Convention:** Standardized output files to `{file}_rc{rc}_rv{rv}_{method}_{YYYYMMDD_HHMMSS}.csv` and `..._narrative.md` to prevent overwriting previous runs.
-   Architectural migration based entirely on **Classic Python (`snake_case`)** OOP, eliminating legacy Java *camelCase* fragments.
-   Integration of the **Template Method** design pattern, concentrating inheritance and abstraction in the parent class `MATSummarize.py`, reducing redundant code by over 50%.
-   Fixed the `Java Skip Indexing` bug, refining the accuracy of the Outlier Removal Process (which previously spared some outliers by accident).
-   Reliable adherence to probability normalization, ensuring a mathematical `sum = 1.0`.
-   Removed dead code using `Vulture`.

## 📄 Model Authors / Contact
This research was partially funded by the SoBigData++ Project via Transnational Access (TNA), as well as the European Union’s Horizon 2020 research and innovation programme under GA N. 777695 (EU Project MASTER - Multiple ASpects TrajEctoRy management and analysis) until 2024. 

Currently, this research is associated with the **Análise Avançada de Dados de Trajetórias** project (EDITAL PROPESP 05-2025). We gratefully acknowledge that the conversion of the methods to Python, the extension of the method for the use of Features, and the utilization of the Template Method pattern were realized with the support and assistance of this IFSUL research project.

For a formal and in-depth architectural understanding based on the original research articles, refer to:
- **MAT-SG ([Springer Link](https://link.springer.com/chapter/10.1007/978-3-031-12423-5_33))**: *Multiple-Aspect Trajectory Summarization based on a spatial Grid*
- **MAT-SGT ([JIDM](https://journals-sol.sbc.org.br/index.php/jidm/article/view/4110))**: *Multiple-Aspect Trajectory Summarization via Grid and Time*

## 📞 Contacts
- [Vanessa Lago Machado](mailto:vanessalagomachado@gmail.com)
- [Jonathan Roberto Fragoso Bonatto](mailto:jonathanrfbonatto@gmail.com)