Metadata-Version: 2.4
Name: refdatagen
Version: 0.1.1
Summary: Refactoring dataset generator
Author: vlvlk
License-Expression: MIT
Requires-Python: >=3.12
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pydantic>=2.13.4
Requires-Dist: python-dotenv>=1.2.2
Requires-Dist: tree-sitter<0.26,>=0.24.7
Requires-Dist: tree-sitter-language-pack>=1.14.3
Requires-Dist: libcst>=1.0.0
Requires-Dist: radon>=6.0.0
Requires-Dist: lizard>=1.23.0
Requires-Dist: pyarrow>=25.0.0
Requires-Dist: loguru>=0.7.3
Requires-Dist: aiohttp>=3.9.0
Requires-Dist: certifi>=2024.2.0
Requires-Dist: GitPython>=3.1.0
Requires-Dist: redis>=8.0.1
Dynamic: license-file

# Refactoring Dataset Generator

[![Python](https://img.shields.io/badge/Python-3.12%20%7C%203.13%20%7C%203.14-blue.svg)](pyproject.toml)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
[![Type checking: mypy strict](https://img.shields.io/badge/type%20checking-mypy%20strict-blueviolet.svg)](pyproject.toml)
[![Code style: ruff](https://img.shields.io/badge/code%20style-ruff-000000.svg)](ruff.toml)

**Refactoring Dataset Generator** (`refdatagen`) is a tool for automatically collecting and building **refactoring datasets** from open-source repositories. It scans real-world projects on GitHub, finds concrete examples of how developers refactor code, and packages those examples into a structured dataset suitable for analysis or for training machine-learning models.

> ⚠️ **Early stage.** This project is under active development. The public API, CLI commands, and configuration options may change without notice between releases. Treat it as a work in progress and pin the version you depend on.

---

## Table of Contents

- [Why this project exists](#why-this-project-exists)
- [Features](#features)
- [How it works](#how-it-works)
- [Architecture](#architecture)
- [Technology stack](#technology-stack)
- [Installation](#installation)
- [Quick start](#quick-start)
- [Usage](#usage)
  - [Command-line interface](#command-line-interface)
  - [Python API](#python-api)
- [Configuration](#configuration)
- [Output formats](#output-formats)
- [Project structure](#project-structure)
- [Troubleshooting](#troubleshooting)
- [FAQ](#faq)
- [Documentation](#documentation)
- [Contributing](#contributing)
- [License](#license)

---

## Why this project exists

Refactoring is the process of changing code **without changing its behavior** — renaming symbols, extracting methods, simplifying conditions, and so on. Real-world examples of refactoring are valuable for:

- Training models that can **suggest refactorings**;
- Analyzing development practices in real projects;
- Building benchmarks for code-analysis tools.

This project automates the otherwise tedious process of finding, verifying, and packaging such examples.

---

## Features

- **Repository collection** from GitHub (with GitLab and local collectors available) filtered by programming language and minimum star count.
- **Refactoring detection** using three interchangeable strategies:
  - **Regex** — fast, pattern-based detection;
  - **AST** — tree-sitter based comparison of parse trees before/after a change;
  - **ML** — machine-learning based detection (scaffold).
- **Evidence-rich output** — each refactoring includes a diff, AST changes, and complexity metrics (e.g., cyclomatic complexity reduction).
- **Multiple export formats** — JSON, Parquet (via PyArrow), and HuggingFace Datasets.
- **Caching** — memory, disk, and Redis backends to speed up repeated collection runs.
- **Clean Architecture** with a dependency-injection container, making the tool easy to extend and test.
- **Strict typing** — mypy in strict mode, ruff linting, and pre-commit hooks.

### Component status

| Component | Status |
| ----------- | -------- |
| Regex detector | ✅ ready |
| AST detector (tree-sitter + Java) | ✅ ready |
| ML detector | 🚧 roadmap |
| GitHub collector | ✅ ready |
| GitLab collector | 🚧 roadmap |
| Local collector | ✅ ready |
| Memory / Disk / Redis cache | ✅ ready |
| Metrics (radon + lizard) | ✅ ready |
| JSON / Parquet / HuggingFace export | ✅ ready |
| `ValidateDatasetUseCase` | ✅ ready |

---

## How it works

The project implements a three-stage pipeline:

```mermaid
flowchart LR
    A[Collect repositories<br/>GitHub / GitLab / local] --> B[Detect refactorings<br/>regex / AST / ML]
    B --> C[Build dataset<br/>JSON / Parquet / HuggingFace]
```

1. **Collect repositories** — [`CollectRepositoriesUseCase`](src/refdatagen/application/use_cases/collect_repositories.py) finds repositories by language and star count (via [`GitHubCollector`](src/refdatagen/infrastructure/collectors/github_collector.py); GitLab and local collectors are also available).
2. **Detect refactorings** — [`DetectRefactoringsUseCase`](src/refdatagen/application/use_cases/detect_refactorings.py) analyzes commits and finds refactorings using one of three detectors:
   - [`RegexRefactoringDetector`](src/refdatagen/infrastructure/detectors/regex_refactoring_detector.py) — regular expressions;
   - [`ASTRefactoringDetector`](src/refdatagen/infrastructure/detectors/ast_refactoring_detector.py) — tree-sitter parse-tree comparison before/after;
   - [`MLDetector`](src/refdatagen/infrastructure/detectors/ml_detector.py) — machine learning.
3. **Build the dataset** — [`BuildDatasetUseCase`](src/refdatagen/application/use_cases/build_dataset.py) combines the results and exports them to JSON, Parquet, or HuggingFace.

---

## Architecture

The project follows **Clean Architecture** with a clear separation of layers:

```mermaid
flowchart TB
    subgraph Presentation
        CLI[CLI<br/>refdatagen.cli]
        API[Public API<br/>refdatagen.api]
    end
    subgraph Application
        UC[Use Cases<br/>collect / detect / build / validate]
        DTO[DTO]
        IF[Interfaces<br/>collector / detector / cache / exporter]
    end
    subgraph Domain
        ENT[Entities<br/>Repository / Refactoring / Dataset / Commit]
        VO[Value Objects<br/>Language / License / Metrics]
        EV[Events]
    end
    subgraph Infrastructure
        COL[Collectors<br/>GitHub / GitLab / local]
        DET[Detectors<br/>regex / AST / ML]
        CACHE[Cache<br/>memory / disk / redis]
        EXP[Exporters<br/>JSON / Parquet / HuggingFace]
    end
    Presentation --> Application --> Domain
    Application --> Infrastructure
```

Key characteristics:

- **DI container** — [`Container`](src/refdatagen/bootstrap.py:28) supports dependency overrides, which is convenient for tests and mock injection.
- **Interfaces** in [`application/interfaces/`](src/refdatagen/application/interfaces/) decouple business logic from concrete implementations.
- **Domain entities** — [`Refactoring`](src/refdatagen/domain/entities/refactoring.py) carries evidence (diff, AST changes, complexity metrics), [`Repository`](src/refdatagen/domain/entities/repository.py), [`Commit`](src/refdatagen/domain/entities/commit.py), and [`Dataset`](src/refdatagen/domain/entities/dataset.py).
- **Caching** — memory/disk/Redis to speed up repeated collection runs.
- **Events** — [`domain/events/`](src/refdatagen/domain/events/) (e.g., `refactoring_detected`, `dataset_completed`).

See [docs/architecture.md](docs/architecture.md) for a detailed design discussion.

---

## Technology stack

From [`pyproject.toml`](pyproject.toml):

| Area                 | Technology                                       |
|----------------------|--------------------------------------------------|
| Language             | Python ≥ 3.12 (3.12, 3.13, 3.14+)                |
| Validation           | Pydantic                                         |
| Code parsing         | tree-sitter, tree-sitter-language-pack, libcst   |
| Complexity metrics   | radon                                            |
| Parquet export       | PyArrow                                          |
| Async HTTP           | aiohttp                                          |
| Git operations       | GitPython                                        |
| Logging              | loguru                                           |
| CLI                  | argparse                                         |
| Type checking        | mypy (strict), pyright                           |
| Linting / formatting | ruff                                             |
| Testing              | pytest, pytest-asyncio, pytest-cov, pytest-mock  |
| CI hooks             | pre-commit                                       |

---

## Installation

### Prerequisites

- Python **3.12 or newer** (3.12, 3.13, 3.14+).
- A [GitHub personal access token](https://github.com/settings/tokens) (recommended) to avoid rate limits when collecting repositories.

### Install from source

```bash
git clone https://github.com/vlvlk/refactoring-dataset-generator.git
cd refactoring-dataset-generator

# Create and activate a virtual environment
python -m venv .venv
source .venv/bin/activate   # on Windows: .venv\Scripts\activate

# Install the package (editable, with dev dependencies)
pip install -e ".[dev]"
```

> The `[dev]` extra installs development tools (mypy, ruff, pytest, pre-commit, etc.). For a production-only install, use `pip install -e .`.

### Verify the installation

```bash
refdatagen --help
refdatagen info
```

---

## Quick start

The fastest way to see the tool in action is to collect a small set of Python repositories and detect refactorings in them:

```bash
# Set your GitHub token (optional but recommended)
export GITHUB_API_TOKEN="ghp_..."

# Collect 5 popular Python repos and detect refactorings with the AST detector
refdatagen collect \
  --language python \
  --min-stars 1000 \
  --max-results 5 \
  --detect \
  --detector-type ast \
  --commits-limit 10 \
  --output-dir ./data
```

This produces the following files in `./data`:

- `repositories_metadata.json` — collected repository metadata;
- `commits.json` — full commits (if detection ran);
- `refactorings.json` — detected refactorings;
- `dataset.json` — the assembled dataset (if `--build-dataset` is used).

---

## Usage

### Command-line interface

The CLI exposes four commands: `collect`, `detect`, `export`, and `info`.

#### Global options

| Option          | Description                    |
|-----------------|--------------------------------|
| `-v, --verbose` | Enable verbose (debug) logging |
| `-q, --quiet`   | Suppress non-essential output  |

#### `collect`

Collects repositories and optionally detects refactorings and builds a dataset.

```bash
refdatagen collect \
  --language python \
  --min-stars 1000 \
  --max-results 10 \
  --detect \
  --commits-limit 100 \
  --detector-type regex \
  --build-dataset \
  --output-format json \
  --output-dir ./data \
  --use-cache \
  --cache-type memory \
  --token ghp_...
```

| Option            | Default      | Description                                                |
|-------------------|--------------|------------------------------------------------------------|
| `--language`      | *(required)* | Programming language: `python`, `javascript`, `typescript` |
| `--min-stars`     | `1000`       | Minimum number of stars                                    |
| `--max-results`   | `10`         | Maximum number of repositories                             |
| `--detect`        | off          | Detect refactorings after collection                       |
| `--commits-limit` | `100`        | Commit limit per repository                                |
| `--detector-type` | `regex`      | Detector: `regex`, `ast`, `ml`                             |
| `--build-dataset` | off          | Build a dataset after detection (requires `--detect`)      |
| `--output-format` | `json`       | Dataset format: `json`, `parquet`, `huggingface`           |
| `--output-dir`    | `./data`     | Output directory                                           |
| `--use-cache`     | off          | Use the cache during collection                            |
| `--cache-type`    | `memory`     | Cache backend: `memory`, `disk`, `redis`                   |
| `--token`         | —            | GitHub token                                               |

#### `detect`

Detects refactorings in previously collected data.

```bash
refdatagen detect \
  --input ./data \
  --output ./data \
  --commits-limit 100 \
  --detector-type ast \
  --cache-type memory \
  --token ghp_...
```

| Option            | Default      | Description                                       |
|-------------------|--------------|---------------------------------------------------|
| `--input`         | *(required)* | Directory containing `repositories_metadata.json` |
| `--output`        | `./data`     | Output directory                                  |
| `--commits-limit` | `100`        | Commit limit per repository                       |
| `--detector-type` | `regex`      | Detector: `regex`, `ast`, `ml`                    |
| `--token`         | —            | GitHub token                                      |
| `--cache-type`    | `memory`     | Cache backend: `memory`, `disk`, `redis`          |

#### `export`

Exports a dataset from previously collected data.

```bash
refdatagen export \
  --input ./data \
  --output ./dataset.parquet \
  --output-format parquet
```

| Option | Default | Description |
| -------- | --------- | ------------- |
| `--input` | *(required)* | Directory with `repositories_metadata.json` and `refactorings.json` |
| `--output` | *(required)* | Path for the output dataset |
| `--output-format` | `json` | Dataset format: `json`, `parquet`, `huggingface` |

#### `info`

Prints project information and usage examples.

```bash
refdatagen info
```

### Python API

The public programmatic API lives in [`refdatagen.api`](src/refdatagen/api.py). All functions are `async`.

#### `collect_repositories(...)`

Collects repositories from GitHub and optionally detects refactorings and builds a dataset.

```python
import asyncio
from refdatagen.api import collect_repositories


async def main() -> None:
    repos, refactorings, dataset = await collect_repositories(
        language="python",
        min_stars=1000,
        max_results=5,
        output_dir="./data",
        detect=True,
        commits_limit=10,
        detector_type="ast",
        build_dataset=True,
        output_format="json",
        use_cache=False,
        cache_type="memory",
        github_token="ghp_...",
    )
    print(f"Repositories: {len(repos)}")
    print(f"Refactorings: {len(refactorings)}")
    if dataset:
        print(f"Dataset: {dataset.dataset_id} ({dataset.total_refactorings} refactorings)")


asyncio.run(main())
```

**Parameters:**

| Parameter | Default | Description |
| ------------ | --------- | ------------- |
| `language` | *(required)* | Programming language (`python`, `javascript`, `typescript`) |
| `min_stars` | `1000` | Minimum number of stars |
| `max_results` | `10` | Maximum number of repositories |
| `output_dir` | `./data` | Output directory |
| `detect` | `False` | Detect refactorings |
| `commits_limit` | `100` | Commit limit per repository |
| `detector_type` | `regex` | Detector: `regex`, `ast`, `ml` |
| `build_dataset` | `False` | Build a dataset after detection (only if `detect=True`) |
| `output_format` | `json` | Dataset format: `json`, `parquet`, `huggingface` |
| `use_cache` | `False` | Use the cache during collection |
| `cache_type` | `memory` | Cache backend: `memory`, `disk`, `redis` |
| `github_token` | `None` | GitHub token |

**Returns:** a tuple `(list[Repository], list[Refactoring], DatasetResponse | None)`.

#### `detect_refactorings(...)`

Detects refactorings in already-collected repositories.

```python
import asyncio
from refdatagen.api import detect_refactorings


async def main() -> None:
    refactorings = await detect_refactorings(
        input_dir="./data",
        output_dir="./data",
        commits_limit=100,
        detector_type="regex",
        github_token="ghp_...",
        cache_type="memory",
    )
    print(f"Found {len(refactorings)} refactorings")


asyncio.run(main())
```

**Returns:** `list[Refactoring]`.

#### `export_dataset(...)`

Exports a dataset from previously collected data.

```python
import asyncio
from refdatagen.api import export_dataset


async def main() -> None:
    dataset = await export_dataset(
        input_dir="./data",
        output_dir="./data",
        output_format="parquet",
    )
    print(f"Exported dataset {dataset.dataset_id}")


asyncio.run(main())
```

**Raises:** `ValueError` if there are no repositories or refactorings to export.

**Returns:** `DatasetResponse`.

---

## Configuration

Configuration is handled through environment variables and the [`Settings`](src/refdatagen/core/config/settings.py) dataclass. The project loads a `.env` file automatically (via `python-dotenv`).

| Environment variable | Default | Description |
| ---------------------- | --------- | ------------- |
| `GITHUB_API_TOKEN` | — | GitHub token used for repository collection |
| `GITHUB_TOKEN` | — | GitHub token (used by the DI container) |
| `GITLAB_TOKEN` | — | GitLab token |
| `REDIS_URL` | `redis://localhost:6379/0` | Redis connection URL (for the Redis cache) |
| `CACHE_TTL` | `300` | Default cache TTL in seconds |
| `API_RATE_LIMIT` | `100` | API rate limit |

Example `.env` file:

```dotenv
GITHUB_API_TOKEN=ghp_...
REDIS_URL=redis://localhost:6379/0
CACHE_TTL=600
```

---

## Output formats

### JSON

A human-readable JSON file (`dataset.json`) with `indent=2` and Unicode preserved. Nested structures (repositories, commits, refactorings) are kept intact.

### Parquet

A columnar Parquet file (`dataset.parquet`) written with PyArrow. Nested dictionaries are flattened; empty nested structures are serialized as JSON strings because Parquet does not support empty struct types.

### HuggingFace

A directory structure compatible with the HuggingFace `datasets` library:

```bash
output_path/
├── dataset_info.json
├── README.md
└── data/
    └── train.jsonl
```

This can be loaded with `datasets.load_dataset()`.

---

## Project structure

```bash
.
├── main.py                          # Standalone example entry point
├── pyproject.toml                   # Project metadata, dependencies, tool
├── ruff.toml                        # Ruff linting configuration
├── mypy.ini                         # mypy configuration
├── pytest.ini                       # pytest configuration
├── .coveragerc                      # Coverage configuration
├── .pre-commit-config.yaml          # pre-commit hooks
├── Makefile                         # Common development tasks
├── src/refdatagen/
│   ├── api.py                       # Public programmatic API
│   ├── bootstrap.py                 # DI container and wiring
│   ├── cli/                         # Command-line interface
│   ├── application/                 # Use cases, DTOs, interfaces
│   ├── domain/                      # Entities, value objects, events
│   ├── infrastructure/              # Collectors, detectors, cache
│   ├── exporters
│   └── core/                        # Config, DI, exceptions, utilities
└── tests/                           # Unit, integration, and e2e tests
```

See [docs/architecture.md](docs/architecture.md) for a full breakdown of each layer.

---

## Troubleshooting

### "Missing GitHub API token"

The standalone [`main.py`](main.py) requires a `GITHUB_API_TOKEN` environment variable. Set it before running:

```bash
export GITHUB_API_TOKEN="ghp_..."
```

The CLI itself does not require a token, but without one you may hit GitHub rate limits.

### No repositories found

- Increase `--min-stars` or check the `--language` value (must be `python`, `javascript`, or `typescript`).
- If you are rate-limited, provide a `--token`.

### No refactorings found

- Detection depends on the detector and the commit history. Try `--detector-type ast` or increase `--commits-limit`.
- Some repositories are excluded automatically (e.g., catalog/list repositories) — see [`is_excluded_repo`](src/refdatagen/infrastructure/detectors/preprocessing.py).

### Redis cache errors

The Redis cache requires a running Redis instance. Set `REDIS_URL` correctly, or use `--cache-type memory` / `disk`.

### `UnsupportedFormatError`

You passed an unsupported `--output-format`. Valid values are `json`, `parquet`, and `huggingface`.

---

## FAQ

**What languages are supported?**
Currently Python, JavaScript, and TypeScript. Java, Go, Rust, and others are planned (see [`ProgrammingLanguage`](src/refdatagen/domain/value_objects/language.py)).

**What refactoring types are detected?**
The [`Refactoring`](src/refdatagen/domain/entities/refactoring.py) entity defines a set of valid types, including `extract_method`, `rename_class`, `move_method`, `extract_class`, `inline_method`, `rename_method`, `move_field`, `extract_interface`, and `extract_superclass`.

**Do I need a GitHub token?**
Not strictly, but it is strongly recommended to avoid API rate limits.

**Can I use this as a library?**
Yes. The public API in [`refdatagen.api`](src/refdatagen/api.py) is designed for programmatic use.

**How do I add a new detector or collector?**
Implement the corresponding interface in [`application/interfaces/`](src/refdatagen/application/interfaces/) and register it in the DI container ([`bootstrap.py`](src/refdatagen/bootstrap.py)). See [docs/architecture.md](docs/architecture.md).

---

## Documentation

- [Architecture & design](docs/architecture.md)
- [Building, testing & deployment](docs/building-testing-deployment.md)
- [Tutorials & examples](docs/tutorials.md)
- [References](docs/references.md)

---

## Contributing

Contributions are welcome! Please read [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines on coding standards, branch naming, commit messages, and the pull request process. All contributors are expected to follow the [Code of Conduct](CODE_OF_CONDUCT.md).

---

## License

This project is licensed under the [MIT License](LICENSE).
