Metadata-Version: 2.5
Name: opensynthlab
Version: 0.1.0a1
Summary: Configurable synthetic data workflows for LLMs: template-based generation, validation, and JSONL export.
Project-URL: Homepage, https://github.com/Excelius-Wang/OpenSynthLab
Project-URL: Repository, https://github.com/Excelius-Wang/OpenSynthLab
Project-URL: Issues, https://github.com/Excelius-Wang/OpenSynthLab/issues
Project-URL: Changelog, https://github.com/Excelius-Wang/OpenSynthLab/blob/main/CHANGELOG.md
Author: OpenSynthLab contributors
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: data-generation,datasets,llm,synthetic-data
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Description-Content-Type: text/markdown

# OpenSynthLab

**An end-to-end synthetic data framework for large language models.**

Built-in recipes. Customizable pipelines. Quality validation.

[简体中文](README.zh-CN.md) · [Roadmap](ROADMAP.md) · [Contributing](CONTRIBUTING.md)

> **Status: early alpha (`0.1.0a1`).** The current implementation supports offline template-based synthesis: seed expansion, validation, exact deduplication, and JSONL export. Model-driven synthesis and agent trajectories are planned, not implemented. APIs may change.

## Available now

- Expand seed records over independent variable combinations, preserving question/answer pairing.
- Render configurable output fields with `${variable}` templates.
- Validate required fields, limit candidate counts, and remove exact duplicate records.
- Export UTF-8 JSONL atomically without overwriting existing files.
- Use the Python API or CLI with Python 3.10+ and no runtime dependencies.

This version makes no model or network calls. Template expansion increases coverage but does not add new knowledge or establish semantic correctness.

## What we are building

OpenSynthLab aims to make high-quality training data easier to build, from source material or task definitions through generation, validation, filtering, and export.

Users should be able to start with a built-in recipe, configure models and data, and run a complete workflow. Developers should be able to replace individual steps or contribute new methods without rebuilding the surrounding infrastructure.

```text
Source data / task definitions
          ↓
Built-in recipe / custom pipeline
          ↓
Generation / environment execution
          ↓
Validation → scoring → deduplication → filtering
          ↓
Training-ready datasets + provenance + run reports
```

## Intended scope

| Data type | Intended workflows |
| --- | --- |
| Instruction and dialogue | Grounded question-answer generation, instruction expansion, multi-turn conversations |
| Reasoning | Problem construction, solution generation, answer verification |
| Preference | Candidate responses, comparisons, quality judgments |
| Agent trajectories | Task generation, tool and environment interaction, outcome verification |
| Multimodal | Image-text and other multimodal synthesis workflows |

## Design principles

- **Complete workflows:** keep generation, quality checks, and export in one reproducible process.
- **Configurable defaults:** offer useful recipes while allowing users to change models, steps, and quality criteria.
- **Evidence for quality:** preserve validation results and source provenance; generation alone does not establish correctness.
- **Executable agent traces:** distinguish simulated traces from trajectories collected through actual tool or environment execution.
- **Reusable methods:** document method references, assumptions, inputs, outputs, and limitations.

## Getting started

Install from the repository:

```bash
git clone https://github.com/Excelius-Wang/OpenSynthLab.git
cd OpenSynthLab
python -m pip install .
opensynthlab run examples/faq.json --output faq.jsonl
```

The example generates four instruction/answer records from two seed pairs and two phrasing variations. No API key or GPU is needed. The output path must be new and its parent directory must exist.

Python API:

```python
from opensynthlab import generate, write_jsonl

records = generate({
    "seeds": [{"question": "What does SFT stand for?", "answer": "Supervised fine-tuning."}],
    "variables": {"prefix": ["Answer:", "Please answer:"]},
    "templates": {"instruction": "${prefix} ${question}", "output": "${answer}"},
    "required_fields": ["instruction", "output"],
    "max_records": 100,
})
write_jsonl(records, "training.jsonl")
```

Seeds and variable choices must contain strings. Seed rows must have matching fields; seed and variable names must not overlap. Use `$$` for a literal dollar sign. Required fields default to all template fields. The default limit is 10,000 candidates **before** deduplication; oversized recipes fail rather than being silently truncated. This initial implementation holds generated records in memory.

The package is named `opensynthlab`. PyPI publication status is recorded in [GitHub Releases](https://github.com/Excelius-Wang/OpenSynthLab/releases); repository availability alone does not imply a PyPI release.

Follow the [roadmap](ROADMAP.md) or [open an issue](https://github.com/Excelius-Wang/OpenSynthLab/issues) with a concrete use case.

## Contributing

We welcome discussions about real data-generation needs, reproducible methods, and quality evaluation. See [CONTRIBUTING.md](CONTRIBUTING.md) before proposing changes.

## License

[Apache License 2.0](LICENSE).
