Metadata-Version: 2.4
Name: sci-etl-core
Version: 0.4.0
Summary: Reusable, domain-agnostic ETL core for scientific text mining.
License-Expression: MIT
Project-URL: Homepage, https://github.com/xueromll/sci-etl-core
Project-URL: Documentation, https://xueromll.github.io/sci-etl-core/
Project-URL: Repository, https://github.com/xueromll/sci-etl-core
Project-URL: Issues, https://github.com/xueromll/sci-etl-core/issues
Keywords: etl,pipeline,text-mining,scientific-literature,data-extraction
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: Text Processing
Classifier: Typing :: Typed
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pydantic>=2
Requires-Dist: pyyaml
Requires-Dist: python-dotenv
Requires-Dist: beautifulsoup4
Requires-Dist: lxml
Requires-Dist: pandas
Provides-Extra: async
Requires-Dist: httpx; extra == "async"
Requires-Dist: aiofiles; extra == "async"
Requires-Dist: aiolimiter; extra == "async"
Provides-Extra: llm
Requires-Dist: openai; extra == "llm"
Requires-Dist: tiktoken; extra == "llm"
Provides-Extra: pdf
Requires-Dist: pdfplumber; extra == "pdf"
Provides-Extra: sql
Requires-Dist: sqlalchemy>=2; extra == "sql"
Requires-Dist: aiosqlite; extra == "sql"
Provides-Extra: viz
Requires-Dist: plotly; extra == "viz"
Requires-Dist: aiofiles; extra == "viz"
Provides-Extra: cluster
Requires-Dist: scikit-learn; extra == "cluster"
Requires-Dist: numpy; extra == "cluster"
Provides-Extra: embeddings
Requires-Dist: numpy; extra == "embeddings"
Requires-Dist: openai; extra == "embeddings"
Provides-Extra: embeddings-local
Requires-Dist: numpy; extra == "embeddings-local"
Requires-Dist: sentence-transformers; extra == "embeddings-local"
Provides-Extra: search
Provides-Extra: dev
Requires-Dist: pytest; extra == "dev"
Requires-Dist: pytest-asyncio; extra == "dev"
Requires-Dist: pytest-mock; extra == "dev"
Requires-Dist: pytest-cov; extra == "dev"
Requires-Dist: hypothesis; extra == "dev"
Provides-Extra: lint
Requires-Dist: ruff; extra == "lint"
Requires-Dist: mypy; extra == "lint"
Requires-Dist: types-PyYAML; extra == "lint"
Requires-Dist: types-requests; extra == "lint"
Requires-Dist: types-aiofiles; extra == "lint"
Provides-Extra: docs
Requires-Dist: mkdocs<2,>=1.6; extra == "docs"
Requires-Dist: mkdocs-material<10,>=9.7; extra == "docs"
Requires-Dist: mkdocstrings[python]<2,>=1.0; extra == "docs"
Requires-Dist: mkdocs-click<1,>=0.9; extra == "docs"
Requires-Dist: mike<3,>=2.1; extra == "docs"
Requires-Dist: ruff; extra == "docs"
Provides-Extra: full
Requires-Dist: httpx; extra == "full"
Requires-Dist: aiofiles; extra == "full"
Requires-Dist: aiolimiter; extra == "full"
Requires-Dist: openai; extra == "full"
Requires-Dist: tiktoken; extra == "full"
Requires-Dist: pdfplumber; extra == "full"
Requires-Dist: sqlalchemy>=2; extra == "full"
Requires-Dist: aiosqlite; extra == "full"
Requires-Dist: plotly; extra == "full"
Requires-Dist: scikit-learn; extra == "full"
Requires-Dist: numpy; extra == "full"
Requires-Dist: requests; extra == "full"
Dynamic: license-file

# sci-etl-core

[![CI](https://github.com/xueromll/sci-etl-core/actions/workflows/ci.yml/badge.svg)](https://github.com/xueromll/sci-etl-core/actions/workflows/ci.yml)
[![Docs](https://github.com/xueromll/sci-etl-core/actions/workflows/docs.yml/badge.svg)](https://xueromll.github.io/sci-etl-core/)
[![PyPI](https://img.shields.io/pypi/v/sci-etl-core)](https://pypi.org/project/sci-etl-core/)

A reusable, **domain-agnostic** Python library for scientific text mining and
ETL. `sci-etl-core` gives you composable building blocks — extractors, parsers,
LLM clients, embedding memory, processors, exporters, and state managers —
behind abstract base classes, so you can assemble a pipeline for *any* corpus
without inheriting constants tied to a specific field of science.

The library is **async-first**. Every component is an `async` implementation,
orchestrated by `AsyncETLPipeline`. For scripts that don't want to manage an
event loop, `ETLPipeline` is a single blocking entrypoint that runs the same
pipeline on a background loop.

**Documentation: https://xueromll.github.io/sci-etl-core/**

## Features

- **Pluggable async interfaces** for every stage, with `Sync*Adapter` wrappers
  for existing blocking implementations.
- **Built-in orchestration** with bounded concurrency, resumable crash-safe
  state, graceful shutdown, polite retries that honor `Retry-After`, shared
  and per-host rate limits, progress events and run metrics, and explicit
  failure signaling through `PipelineAborted`.
- **LLM response caching** in memory or SQLite, so a rerun doesn't pay for the
  same prompt twice.
- **Semantic memory and local search** — embed full texts into a vector store,
  query a SQLite FTS5 index with Boolean syntax, fuse BM25 with embedding
  similarity, filter by metadata facets, and grow graphs of related papers.
- **Concrete implementations included** — arXiv, PubMed, Semantic Scholar,
  and OpenAlex extractors; OpenAI-compatible chat and embedding clients; PDF,
  LaTeX, HTML, DOCX, and JATS XML parsers; CSV, SQL, and Plotly exporters;
  dataframe processors and record validators.
- **Typed configuration** from YAML and `.env`, an offline test suite at 100%
  coverage, and PEP 561 type information.

## Installation

Python 3.10 or newer is required.

```bash
pip install "sci-etl-core[async,llm,pdf]"   # everything the example below uses
pip install "sci-etl-core[full]"            # every bundled component except local embeddings
```

Components load their optional dependencies only when you import them. The
[installation guide](https://xueromll.github.io/sci-etl-core/latest/getting-started/installation/)
lists what each extra adds.

Prefer configuration to code? [sci-etl-cli](https://github.com/xueromll/sci-etl-cli)
runs these pipelines from a single YAML file.

## Example

```python
import asyncio
import os

from sci_etl_core import (
    AsyncArxivExtractor,
    AsyncCsvUpsertExporter,
    AsyncETLPipeline,
    AsyncFileStateManager,
    AsyncLLMEntityExtractor,
    AsyncLLMRelevanceFilter,
    AsyncOpenAICompatibleClient,
)
from sci_etl_core.http_async import build_async_client
from sci_etl_core.parsers import LatexTarballParser, PdfPlumberParser
from sci_etl_core.processors import DefaultKeyNormalizer

RELEVANCE_PROMPT = 'Does the paper report measurements of galaxies? Reply with JSON: {"relevant": true} or {"relevant": false}.'
EXTRACTION_PROMPT = 'Extract every measured object. Reply with JSON: {"items": [{"name": "...", "value_a": 0.0}]}.'


async def main() -> None:
    client = build_async_client()
    llm = AsyncOpenAICompatibleClient(api_key=os.environ["LLM_API_KEY"], base_url="https://api.openai.com/v1", model="gpt-4o-mini")
    pipeline = AsyncETLPipeline(
        extractor=AsyncArxivExtractor(client=client, pdf_parser=PdfPlumberParser(), latex_parser=LatexTarballParser()),
        relevance_filter=AsyncLLMRelevanceFilter(llm_client=llm, system_prompt=RELEVANCE_PROMPT),
        entity_extractor=AsyncLLMEntityExtractor(llm_client=llm, system_prompt=EXTRACTION_PROMPT),
        exporter=AsyncCsvUpsertExporter(key_column="name", value_columns=["value_a"], normalizer=DefaultKeyNormalizer()),
        state_manager=AsyncFileStateManager("state/processed.txt", "state/metadata.json"),
        destination="results.csv",
        closeables=[client, llm],
    )
    async with pipeline:
        processed = await pipeline.run(query="all:galaxy", total_limit=50)
    print(f"Processed {processed} relevant records")


asyncio.run(main())
```

The [quick start](https://xueromll.github.io/sci-etl-core/latest/getting-started/quick-start/)
explains what a run does, how it resumes, and what the prompts must ask for.

## Documentation

| Topic | Where |
|-------|-------|
| Installation, quick start, blocking usage, configuration | [Getting started](https://xueromll.github.io/sci-etl-core/latest/getting-started/installation/) |
| Sources, post-processing, semantic memory, state, shutdown, retries, rate limiting, events, caching | [Guide](https://xueromll.github.io/sci-etl-core/latest/guide/sources/) |
| Boolean and hybrid search, facets, discovery graphs | [Local search and discovery](https://xueromll.github.io/sci-etl-core/latest/guide/search/) |
| Components and how they connect | [Architecture](https://xueromll.github.io/sci-etl-core/latest/guide/architecture/) |
| Every public class and function | [API reference](https://xueromll.github.io/sci-etl-core/latest/reference/) |
| The `sci-etl` command-line tool | [CLI](https://xueromll.github.io/sci-etl-core/latest/cli/) |

## Testing

```bash
pip install -e ".[full,dev,lint]"
pytest --cov=sci_etl_core --cov-report=term-missing
ruff check .
mypy
```

The suite runs offline, and `pytest --cov` fails if line coverage drops below
100%.

## Contributing

Contributions are welcome — new extractors, parsers, exporters, and embedding
backends especially. See [CONTRIBUTING.md](CONTRIBUTING.md) to get set up, and
browse [good first issues](.github/ISSUE_TEMPLATE/good_first_issue.md) if
you're new. Moving an existing pipeline onto the library? See [MIGRATION.md](MIGRATION.md).
What's planned is in [ROADMAP.md](ROADMAP.md), and releases are recorded in
[CHANGELOG.md](CHANGELOG.md). All participation is governed by our
[Code of Conduct](CODE_OF_CONDUCT.md).

## Security

Please report vulnerabilities privately — see [SECURITY.md](SECURITY.md).

## License

Released under the MIT License. See [LICENSE](LICENSE) for details.
