Metadata-Version: 2.4
Name: plurai-extract
Version: 1.0.2
Summary: Extract documents and records from data sources for Plurai simulation
Author: Plurai
License: Proprietary
Project-URL: Homepage, https://github.com/plurai-ai/plurai-extract
Project-URL: Repository, https://github.com/plurai-ai/plurai-extract
Project-URL: Issues, https://github.com/plurai-ai/plurai-extract/issues
Project-URL: Changelog, https://github.com/plurai-ai/plurai-extract/blob/main/CHANGELOG.md
Keywords: plurai,extraction,rag,sql,corpus,simulation
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: License :: Other/Proprietary License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Database
Classifier: Topic :: Text Processing
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pydantic>=2.0
Requires-Dist: tiktoken
Provides-Extra: pdf
Requires-Dist: docling>=2.69.1; extra == "pdf"
Requires-Dist: pypdfium2>=4.30.0; extra == "pdf"
Provides-Extra: http
Requires-Dist: aiohttp>=3.8; extra == "http"
Provides-Extra: vertex
Requires-Dist: google-cloud-discoveryengine>=0.11; extra == "vertex"
Provides-Extra: bigquery
Requires-Dist: google-cloud-bigquery>=3.0; extra == "bigquery"
Provides-Extra: redshift
Requires-Dist: boto3>=1.26; extra == "redshift"
Provides-Extra: azure-sql
Requires-Dist: pyodbc>=4.0; extra == "azure-sql"
Provides-Extra: databricks
Requires-Dist: databricks-sql-connector>=2.0; extra == "databricks"
Provides-Extra: snowflake
Requires-Dist: snowflake-connector-python>=3.0; extra == "snowflake"
Provides-Extra: bedrock
Requires-Dist: langchain-aws>=0.2.0; extra == "bedrock"
Provides-Extra: all
Requires-Dist: plurai-extract[azure-sql,bedrock,bigquery,databricks,http,pdf,redshift,snowflake,vertex]; extra == "all"
Provides-Extra: dev
Requires-Dist: pytest; extra == "dev"
Requires-Dist: pytest-asyncio; extra == "dev"
Requires-Dist: pypdfium2>=4.30.0; extra == "dev"
Requires-Dist: pyyaml; extra == "dev"
Dynamic: license-file

# plurai-extract

Extract documents and records from data sources for Plurai simulation. The package reads
documents, schemas and records from a data source with the caller's own credentials, on the
caller's machine, and emits a deterministic set of chunks plus structural relationships. It
involves no LLM and no embeddings; the output is plain dataclasses that serialize with
`to_dict()`.

## Install

```bash
pip install plurai-extract
```

The core install depends only on `pydantic` and `tiktoken`. Each data-source driver is an
opt-in extra, so you install only what your source needs:

```bash
pip install "plurai-extract[bigquery]"
pip install "plurai-extract[pdf,snowflake]"
```

| Extra | Source | Driver |
|---|---|---|
| `pdf` | PDF directories | docling, pypdfium2 |
| `http` | Azure AI Search | aiohttp |
| `vertex` | Vertex AI Search | google-cloud-discoveryengine |
| `bigquery` | BigQuery | google-cloud-bigquery |
| `redshift` | Redshift | boto3 |
| `azure-sql` | Azure SQL | pyodbc |
| `databricks` | Databricks | databricks-sql-connector |
| `snowflake` | Snowflake | snowflake-connector-python |
| `bedrock` | Bedrock Knowledge Base | langchain-aws |
| `all` | every extra above | |

SQLite, JSON and JSON Lines records, Markdown and text directories, and the LangChain and
LlamaIndex vector-store sources need no extra.

Python 3.11, 3.12 and 3.13 are supported.

## Usage

```python
from plurai_extract import load_only
from plurai_extract.sources.directory import DirectorySource

documents, relationships = load_only(DirectorySource(path="./docs"))
payload = [d.to_dict() for d in documents]
```

`load_only` takes a source instance, runs the extraction, and returns
`(documents, relationships)`. Source classes are importable from `plurai_extract.sources`
(or from their own module, as above).

## Issues and release notes

See [CONTRIBUTING.md](https://github.com/plurai-ai/plurai-extract/blob/main/CONTRIBUTING.md) for how to report issues, and
[CHANGELOG.md](https://github.com/plurai-ai/plurai-extract/blob/main/CHANGELOG.md) for release notes.
