Metadata-Version: 2.3
Name: polymo
Version: 0.10.0
Summary: Declarative REST API ingestion for PySpark
Keywords: spark,pyspark,rest,api,ingestion,data-engineering,etl,http
Author: Daniel Tom
Author-email: Daniel Tom <d.e.tom89@gmail.com>
License: BSD-3
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Information Technology
Classifier: Topic :: Internet :: WWW/HTTP
Classifier: Topic :: Software Development :: Libraries
Classifier: Topic :: Database
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Operating System :: OS Independent
Classifier: Typing :: Typed
Requires-Dist: fsspec>=2025.9.0
Requires-Dist: httpx>=0.26
Requires-Dist: jinja2>=3.1.6
Requires-Dist: pyyaml>=6.0.1
Requires-Dist: truststore>=0.10
Requires-Dist: pyspark>=4 ; extra == 'benchmark'
Requires-Dist: pyarrow>=13 ; extra == 'benchmark'
Requires-Dist: httpx>=0.26 ; extra == 'benchmark'
Requires-Dist: notebook>=7.4.7 ; extra == 'benchmark'
Requires-Dist: fastapi>=0.110 ; extra == 'benchmark'
Requires-Dist: uvicorn>=0.24 ; extra == 'benchmark'
Requires-Dist: fastapi>=0.110 ; extra == 'builder'
Requires-Dist: uvicorn>=0.24 ; extra == 'builder'
Requires-Dist: jinja2>=3.1.6 ; extra == 'builder'
Requires-Dist: pyspark>=4 ; extra == 'builder'
Requires-Dist: pyarrow>=13 ; extra == 'builder'
Requires-Dist: pyspark>=4 ; extra == 'smoke'
Requires-Dist: pyarrow>=13 ; extra == 'smoke'
Requires-Python: >=3.10
Project-URL: Documentation, https://dan1elt0m.github.io/polymo/
Project-URL: Homepage, https://github.com/dan1elt0m/polymo
Project-URL: Issues, https://github.com/dan1elt0m/polymo/issues
Project-URL: Repository, https://github.com/dan1elt0m/polymo.git
Provides-Extra: benchmark
Provides-Extra: builder
Provides-Extra: smoke
Description-Content-Type: text/markdown

<p align="center">
  <img src="builder-ui/public/logo.png" alt="Polymo" width="220">
</p>

<p align="center">
    <em>Single pyspark API connector for all REST API's. 
</em>
</p>

<p align="center">
  <a href="https://github.com/dan1elt0m/polymo/actions/workflows/test.yml"><img alt="test" src="https://github.com/dan1elt0m/polymo/actions/workflows/test.yml/badge.svg"></a>
  <a href="https://github.com/dan1elt0m/polymo/actions/workflows/gh-pages.yml"><img alt="docs" src="https://github.com/dan1elt0m/polymo/actions/workflows/gh-pages.yml/badge.svg"></a>
  <img alt="PyPI - Python Version" src="https://img.shields.io/pypi/pyversions/polymo">
</p>

# Welcome to Polymo

Declarative API ingestion with Pyspark. Uses the new Pyspark 4 custom data sources under the hood.

<!-- Centered clickable screenshot -->
<p align="center">
  <a href="docs/ui.png">
    <img src="docs/ui.png" alt="Polymo Builder UI - connector preview screen" width="860">
  </a>
</p>

## How does it work?

Define a config file manually or use the recommended, lightweight builder UI. 
Once you are happy with your config, all you need to do is register the Polymo reader and tell Spark where to find the config:

```python
from pyspark.sql import SparkSession
from polymo import ApiReader

spark = SparkSession.builder.getOrCreate()
spark.dataSource.register(ApiReader)

df = (
    spark.read.format("polymo")
    .option("config_path", "./config.yml")  # YAML you saved from the Builder
    .option("token", "YOUR_TOKEN")  # Only if the API needs one
    .load()
)

df.show()
```
Streaming works too:
```python
spark.readStream.format("polymo")
```

Prefer everything in Python? Use the PolymoConfig model.
```python
from pyspark.sql import SparkSession
from polymo import ApiReader, PolymoConfig

spark = SparkSession.builder.getOrCreate()
spark.dataSource.register(ApiReader)

jp_posts = PolymoConfig(
    base_url="https://jsonplaceholder.typicode.com",
    path="/posts",
)

df = (
    spark.read.format("polymo")
    .option("config_json", jp_posts.config_json())
    .load()
)
df.show()
```
Polymo reads in batches and can read pages in parallel. Therefore Polymo can be much faster than row based solutions like UDFs.

## How to start?
Locally you probably want to install polymo along with the Builder UI: 

```bash
pip install "polymo[builder]"
```

This comes with all UI deps such as pyspark

Running Polymo on a spark cluster usually doesn't require these UI deps.
In that case, just install the bare minimum deps with
```bash
pip install polymo
```

## Launch the builder UI 

```bash 
polymo builder
```

#### (Optional) Run the Builder in Docker

```bash
docker compose up --build builder
```

- The service listens on port `8000`; open <http://localhost:8000> once Uvicorn reports it is running.

## Where to Next
Read the docs [here](https://dan1elt0m.github.io/polymo/)

Other material:
- Step by step example: [medium blogpost](https://medium.com/@d.e.tom89/turn-any-rest-api-into-spark-dataframes-in-minutes-with-polymo-028a48113eb1)


## Contributing
It's still early days, but Polymo already supports a lot of features!
Is there something missing? Raise an issue or contribute!

Contributions and early feedback welcome!

---
If Polymo helped, a ⭐ makes my day
