Metadata-Version: 2.4
Name: aiosafeconsumer
Version: 0.0.7
Summary: Safely consume and process data.
Author-email: Kostiantyn Korikov <lostclus@gmail.com>
Project-URL: Repository, http://github.com/lostclus/aiosafeconsumer
Keywords: asyncio,consumer,microservices
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: No Input/Output (Daemon)
Classifier: Framework :: AsyncIO
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: BSD License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Typing :: Typed
Requires-Python: >=3.10
Description-Content-Type: text/x-rst
License-File: LICENSE
Requires-Dist: croniter
Provides-Extra: kafka
Requires-Dist: aiokafka; extra == "kafka"
Provides-Extra: redis
Requires-Dist: redis[hiredis]; extra == "redis"
Provides-Extra: elasticsearch
Requires-Dist: elasticsearch[async]; extra == "elasticsearch"
Provides-Extra: mongo
Requires-Dist: pymongo; extra == "mongo"
Provides-Extra: postgres
Requires-Dist: asyncpg; extra == "postgres"
Provides-Extra: tests
Requires-Dist: aiokafka; extra == "tests"
Requires-Dist: elasticsearch[async]; extra == "tests"
Requires-Dist: pytest; extra == "tests"
Requires-Dist: pytest-asyncio; extra == "tests"
Requires-Dist: pytest-cov; extra == "tests"
Requires-Dist: redis[hiredis]; extra == "tests"
Requires-Dist: pymongo; extra == "tests"
Requires-Dist: asyncpg; extra == "tests"
Requires-Dist: asyncpg-stubs; extra == "tests"
Requires-Dist: camel-converter; extra == "tests"
Dynamic: license-file

aiosafeconsumer
===============

.. image:: https://github.com/lostclus/aiosafeconsumer/actions/workflows/tests.yml/badge.svg
    :target: https://github.com/lostclus/aiosafeconsumer/actions

.. image:: https://img.shields.io/pypi/v/aiosafeconsumer.svg
    :target: https://pypi.org/project/aiosafeconsumer/
    :alt: Current version on PyPi

.. image:: https://img.shields.io/pypi/pyversions/aiosafeconsumer
    :alt: PyPI - Python Version

aiosafeconsumer is a library that provides abstractions and some implementations
to consume data somewhere and process it.

Features:

* Based on AsyncIO
* Type annotated
* Use logging with contextual information

Abstractions:

* `DataSource` - waits for data and returns batch of records using Python generator
* `DataProcessor` - accepts batch of records and precess it
* `DataTransformer` - accepts batch of records and transform it and calls
  another processor to precess it. Extends `DataProcessor`
* `Worker` - abstract worker. Do a long running task
* `ConsumerWorker` - connects `DataSource` and `DataProcessor`. Extends Worker
* `DataWriter` - base abstraction to perform data synchronization. Extends DataProcessor

Current implementations:

* `KafkaSource` - read data from Kafka
* `RedisWriter` - synchronize data in Redis
* `ElasticsearchWriter` - synchronize data in Elasticsearch
* `WorkerPool` - controller to setup and run workers in parallel. Can handle worker failures and restarts workers when it fails or exits.

Install::

    pip install aiosafeconsumer

Install with Kafka::

    pip install aiosafeconsumer[kafka]

Install with Redis::

    pip install aiosafeconsumer[redis]

Install with Elasticsearch::

    pip install aiosafeconsumer[elasticsearch]

Install with MongoDB::

    pip install aiosafeconsumer[mongo]

Install with PostgreSQL::

    pip install aiosafeconsumer[postgres]

Links:

* Producer library: https://github.com/lostclus/django-kafka-streamer
* Example application: https://github.com/lostclus/WeatherApp

TODO:

* Redis Streams source
* Enumerate IDs message type support in PostgreSQL writer
* ClickHouse writer