Metadata-Version: 2.1
Name: Torrelque
Version: 0.6.0
Summary: Asynchronous Redis-backed reliable queue package
Home-page: https://heptapod.host/saajns/torrelque
Author: saaj
Author-email: mail@saaj.me
License: LGPL-3.0-only
Project-URL: Source Code, https://heptapod.host/saajns/torrelque
Project-URL: Documentation, https://torrelque.readthedocs.io/
Project-URL: Release Notes, https://torrelque.readthedocs.io/en/latest/history.html
Keywords: python redis asynchronous job-queue work-queue
Platform: Any
Classifier: Topic :: Software Development :: Libraries
Classifier: Framework :: AsyncIO
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: Implementation :: CPython
Classifier: Programming Language :: Python :: Implementation :: PyPy
Classifier: Intended Audience :: Developers
Provides-Extra: test
Provides-Extra: manual

.. image:: https://img.shields.io/pypi/l/Torrelque.svg
   :target: https://spdx.org/licenses/LGPL-3.0-only.html
   :alt: PyPI - License
.. image:: https://heptapod.host/saajns/torrelque/badges/branch/default/pipeline.svg
   :target: https://heptapod.host/saajns/torrelque/-/commits/branch/default
   :alt: Pipeline status
.. image:: https://heptapod.host/saajns/torrelque/badges/branch/default/coverage.svg
   :target: https://heptapod.host/saajns/torrelque/-/commits/branch/default
   :alt: Test code coverage
.. image:: https://badge.fury.io/py/Torrelque.svg
   :target: https://pypi.python.org/pypi/Torrelque
   :alt: PyPI
.. image:: https://readthedocs.org/projects/torrelque/badge/?version=latest
   :target: https://torrelque.readthedocs.io/en/latest/?badge=latest
   :alt: RTFD

*********
Torrelque
*********
Torrelque is a Python package that provides minimal asynchronous reliable
distributed Redis-backed work queues. It is built:

1. Lock-free. It relies on Redis transactions and its single-threaded
   execution model.
2. Poll-free. Waiting subset of the Python API relies either on blocking Redis
   commands or notifications.
3. Bulk-friendly. Tasks can be produced and consumed in bulk.
4. Introspectable. Task stats, task status transition watching API, and
   the data model comprehensible directly in Redis.

.. note::

   Prior to version 0.2 Torrelque was Tornado-specific. Since version 5 Tornado
   runs on ``asyncio`` event loop by default, hence newer Torrelque is still
   compatible with (newer) Tornado, even though it's based on ``asyncio`` now.

Install
=======
::

   pip install Torrelque

Quickstart
==========
Producer:

.. sourcecode:: python

   redis = yaaredis.StrictRedis()
   queue = torrelque.Torrelque(redis, queue='email')
   queue.schedule_sweep()

   task_data = {'addr': 'joe@doe.com', 'subj': 'hello', 'body': '...'}
   task_id = await queue.enqueue(task)
   logger.info('Email task enqueued %s', task_id)

Consumer:

.. sourcecode:: python

   redis = yaaredis.StrictRedis()
   queue = torrelque.Torrelque(redis, queue='email')

   while True:
       task_id, task_data = await queue.dequeue()
       try:
           await some_email_client.send(**task_data)
       except SomeEmailError:
           logger.exception('Sending error, retrying in 30 seconds')
           await queue.requeue(task_id, delay=30)
       else:
           await queue.release(task_id)


Example list
============
- `Producer-consumer <e1_>`_. Infinite producing and consuming loops.
- `Batch processing <e2_>`_. Finite number of tasks, consumers stop with a
  poison pill, bulk enqueue. This example can be used as a synthetic benchmark.
  Because there's no IO-bound workload, it'll be CPU-bound which isn't normal
  mode of operation for an asynchronous application. But it can be used to
  compare between CPython, PyPy and concurrency parameters.
- `Web application background task <e3_>`_. This tornado application allows
  to start a task and push server-sent events (SSE) to UI about its status. UI
  starts a task and waits for it to complete. When a task fails it's requeued
  with exponential back-off.


.. _e1: https://heptapod.host/saajns/torrelque/blob/branch/default/example/producer_consumer.py
.. _e2: https://heptapod.host/saajns/torrelque/blob/branch/default/example/batch_processing.py
.. _e3: https://heptapod.host/saajns/torrelque/blob/branch/default/example/wait_until_complete.py


