Metadata-Version: 2.0
Name: cadasta-workertoolbox
Version: 0.2.1
Summary: Cadasta Worker Toolbox
Home-page: https://github.com/Cadasta/cadasta-workertoolbox
Author: Anthony Lukach
Author-email: alukach@cadasta.org
License: GNU Affero General Public License v3.0
Platform: UNKNOWN
Classifier: Development Status :: 3 - Alpha
Requires-Dist: SQLAlchemy (>=1.1.11,<=1.2)
Requires-Dist: boto3 (<=1.5,>=1.4.4)
Requires-Dist: celery (<=4.2,>=4.1.0)
Requires-Dist: kombu (<=4.2,>=4.1.0)
Requires-Dist: mock (>=2.0.0)
Requires-Dist: opbeat (>=3.5.2)
Requires-Dist: psycopg2 (>=2.7.1)
Requires-Dist: pycurl (<=7.44,>=7.43.0)

Cadasta Worker Toolbox
======================

|PyPI version| |Build Status| |Requirements Status|

A collection of helpers to assist in quickly building asynchronous
workers for the Cadasta system.

Library
-------

``cadasta.workertoolbox.conf.Config``
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The ``Config`` class was built to simplify configuring Celery settings,
helping to ensure that all workers adhere to the architecture
requirements of the Cadasta asynchronous system. It essentially offers a
diff between Celery's default configuration and the configuration
required by our system. It is the aim of the class to not require much
customization on the part of the developer, however some customization
may be needed when altering configuration between environments (e.g. if
dev settings vary greatly from prod settings).

Any `Celery
setting <http://docs.celeryproject.org/en/v4.0.2/userguide/configuration.html#new-lowercase-settings>`__
may be submitted. It is internal convention that we use the Celery's
newer lowercase settings rather than their older uppercase counterparts.
This will ensure that they are displayed when calling ``repr`` on the
``Conf`` instance.

Once applied, all settings (and internal variables) are available on the
Celery ``app`` instance's ``app.conf`` object.

Provided Configuration
^^^^^^^^^^^^^^^^^^^^^^

Below is the configuration that the ``Config`` class will provide to a
``Celery`` instance.

``result_backend``
''''''''''''''''''

Defaults to
``'db+postgresql://{0.RESULT_DB_USER}:{0.RESULT_DB_PASS}@{0.RESULT_DB_HOST}/{0.RESULT_DB_NAME}'``
rendered with ``self``.

``broker_transport``
''''''''''''''''''''

Defaults to ``'sqs``'.

``broker_transport_options``
''''''''''''''''''''''''''''

Defaults to:

.. code:: python

    {
        'region': 'us-west-2',
        'queue_name_prefix': '{}-'.format(QUEUE_NAME_PREFIX)
    }

``task_queues``
'''''''''''''''

Defaults to the following ``set`` of ``kombu.Queue`` objects, where
``queues`` is the configuration's internal ``QUEUES`` variable and
``exchange`` is a ``kombu.Exchange`` object constructed from the
``task_default_exchange`` and ``task_default_exchange_type`` settings:

.. code:: python

    set([
        Queue('celery', exchange, routing_key='celery'),
        Queue(platform_queue, exchange, routing_key='#'),
    ] + [
        Queue(q_name, exchange, routing_key=q_name)
        for q_name in queues
    ])

*Note: It is recommended that developers not alter this setting.*

``task_routes``
'''''''''''''''

Defaults to a function that will generate a dict with the
``routing_key`` matching the value at the first index of a task name
split on the ``.`` and the ``exchange`` set to a ``kombu.Exchange``
object constructed from the ``task_default_exchange`` and
``task_default_exchange_type`` settings

*Note: It is recommended that developers not alter this setting.*

``task_default_exchange``
'''''''''''''''''''''''''

Defaults to ``'task_exchange'``

``task_default_exchange_type``
''''''''''''''''''''''''''''''

Defaults to ``'topic'``

``task_track_started``
''''''''''''''''''''''

Defaults to ``True``.

Internal Variables
^^^^^^^^^^^^^^^^^^

Below are arguments and environmental variables that can be used to
customize the above provided configuration. By convention, all variables
used to construct Celery configuration should should be written entirely
uppercase.

``QUEUES`` *(provided via argument)*
''''''''''''''''''''''''''''''''''''

This should contain an array of names for all service-related queues
used by the Cadasta Platform. These values are used to construct the
``task_queues`` configuration. For the purposes of routing followup
tasks, it's important that every task consumer is aware of all queues
available. For this reason, if a queue is used by any service worker
then it should be specified within this array. It is not necessary to
include the ``'celery'`` or ``'platform.fifo'`` queues. Defaults to the
contents of the ``DEFAULT_QUEUES`` variable in the modules
```__init__.py`` file </cadasta/workertoolbox/__init__.py>`__.

``PLATFORM_QUEUE_NAME`` *(provided via argument)*
'''''''''''''''''''''''''''''''''''''''''''''''''

Defaults to ``'platform.fifo'``.

*Note: It is recommended that developers not alter this setting.*

``CHORD_UNLOCK_MAX_RETRIES`` *(provided via argument)*
''''''''''''''''''''''''''''''''''''''''''''''''''''''

Used to set the maximum number of times a ``celery.chord_unlock`` task
may retry before giving up. See celery/celery#2725. Defaults to
``43200`` (meaning to give up after 6 hours, assuming the default of the
task's ``default_retry_delay`` being set to 1 second).

``SETUP_LOGGING`` *(provided via argument)*
'''''''''''''''''''''''''''''''''''''''''''

Controls whether a default logging configuration should be applied to
the application. At a bare minimum, this includes: \* creating a console
log handler for ``INFO`` level logs \* a file log handlers for ``INFO``
level logs, saved to ``app.info.log`` \* a file log handlers for
``ERROR`` level logs, saved to ``app.error.log``

If the ``OPBEAT_ORGANIZATION_ID`` environment variable is set, the
following logging configuration take place: \* add an
`OpBeat <https://opbeat.com/>`__ file handle for ``ERROR`` level logs \*
add an `OpBeat <https://opbeat.com/>`__ `task\_failure
signal <http://docs.celeryproject.org/en/latest/userguide/signals.html#task-failure>`__
handler to log all faild tasks

Defaults to ``True``.

``QUEUE_NAME_PREFIX`` *(provided via environment variable)*
'''''''''''''''''''''''''''''''''''''''''''''''''''''''''''

Used to populate the ``queue_name_prefix`` value of the connections
``broker_transport_options``. Defaults to value of ``QUEUE_PREFIX``
environment variable if populated, ``'dev'`` if not.

``RESULT_DB_USER`` *(provided via environment variable)*
''''''''''''''''''''''''''''''''''''''''''''''''''''''''

Used to populate the default ``result_backend`` template. Defaults to
``RESULT_DB_USER`` environment variable if populated, ``'cadasta'`` if
not.

``RESULT_DB_PASS`` *(provided via environment variable)*
''''''''''''''''''''''''''''''''''''''''''''''''''''''''

Used to populate the default ``result_backend`` template. Defaults to
``RESULT_DB_PASS`` environment variable if populated, ``'cadasta'`` if
not.

``RESULT_DB_HOST`` *(provided via environment variable)*
''''''''''''''''''''''''''''''''''''''''''''''''''''''''

Used to populate the default ``result_backend`` template. Defaults to
``RESULT_DB_HOST`` environment variable if populated, ``'localhost'`` if
not.

``RESULT_DB_PORT`` *(provided via environment variable)*
''''''''''''''''''''''''''''''''''''''''''''''''''''''''

Used to populate the default ``result_backend`` template. Defaults to
``RESULT_DB_PORT`` environment variable if populated, ``'cadasta'`` if
not.

``RESULT_DB_NAME`` *(provided via environment variable)*
''''''''''''''''''''''''''''''''''''''''''''''''''''''''

Used to populate the default ``result_backend`` template. Defaults to
``RESULT_DB_PORT`` environment variable if populated, ``'5432'`` if not.

``cadasta.workertoolbox.setup.setup_app``
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

After the Celery application is provided a configuration object, there
are other steups that must follow to properly configure the application.
For example, the exchanges and queues described in the configuration
must be declared. This function calls those required followup
procedures. Typically, it is called automatically by the
```worker_init`` <http://docs.celeryproject.org/en/latest/userguide/signals.html#worker-init>`__
signal, however it must be called manually by codebases that are run
only as task producers or from within a Python shell.

It takes two arguments:

-  ``app`` - A ``Celery()`` app instance. *Required*
-  ``throw`` - Boolean stipulating if errors should be raise on failed
   setup. Otherwise, errors will simply be logged to the module logger
   at ``exception`` level. *Optional, default: True*

``cadasta.workertoolbox.tests.build_functional_tests``
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

When provided with a Celery app instance, this function generates a
suite of functional tests to ensure that the provided application's
configuration and functionality conforms with the architecture of the
Cadasta asynchronous system.

An example, where an instanciated and configured ``Celery()`` app
instance exists in a parallel ``celery`` module:

.. code:: python

    from cadasta.workertoolbox.tests import build_functional_tests

    from .celery import app

    FunctionalTests = build_functional_tests(app)

To run these tests, use your standard test runner (e.g. ``pytest``) or
call manually from the command-line:

.. code:: bash

    python -m unittest path/to/tests.py

Contributing
------------

Testing
~~~~~~~

.. code:: bash

    pip install -e .
    pip install -r requirements-test.txt
    ./runtests

Deploying
~~~~~~~~~

.. code:: bash

    pip install -r requirements-deploy.txt
    python setup.py test clean build tag publish

.. |PyPI version| image:: https://badge.fury.io/py/cadasta-workertoolbox.svg
   :target: https://badge.fury.io/py/cadasta-workertoolbox
.. |Build Status| image:: https://travis-ci.org/Cadasta/cadasta-workertoolbox.svg?branch=master
   :target: https://travis-ci.org/Cadasta/cadasta-workertoolbox
.. |Requirements Status| image:: https://requires.io/github/Cadasta/cadasta-workertoolbox/requirements.svg?branch=master
   :target: https://requires.io/github/Cadasta/cadasta-workertoolbox/requirements/?branch=master


