Metadata-Version: 1.1
Name: cadasta-workertoolbox
Version: 0.3.3
Summary: Cadasta Worker Toolbox
Home-page: https://github.com/Cadasta/cadasta-workertoolbox
Author: Anthony Lukach
Author-email: alukach@cadasta.org
License: GNU Affero General Public License v3.0
Description: Cadasta Worker Toolbox
        ======================
        
        |PyPI version| |Build Status| |Requirements Status|
        
        A collection of helpers to assist in quickly building asynchronous
        workers for the Cadasta system.
        
        Library
        -------
        
        ``cadasta.workertoolbox.conf.Config``
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
        
        The ``Config`` class was built to simplify configuring Celery settings,
        helping to ensure that all workers adhere to the architecture
        requirements of the Cadasta asynchronous system. It essentially offers a
        diff between Celery's default configuration and the configuration
        required by our system. It is the aim of the class to not require much
        customization on the part of the developer, however some customization
        may be needed when altering configuration between environments (e.g. if
        dev settings vary greatly from prod settings).
        
        Any `Celery
        setting <http://docs.celeryproject.org/en/v4.0.2/userguide/configuration.html#new-lowercase-settings>`__
        may be submitted via keyword argument or via environment variable.
        Arguments submitted via keyword argument are expected to comply with
        Celery's newer lowercase settings rather than their older uppercase
        counterparts. Arguments provided by environment variable should be
        uppercase and be prepended with the prefix ``CELERY_`` (e.g. to set the
        ``task_track_started`` value, an environment variable of
        ``CELERY_TASK_TRACK_STARTED`` should be set). The prefix can be
        customized with a provided ``ENV_PREFIX`` keyword argument or
        ``CELERY_ENV_PREFIX`` environment variable. If both a keyword argument
        and environment variable are provided for a setting, the keyword
        argument takes precedence. Settings with non-string defaults will have
        the environment variable values run through
        ```ast.literal_eval`` <https://docs.python.org/3/library/ast.html#ast.literal_eval>`__,
        supporting Python native types like ``bool`` or ``tuple``. Only
        lowercase settings are shown when calling ``repr`` on the ``Conf``
        instance.
        
        Once applied, all settings (and internal variables) are available on the
        Celery ``app`` instance's ``app.conf`` object.
        
        Provided Configuration
        ^^^^^^^^^^^^^^^^^^^^^^
        
        Below is the configuration that the ``Config`` class will provide to a
        ``Celery`` instance.
        
        ``result_backend``
        ''''''''''''''''''
        
        Defaults to
        ``'db+postgresql://{0.RESULT_DB_USER}:{0.RESULT_DB_PASS}@{0.RESULT_DB_HOST}/{0.RESULT_DB_NAME}'``
        rendered with ``self``.
        
        ``broker_transport``
        ''''''''''''''''''''
        
        Defaults to ``'sqs``'.
        
        ``broker_transport_options``
        ''''''''''''''''''''''''''''
        
        Defaults to:
        
        .. code:: python
        
            {
                'region': 'us-west-2',
                'queue_name_prefix': '{}-'.format(QUEUE_NAME_PREFIX)
            }
        
        ``task_queues``
        '''''''''''''''
        
        Defaults to the following ``set`` of ``kombu.Queue`` objects, where
        ``queues`` is the configuration's internal ``QUEUES`` variable and
        ``exchange`` is a ``kombu.Exchange`` object constructed from the
        ``task_default_exchange`` and ``task_default_exchange_type`` settings:
        
        .. code:: python
        
            set([
                Queue('celery', exchange, routing_key='celery'),
                Queue(platform_queue, exchange, routing_key='#'),
            ] + [
                Queue(q_name, exchange, routing_key=q_name)
                for q_name in queues
            ])
        
        *Note: It is recommended that developers not alter this setting.*
        
        ``task_routes``
        '''''''''''''''
        
        Defaults to a function that will generate a dict with the
        ``routing_key`` matching the value at the first index of a task name
        split on the ``.`` and the ``exchange`` set to a ``kombu.Exchange``
        object constructed from the ``task_default_exchange`` and
        ``task_default_exchange_type`` settings
        
        *Note: It is recommended that developers not alter this setting.*
        
        ``task_default_exchange``
        '''''''''''''''''''''''''
        
        Defaults to ``'task_exchange'``
        
        ``task_default_exchange_type``
        ''''''''''''''''''''''''''''''
        
        Defaults to ``'topic'``
        
        ``task_track_started``
        ''''''''''''''''''''''
        
        Defaults to ``True``.
        
        Internal Variables
        ^^^^^^^^^^^^^^^^^^
        
        Below are arguments and environmental variables that can be used to
        customize the above provided configuration. By convention, all variables
        used to construct Celery configuration should should be written entirely
        uppercase. Unless otherwise stated, all variables may be specified via
        argument or environment variable (with preference given to argument).
        
        ``QUEUES``
        ''''''''''
        
        This should contain an array of names for all service-related queues
        used by the Cadasta Platform. These values are used to construct the
        ``task_queues`` configuration. For the purposes of routing followup
        tasks, it's important that every task consumer is aware of all queues
        available. For this reason, if a queue is used by any service worker
        then it should be specified within this array. It is not necessary to
        include the ``'celery'`` or ``'platform.fifo'`` queues. Defaults to the
        contents of the ``DEFAULT_QUEUES`` variable in the modules
        ```__init__.py`` file </cadasta/workertoolbox/__init__.py>`__.
        
        ``PLATFORM_QUEUE_NAME``
        '''''''''''''''''''''''
        
        Defaults to ``'platform.fifo'``.
        
        *Note: It is recommended that developers not alter this setting.*
        
        ``CHORD_UNLOCK_MAX_RETRIES``
        ''''''''''''''''''''''''''''
        
        Used to set the maximum number of times a ``celery.chord_unlock`` task
        may retry before giving up. See celery/celery#2725. Defaults to
        ``43200`` (meaning to give up after 6 hours, assuming the default of the
        task's ``default_retry_delay`` being set to 1 second).
        
        ``SETUP_LOGGING``
        '''''''''''''''''
        
        Controls whether a default logging configuration should be applied to
        the application. At a bare minimum, this includes:
        
        -  creating a console log handler for ``INFO`` level logs
        -  a file log handlers for ``INFO`` level logs, saved to
           ``app.info.log``
        -  a file log handlers for ``ERROR`` level logs, saved to
           ``app.error.log``
        
        If the ``OPBEAT_ORGANIZATION_ID`` environment variable is set, the
        following logging configuration take place:
        
        -  add an `OpBeat <https://opbeat.com/>`__ file handle for ``ERROR``
           level logs
        -  add an `OpBeat <https://opbeat.com/>`__ `task\_failure
           signal <http://docs.celeryproject.org/en/latest/userguide/signals.html#task-failure>`__
           handler to log all faild tasks
        
        Defaults to ``True``.
        
        ``QUEUE_PREFIX``
        ''''''''''''''''
        
        Used to populate the ``queue_name_prefix`` value of the connections
        ``broker_transport_options``. Defaults to value of ``QUEUE_PREFIX``
        environment variable if populated, ``'dev'`` if not.
        
        ``RESULT_DB_USER``
        ''''''''''''''''''
        
        Used to populate the default ``result_backend`` template. Defaults to
        ``RESULT_DB_USER`` environment variable if populated, ``'cadasta'`` if
        not.
        
        ``RESULT_DB_PASS``
        ''''''''''''''''''
        
        Used to populate the default ``result_backend`` template. Defaults to
        ``RESULT_DB_PASS`` environment variable if populated, ``'cadasta'`` if
        not.
        
        ``RESULT_DB_HOST``
        ''''''''''''''''''
        
        Used to populate the default ``result_backend`` template. Defaults to
        ``RESULT_DB_HOST`` environment variable if populated, ``'localhost'`` if
        not.
        
        ``RESULT_DB_PORT``
        ''''''''''''''''''
        
        Used to populate the default ``result_backend`` template. Defaults to
        ``RESULT_DB_PORT`` environment variable if populated, ``'cadasta'`` if
        not.
        
        ``RESULT_DB_NAME``
        ''''''''''''''''''
        
        Used to populate the default ``result_backend`` template. Defaults to
        ``RESULT_DB_NAME`` environment variable if populated, ``'5432'`` if not.
        
        ``cadasta.workertoolbox.setup.setup_app``
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
        
        After the Celery application is provided a configuration object, there
        are other steups that must follow to properly configure the application.
        For example, the exchanges and queues described in the configuration
        must be declared. This function calls those required followup
        procedures. Typically, it is called automatically by the
        ```worker_init`` <http://docs.celeryproject.org/en/latest/userguide/signals.html#worker-init>`__
        signal, however it must be called manually by codebases that are run
        only as task producers or from within a Python shell.
        
        It takes two arguments:
        
        -  ``app`` - A ``Celery()`` app instance. *Required*
        -  ``throw`` - Boolean stipulating if errors should be raise on failed
           setup. Otherwise, errors will simply be logged to the module logger
           at ``exception`` level. *Optional, default: True*
        
        ``cadasta.workertoolbox.tests.build_functional_tests``
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
        
        When provided with a Celery app instance, this function generates a
        suite of functional tests to ensure that the provided application's
        configuration and functionality conforms with the architecture of the
        Cadasta asynchronous system.
        
        An example, where an instanciated and configured ``Celery()`` app
        instance exists in a parallel ``celery`` module:
        
        .. code:: python
        
            from cadasta.workertoolbox.tests import build_functional_tests
        
            from .celery import app
        
            FunctionalTests = build_functional_tests(app)
        
        To run these tests, use your standard test runner (e.g. ``pytest``) or
        call manually from the command-line:
        
        .. code:: bash
        
            python -m unittest path/to/tests.py
        
        Contributing
        ------------
        
        Testing
        ~~~~~~~
        
        .. code:: bash
        
            pip install -e .
            pip install -r requirements-test.txt
            ./runtests
        
        Deploying
        ~~~~~~~~~
        
        .. code:: bash
        
            pip install -r requirements-deploy.txt
            python setup.py test clean build tag publish
        
        .. |PyPI version| image:: https://badge.fury.io/py/cadasta-workertoolbox.svg
           :target: https://badge.fury.io/py/cadasta-workertoolbox
        .. |Build Status| image:: https://travis-ci.org/Cadasta/cadasta-workertoolbox.svg?branch=master
           :target: https://travis-ci.org/Cadasta/cadasta-workertoolbox
        .. |Requirements Status| image:: https://requires.io/github/Cadasta/cadasta-workertoolbox/requirements.svg?branch=master
           :target: https://requires.io/github/Cadasta/cadasta-workertoolbox/requirements/?branch=master
        
Platform: UNKNOWN
Classifier: Development Status :: 3 - Alpha
