Metadata-Version: 1.1
Name: airflow-run
Version: 0.1.9
Summary: Simplified Airflow CLI Tool for Lauching CeleryExecutor Deployment
Home-page: https://github.com/paulokuong/airflow-run
Author: Paulo Kuong
Author-email: paulo.kuong@gmail.com
License: MIT
Description: Airflow Run
        ----------------
        
        Python tool for deploying Airflow Multi-Node Cluster.
        
        Requirements
        ------------
        
        -  Python >=3.6 (tested)
        
        Goal
        ----
        
        | To provide a quick way to setup Airflow Multi-Node Cluster (a.k.a. Celery Executor Setup).
        
        Steps
        -----
        1. Generate config yaml file.
        2. Run commands to start webserver, scheduler, worker, (rabbitmq, postgres).
        3. Add dag files and run initdb.
        
        Generate config file:
        ---------------------
        
        .. code:: python
        
            afr --generate_config
        
        Running the tool in the same directory as config file:
        ------------------------------------------------------
        
        .. code:: python
        
            afr --run postgresql
            afr --run initdb
            afr --run rabbitmq
            afr --run webserver
            afr --run scheduler
            afr --run worker --queue {queue name}
            afr --run flower
        
        Or, running the tool specifying config path:
        --------------------------------------------
        
        .. code:: python
        
            afr --run postgresql --config /path/config.yaml
        
        Or, use this environment variable to set the config path:
        ---------------------------------------------------------
        
        .. code:: python
        
            export AIRFLOWRUN_CONFIG_PATH="/some_path/config.yaml"
        
        After running webserver, scheduler and worker (postgres and rabbitmq if needed local instances), Add your dag files in the dags subdirectory in the directory you defined in the config file. Then, run the following:
        
        .. code:: python
        
            afr --run initdb
        
        (* note: make sure you have the correct user permission in the dags, logs subdirectories.)
        
        That is it!!
        
        
        Default Config yaml file:
        -------------------------
        
        .. code-block:: yaml
        
            private_registry: False
            registry_url: registry.hub.docker.com
            username: ""
            password: ""
            repository: pkuong/airflow-run
            image: airflow-run
            tag: latest
            local_dir: {local directory where you want to mount /dags and /logs folder}
            webserver_port: 8000
            flower_port: 5555
            env:
              AIRFLOW__CORE__EXECUTOR: CeleryExecutor
              AIRFLOW__CORE__LOAD_EXAMPLES: "False"
              AIRFLOW__CORE__DAGS_FOLDER: /usr/local/airflow/airflow/dags
              AIRFLOW__CORE__LOGS_FOLDER: /usr/local/airflow/airflow/logs
              AIRFLOW_HOME: /usr/local/airflow
            rabbitmq:
              name: rabbitmq
              username: {username}
              password: {password}
              host: {IP}
              virtual_host: /
              image: rabbitmq:3-management
              home: /var/lib/rabbitmq
              ui_port: 15672
              port: 5672
              env:
                RABBITMQ_DEFAULT_USER: {username}
                RABBITMQ_DEFAULT_PASS: {password}
            postgresql:
              name: postgresql
              username: {username}
              password: {password}
              host: {host}
              image: postgres
              data: /var/lib/postgresql/data
              port: 5432
              env:
                PGDATA: /var/lib/postgresql/data/pgdata
                POSTGRES_USER: {username}
                POSTGRES_PASSWORD: {password}
        
        
        Docker image
        ------------
        
        | This tool is using the following public docker image by default.
        
        .. code:: python
        
            https://hub.docker.com/repository/docker/pkuong/airflow-run
        
        Building the image:
        -------------------
        
        | If you want to build your own image, you can run the following:
        
        .. code:: python
        
            afd --build --config_path={absolute path to config.yaml} --dockerfile_path={absolute path to directory which contains Dockerfile}
        
        Contributors
        ------------
        
        -  Paulo Kuong (`@pkuong`_)
        
        .. _@pkuong: https://github.com/paulokuong
        
        .. |Build Status| image:: https://travis-ci.org/paulokuong/airflow-run.svg?branch=master
        .. target: https://travis-ci.org/paulokuong/airflow-run
        
Keywords: Airflow CeleryExecutor distributed portable deployment runner docker kubernetes
Platform: UNKNOWN
Classifier: Development Status :: 3 - Alpha
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3.4
