Metadata-Version: 2.5
Name: taskmaster-python
Version: 0.2.0
Summary: Client library that reports queue metrics to the Taskmaster autoscaling service.
Author-email: "Zagaran, Inc." <theo@zagaran.com>
License-Expression: MIT
License-File: LICENSE
Keywords: autoscaling,celery,metrics,queue,redis
Classifier: Development Status :: 3 - Alpha
Classifier: Framework :: Celery
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Topic :: System :: Monitoring
Classifier: Typing :: Typed
Requires-Python: >=3.9
Requires-Dist: requests>=2.25
Provides-Extra: celery
Requires-Dist: celery[redis]<6,>=5.0; extra == 'celery'
Requires-Dist: kombu<6; extra == 'celery'
Description-Content-Type: text/markdown

# taskmaster-python

Client library for Taskmaster, an autoscaling service. It runs a background
thread inside your app that periodically collects queue metrics and reports them
to the Taskmaster API.

Currently supported backends:

- **Celery with a Redis broker**, reporting per queue:
  - `queue_depth`: number of jobs waiting.
  - `queue_latency`: seconds the oldest waiting job has been on the queue (0 when empty).

## Installation

```bash
pip install "taskmaster-python[celery]"
```

## Configuration

| Environment variable         | Required | Description                                          |
| ---------------------------- | -------- | ---------------------------------------------------- |
| `TASKMASTER_URL`             | yes      | Base URL of the Taskmaster API.                      |
| `TASKMASTER_AUTH_TOKEN`      | yes      | Sent as `Authorization: Api-Key <token>`.            |
| `TASKMASTER_REPORT_INTERVAL` | no       | Seconds between reports. Defaults to `10`, max `30`. |

If `TASKMASTER_URL` is unset, reporting is disabled and nothing runs. That makes
local development need no setup.

## Usage (Celery)

Call `taskmaster_celery` once, after your Celery app is configured:

```python
from celery import Celery
from taskmaster_python.celery import taskmaster_celery

app = Celery("proj")
app.config_from_object("proj.celeryconfig")

taskmaster_celery(app)
```

This starts a daemon thread in the current process. Every interval it measures
the depth and latency of each queue and POSTs a report to
`TASKMASTER_URL/v1/report_metric`. Errors are logged to the `taskmaster_python`
logger and the thread keeps running.

It also connects a `before_task_publish` signal handler. The handler stamps each
task with its publish time in a `taskmaster_published_at` message header. Queue
latency is measured from that stamp, so **call `taskmaster_celery` in every
process that enqueues tasks** (e.g. your web server), with `TASKMASTER_URL` set.
If the oldest job in any of a queue's priority levels has no stamp, that queue's
latency is left out of the report. Its depth is still reported. The stamp uses
the publishing host's clock, so keep hosts NTP-synced.

Queues are discovered from `task_queues`, `task_default_queue`, and dict-style
`task_routes`. If you route tasks with a custom router function, list the queues
explicitly:

```python
taskmaster_celery(app, queues=["default", "emails", "reports"])
```

Multiple apps can be registered. They share one thread and one report per interval.

## ECS task protection (Celery)

When Taskmaster lowers an ECS service's `desiredCount`, ECS picks which tasks to
stop, and it may pick one in the middle of a job. To stop that, pass
`task_protection_minutes`:

```python
taskmaster_celery(app, task_protection_minutes=120)
```

**Before turning it on**, add `ecs:GetTaskProtection` and
`ecs:UpdateTaskProtection` to the worker's ECS task role. Scope them to the tasks in
the worker's cluster (`arn:aws:ecs:<region>:<account_id>:task/<cluster>/*`).
Taskmaster's clusters page for your project shows the Terraform for this. You can
copy that into your repo's Terraform and apply directly.

While a worker has a job running, its ECS task is marked protected through the
ECS agent's [task scale-in protection endpoint](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-scale-in-protection-endpoint.html).
ECS stops unprotected (idle) tasks first. If every task is busy, ECS waits, and
stops each one once its jobs finish. 

- **Set the grant longer than your longest job.** Protection is renewed halfway
  through each grant while the worker is busy, but a renewal can fail (for
  example, if the ECS agent or API is unavailable). The value can be 1 to 2880
  minutes.
- **One worker per ECS task.** Protection covers the whole ECS task, so a
  second worker in the same task could release it while the first is busy. A
  single worker running several jobs at once (`--concurrency`) is fine.
- **Delayed jobs are protected late.** A job with an ETA or countdown (including
  retries), or one held back by a rate limit, is protected within 10 seconds of
  starting rather than before it starts.
- **Deployments wait too.** A rolling deploy won't stop a protected task until
  its jobs finish or the grant expires. Keep the worker service's
  `maximumPercent` at 200 (the ECS default) so ECS can start every new task
  while old ones finish their jobs. A lower maximum holds back the new version
  until busy workers free up.
- **Recommended backstop:** `task_acks_late = True` and
  `task_reject_on_worker_lost = True` (if your jobs are idempotent). A job that
  lands at the moment protection is being released still gets redelivered if its
  task is stopped. With Redis, redelivery after a hard kill waits for the
  broker's `visibility_timeout` (1 hour by default).

It only runs inside Celery worker processes, and only when both `TASKMASTER_URL`
and `ECS_AGENT_URI` are set (ECS sets the latter in every container), so the
same call is safe in web processes and local development.

## Report format

```json
POST /v1/report_metric
Authorization: Api-Key <TASKMASTER_AUTH_TOKEN>

{
  "metrics": [
    {"queue": "celery", "type": "queue_depth", "value": 12},
    {"queue": "celery", "type": "queue_latency", "value": 4.82},
    {"queue": "emails", "type": "queue_depth", "value": 0},
    {"queue": "emails", "type": "queue_latency", "value": 0.0}
  ]
}
```

## Adding a backend

1. Subclass `taskmaster_python.Collector` and implement `collect()` to return a
   list of `Metric` objects.
2. Register it with `taskmaster_python.reporter.add_collector(...)` and call
   `reporter.ensure_running()`, typically from a setup function in a new
   `taskmaster_python.<backend>` subpackage.
3. Add the backend's dependencies as an optional extra in `pyproject.toml`.

New metric kinds subclass `taskmaster_python.Metric` and set a `type` marker.

## License

MIT