Metadata-Version: 2.4
Name: django-celery-task-view
Version: 0.1.0
Summary: A Celery task monitoring page for Django that reads task results, durations and per-task logs straight from one or more Redis result backends - no extra database tables or event consumers.
Author: Ian Jones
License-Expression: MIT
Project-URL: Homepage, https://github.com/jonesim/django-celery-task-view
Classifier: Programming Language :: Python :: 3
Classifier: Framework :: Django
Classifier: Operating System :: OS Independent
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: celery
Requires-Dist: redis
Dynamic: license-file

# django-celery-task-view

A Celery task monitoring page for Django that reads task results, durations and per-task logs
**straight from the Redis result backend** — no extra database tables, no event consumers, no
separate monitoring process to keep alive.

Because the page reads state Celery has already written (plus two small sibling keys written by
best-effort worker hooks), an outage of the *page* can never lose history: whatever is in Redis
is what you see. This is deliberately different from Flower, which consumes the worker event
stream — events expire in seconds with no consumer attached, and Flower's state is in memory,
so a Flower outage becomes a silent hole. It also differs from `django-celery-results`, which
adds a database table and a second write path.

What you get:

* **All Tasks / Failures tabs** — every task result currently in Redis, newest first, with
  status, worker, args/kwargs and duration; running tasks sort to the top.
* **A detail modal** — full meta, result, traceback, and the task's captured log lines
  (including a still-running task's log so far).
* **Multiple result backends on one page** — e.g. one per region or deployment — via a
  `{label: redis_url}` setting, each row tagged with its source.

## Requirements

* Redis result backend (`CELERY_RESULT_BACKEND = 'redis://...'`).
* For the UI, the following django apps installed (the worker hooks and the reader work
  without them):

      'django_datatables', 'django_modals', 'django_menus', 'ajax_helpers'

  from the PyPI packages

      django-filtered-datatables, django-nested-modals, django-tab-menus, django-ajax-helpers

## Setup

`settings.py`:

```python
INSTALLED_APPS = [
    ...,
    'celery_task_view',
]

CELERY_RESULT_EXTENDED = True        # without this the meta has no task name / args / worker
CELERY_TASK_TRACK_STARTED = True     # running tasks appear with status STARTED
CELERY_RESULT_EXPIRES = 60 * 60 * 24 * 7   # how much history the page shows
```

These are read by the **worker** when it writes the meta, so restart the celery workers after
changing them.

Your project's `celery.py` (for durations and log capture — the page works without this, but
shows no Duration or Log):

```python
import celery_task_view.signals  # noqa: F401
```

`urls.py`:

```python
path('celery-tasks/', include('celery_task_view.urls')),
```

The default views require the `access_admin` permission (override `permission_required` on a
subclass to change this).

## Multiple result backends

```python
CELERY_TASK_VIEW_SOURCES = {
    'UK': 'redis://redis:6379/0',
    'DE': 'redis://redis:6379/1',
    'US': 'redis://redis:6379/2',
}
```

With more than one entry the table gains a Source column and filter, and the detail modal shows
which source the task came from. Without the setting the single source is
`CELERY_RESULT_BACKEND` and no Source column is shown. A source whose Redis is unreachable
contributes no rows rather than blanking the page.

If sources run with different `TIME_ZONE`s and `USE_TZ` off, celery writes local aware
datetimes; the page moves them all onto the viewing project's clock so they sort comparably.

## Per-task logs

With `celery_task_view.signals` imported in the worker, a logging handler on the worker's root
logger copies every record emitted *while a task is running* into a capped Redis list
(`celery-task-log-<uuid>`) beside the task's result, with the same TTL. The worker's normal
stdout logging is untouched — the Redis copy is additional. `print()` output is captured too
while celery's default `worker_redirect_stdouts` is on. Records emitted outside any task
(worker startup, beat) are not captured.

The log is written line by line as the task runs, and **the detail modal is live**: while a
task is still running the modal re-polls its own url every 2 seconds (`refresh_ms` on the modal
class) and updates the status, duration and log in place. The poll stops by itself when the
task finishes or the modal is closed, and each task's modal polls only its own detail, so
switching between task modals never shows another task's log. A task that dies without ever
writing a result still leaves the lines it got through.

Settings (optional):

```python
CELERY_TASK_VIEW_CAPTURE_LOGS = True   # False turns the handler off
CELERY_TASK_VIEW_LOG_LEVEL = 'INFO'    # minimum level captured
CELERY_TASK_VIEW_LOG_LINES = 1000      # lines kept per task (oldest trimmed)
```

Every captured line is one Redis round trip; a task that logs in a tight loop pays for it.
`LOG_LINES` caps memory, not traffic.

## Demo project

`django_examples/` is a runnable demo:

```bash
docker compose up --build
# http://localhost:8031
docker compose exec django python manage.py createsuperuser   # to view the default page
```

The branded page (`/branded-celery-tasks/`) is open; the default page (`/celery-tasks/`)
requires a logged-in user with the `access_admin` permission (a superuser qualifies - the demo
routes anonymous visitors through the admin login).

The home page queues demo tasks (quick, slow, failing, chatty, print) against **two workers on
separate result-backend dbs**, so the tasks page shows a Source column and filter. `slow_task`
runs long enough to watch: it sorts to the top in STARTED, and its log grows live in the detail
modal while it runs. The demo also mounts a branded copy of the page at `/branded-celery-tasks/`
(see `task_examples/views.py` and `templates/task_examples/branded_tasks.html`).

## Tests

The suite lives in `celery_task_view/tests/` and runs with Django's test runner from the demo
project (it needs a real Redis - the reader is SCAN/MGET against one):

```bash
docker compose exec django python manage.py test celery_task_view
```

Tests use two scratch Redis dbs (8 and 9 by default, `CELERY_TASK_VIEW_TEST_REDIS` / `_2` to
override) and refuse to touch anything else; lost-task detection is mocked so no broker
broadcasts happen. A few warning lines in the output (unreachable source, unreadable json,
swallowed redis errors) are the error paths being exercised, not failures.

## Lost tasks

A worker restart kills its in-flight tasks, and no final state is ever written for them - their
meta would say STARTED (or PROGRESS) until it expired. So for any task whose stored state says
it is underway, the page asks that source's workers what they are actually executing
(`inspect().active()`, cached for 10 seconds) and shows the task as **LOST** when no worker has
it. Lost tasks appear on the Failures tab, keep the log lines they got through, sort where they
started rather than at the top, and their modal says why and stops polling. RETRY is exempt - a
task waiting out its retry countdown sits in the queue, which inspect cannot see.

The broadcast needs the broker reachable from the web process. By default the source urls are
used (broker and result backend sharing a Redis db); set `CELERY_TASK_VIEW_BROKERS`
(`{label: broker_url}`) when they differ. If the broker cannot be asked, nothing is flagged.

```python
CELERY_TASK_VIEW_DETECT_LOST = True    # False turns the check off
CELERY_TASK_VIEW_INSPECT_TIMEOUT = 1.0 # seconds to wait for worker replies
```

## How it works

* Task results live in Redis as `celery-task-meta-<uuid>` keys; the page SCANs and MGETs them
  live on each load. Nothing is copied or stored elsewhere.
* Celery never records a task's start time anywhere durable (TRACK_STARTED's meta is
  overwritten by the final write), so a `task_prerun` hook stamps `celery-task-start-<uuid>`;
  duration is `date_done - start`, or `now - start` while running.
* All three keys share the result's TTL (`CELERY_RESULT_EXPIRES`), so a task's row, start stamp
  and log expire together.
* The table is list backed: the whole array (capped at `max_records`, default 5000) is inlined
  into the page and DataTables handles sort/search/paging client side.
* Task args and results are untrusted input: row values are escaped in Python, and result /
  traceback / log render escaped inside `<pre>` in the modal only.

## Branding the page

The views build the tab strip and table into a single HTML string,
`{{ celery_tasks_content }}`, so it can be dropped into your own template. Subclass the base
views and set `template_name`:

```python
from celery_task_view.enhanced_views import CeleryTaskListBaseView, CeleryTaskFailuresBaseView
from celery_task_view.modals import CeleryTaskModalMixin

class MyCeleryTaskList(CeleryTaskListBaseView):
    template_name = 'myapp/celery_tasks.html'

class MyCeleryTaskFailures(CeleryTaskFailuresBaseView):
    template_name = 'myapp/celery_tasks.html'

class MyCeleryTaskModal(MyPermissionModal, CeleryTaskModalMixin):   # or (CeleryTaskModalMixin, Modal)
    pass
```

The template must include the ajax_helpers/datatables/modals libraries and the page script,
then place the content wherever it fits your layout:

```html
{% load ajax_helpers %}
{% lib_include 'ajax_helpers' 'Bootstrap' 'FontAwesome' module='ajax_helpers.includes' %}
{% lib_include 'datatable' module='django_datatables.includes' %}
{% lib_include 'Modals' module='django_modals.includes' %}
{{ ajax_helpers_script }}
...
{{ celery_tasks_content }}
```

Register the subclasses with `celery_task_urlpatterns` so the tab menu and modal (which reverse
the standard `celery_task_view:` URL names) point at your views:

```python
from celery_task_view.urls import celery_task_urlpatterns

urlpatterns = [
    path('celery-tasks/', include((celery_task_urlpatterns(
        list_view=MyCeleryTaskList, failures_view=MyCeleryTaskFailures,
        modal_view=MyCeleryTaskModal), 'celery_task_view'))),
]
```

## Gotchas

* `djcelery_email` sends emails with `ignore_result: True` — they never appear on the page
  (and create no result volume).
* Results written before `CELERY_RESULT_EXTENDED` was enabled show as `(name not recorded)`.
* If your Redis container has no volume, results (and anything queued) are wiped on every
  container recreate — give it one.
* The log handler only works with the Redis result backend (it reuses the backend's client);
  on any other backend it does nothing, silently.
