Metadata-Version: 2.5
Name: tutor-contrib-clickhouse-log-retention
Version: 0.1.2
Summary: ClickHouse system log retention (TTL, profiler off) for Cairn and Aspects, a plugin for Tutor
Project-URL: Documentation, https://github.com/Abstract-Tech/tutor-contrib-clickhouse-log-retention#readme
Project-URL: Issues, https://github.com/Abstract-Tech/tutor-contrib-clickhouse-log-retention/issues
Project-URL: Source, https://github.com/Abstract-Tech/tutor-contrib-clickhouse-log-retention
Author: Emad Ehsanrad
Author-email: emad.ehsanrad@gmail.com
License: AGPL-3.0-only
License-File: LICENSE.txt
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: GNU Affero General Public License v3
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: Implementation :: CPython
Classifier: Programming Language :: Python :: Implementation :: PyPy
Requires-Python: >=3.9
Requires-Dist: tutor>=14.0.0
Provides-Extra: dev
Requires-Dist: ruff; extra == 'dev'
Requires-Dist: tutor[dev]>=14.0.0; extra == 'dev'
Description-Content-Type: text/x-rst

clickhouse-log-retention plugin for `Tutor <https://docs.tutor.edly.io>`__
##########################################################################

Stops ClickHouse's own ``system.*`` log tables (``trace_log``, ``text_log``, ``metric_log``,
``asynchronous_metric_log``, ...) from filling the disk. Works for both **Cairn**
(``cairn-clickhouse``) and **Aspects** (``clickhouse``).

ClickHouse records its own telemetry in the ``system`` database and never expires it. On
Open edX instances this telemetry is often more than 95% of what ClickHouse stores.

What it does
************

1. Sets a TTL on every system log table (default 7 days, ``query_log`` 30 days).
2. Turns the query profiler off (it is what makes ``system.trace_log`` huge).

Nothing else is touched: no data database, no live table is dropped.

Installation
************

.. code-block:: bash

    pip install git+https://github.com/Abstract-Tech/tutor-contrib-clickhouse-log-retention

Usage
*****

.. code-block:: bash

    tutor plugins enable clickhouse-log-retention
    tutor config save
    tutor local restart clickhouse          # Aspects
    tutor local restart cairn-clickhouse    # Cairn

Settings (``tutor config save --set NAME=value``):

- ``CLICKHOUSE_LOG_RETENTION_DAYS``: days to keep the diagnostic logs (default ``7``).
- ``CLICKHOUSE_LOG_RETENTION_QUERY_LOG_DAYS``: days to keep ``system.query_log`` (default ``30``).
  The Aspects performance dashboards read it, so keep it longer.

A value below 1 is treated as 1.

After the first restart (important)
***********************************

ClickHouse renames every existing system log table to ``<name>_0``, ``<name>_1``, ... and
creates a fresh one with the TTL. The renamed copies keep their data and their disk space
until you drop them. List them:

.. code-block:: sql

    SELECT name, formatReadableSize(total_bytes) AS size
    FROM system.tables
    WHERE database = 'system' AND match(name, '_log_[0-9]+$')
    ORDER BY total_bytes DESC;

Each ``system.<name>_N`` listed is an old copy of ClickHouse's own diagnostics: nothing
writes to it and neither Cairn nor Aspects reads it, so ``DROP TABLE system.<name>_N`` is
safe. Never drop a table without a numeric suffix.

To free space that is already used, clean up **before** enabling the plugin: drop old
monthly partitions with ``ALTER TABLE system.<table> DROP PARTITION ID 'YYYYMM'`` (the
renamed copies then stay small).

Disabling the plugin does **not** remove what it rendered. Delete
``env/local/docker-compose.override.yml`` and ``env/plugins/clickhouse-log-retention`` and
run ``tutor config save`` again.

How it works
************

All the XML is in ``tutorclickhouse_log_retention/templates`` (real files, easy to read).
``plugin.py`` has the full explanation.

- Aspects: the TTL goes through Aspects' own patch ``clickhouse-server-config`` (present in
  every release since 0.70). The profiler setting needs its own file in the ``users``
  directory Aspects mounts, because ClickHouse ignores a second ``<profiles>`` block in
  the same file.
- Cairn: no patch exists, so two XML files are rendered and mounted by a
  ``docker-compose.override.yml`` in ``env/local`` (Tutor loads it for ``tutor local``).
  It replaces an existing override file of that name, so do not also write that file by hand
  or with a deploy tool. Other plugins that need compose changes can add their services to the
  same file with the patch ``local-docker-compose-override-services`` (service entries,
  rendered under ``services:``). Kubernetes is not covered.

Requires Tutor 14.0 or newer.

Tested with:

- Cairn: Tutor 14.0 to 22 (Cairn 14 to 22), with a real ``tutor dev start -d cairn-clickhouse``
  on every version, against ClickHouse 21.11, 22.1, 24.1 and 24.2.
- Aspects: Tutor 16 to 22 (Aspects 0.90 to 6.0), against ClickHouse 23.8, 24.3, 24.8 and
  25.8 (Aspects needs Tutor 15 or newer itself).

Hosts that are not Tutor (plain Docker Compose)
***********************************************

Mount the two files in ``manual/`` into the ClickHouse container (the profiler file must be
in ``users.d``, not ``config.d``):

.. code-block:: yaml

    volumes:
      - ./data:/var/lib/clickhouse/
      - ./zz-system-ttl.xml:/etc/clickhouse-server/config.d/zz-system-ttl.xml:ro
      - ./zz-noprofiler.xml:/etc/clickhouse-server/users.d/zz-noprofiler.xml:ro

Recreate the container (``docker compose up -d``). The same renaming as above happens on
the first start.

License
*******

This software is licensed under the terms of the AGPLv3.
