Metadata-Version: 2.5
Name: ekbatan-py-db-scheduler
Version: 0.1.0
Summary: A port of kagkarlsson/db-scheduler: persistent, cluster-friendly scheduling on the database you already have.
License-Expression: Apache-2.0
License-File: LICENSE
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Database
Classifier: Typing :: Typed
Requires-Python: >=3.13
Requires-Dist: croniter>=6.0
Requires-Dist: sqlalchemy>=2.0.30
Provides-Extra: mariadb
Requires-Dist: pymysql>=1.1; extra == 'mariadb'
Provides-Extra: mysql
Requires-Dist: pymysql>=1.1; extra == 'mysql'
Provides-Extra: postgres
Requires-Dist: psycopg[binary]>=3.2; extra == 'postgres'
Description-Content-Type: text/markdown

# db-scheduler, in Python

A port of [kagkarlsson/db-scheduler](https://github.com/kagkarlsson/db-scheduler) — persistent,
cluster-friendly scheduling backed by the database you already have. No broker, no separate
worker protocol: a row in a table, claimed by whichever instance gets there first.

It exists because nothing in the Python ecosystem does this. Every candidate was benched against
two processes sharing one database, running one job every second for eight seconds; the correct
answer is eight executions:

| | result |
| --- | --- |
| huey + SqlHuey (± `lock_task`) | 16 — every slot run twice |
| sqlalchemy-celery-beat | 14 — every slot run twice |
| APScheduler 3.11 + SQLAlchemy store | 14 — six slots run twice |
| APScheduler 4.0.0a6 | 12/12 correct, but no failover without Redis or Postgres, and alpha since 2020 |

The reasoning, and the full comparison, is in `../DIVERGENCE.md`.

## Using it

```python
from datetime import timedelta

from sqlalchemy import MetaData, create_engine

from db_scheduler.builder import create
from db_scheduler.jdbc import scheduled_tasks_table
from db_scheduler.task.helper import Tasks
from db_scheduler.task.schedule import Schedules

engine = create_engine("postgresql+psycopg://localhost/app")

# Once, at deploy time - or run db-scheduler's own DDL, which produces the same table.
metadata = MetaData()
scheduled_tasks_table(metadata)
metadata.create_all(engine)

send_email = Tasks.one_time("send-email", dict).execute(
    lambda instance, context: mailer.send(instance.data["to"])
)
nightly = Tasks.recurring("nightly-report", Schedules.cron("0 0 3 * * ?")).execute(
    lambda instance, context: reports.run()
)

scheduler = create(engine).add_tasks(send_email).add_start_tasks(nightly).threads(10).build()
scheduler.start()

# From anywhere, including a process that runs no scheduler at all:
scheduler.schedule_if_not_exists(
    send_email.instance("welcome-42", {"to": "ada@example.com"}), when
)
```

An application server that only ever *creates* work does not need a scheduler. It builds a
client over the same table - Java's `SchedulerClient.Builder.create(dataSource, tasks...)` - and
the workers pick the rows up:

```python
from db_scheduler.client import SchedulerClient

client = SchedulerClient.builder(engine, send_email).build()
client.schedule_if_not_exists(send_email.instance("welcome-42", {"to": "ada@example.com"}), when)
```

The builder takes Java's settings: `.table_name(...)`, `.serializer(...)`, `.jdbc_customization(...)`
and `.enable_priority()`. The tasks it is given are the ones whose rows it reads back.

Testing is the other half. `ManualScheduler` is the real scheduler with the threads and the
clock taken out, so an hour of retry backoff is one `tick(timedelta(hours=1))`:

```python
from db_scheduler.testhelper import create_manual_scheduler

scheduler = create_manual_scheduler(engine, send_email).clock(SettableClock(now)).start()
scheduler.schedule_if_not_exists(send_email.instance("an-id"), now)
scheduler.run_any_due_executions()      # runs on this thread, and has finished when it returns
```

## Shutting down

`stop()` is bounded, and it does not interrupt a poll that is already running — db-scheduler
makes the same choice, and for the same reason: a poll is mid-conversation with the database,
and cutting it off could leave rows claimed but never handed to a worker. It is allowed to
finish (a query or two), and it stops *claiming* new work the instant shutdown begins.

`stop()` stops polling, waits up to `shutdown_max_wait` for running executions,
then warns and names anything still going. It never blocks past that, and it never keeps the
process alive: the worker threads are daemon threads, because Python has no counterpart to
Java's `Thread.interrupt()` and the interpreter joins every non-daemon thread at exit — so one
handler that ignores shutdown would otherwise hang the process for ever.

An execution abandoned that way is not lost. It is the same state as the machine losing power:
the row stays picked, its heartbeat stops, and the next instance to run dead-execution detection
revives it. Handlers that want to finish cleanly should check
`context.scheduler_state.is_shutting_down`.

## Threads, and forking

Everything here runs on threads. The library never creates a process: three loop threads (poll,
heartbeat, dead-execution detection) plus a worker pool of `threads(n)`.

Build the `Scheduler` **after** any `fork()` — under gunicorn `--preload`, uwsgi, or a manual
fork. Threads do not survive a fork, so a scheduler built beforehand has no workers in the
child, and the database connections it inherited cannot be shared across the fork either. This
is the same rule SQLAlchemy has for engines, for the same reason. A pool that finds itself in a
forked child refuses work with an explanatory error rather than silently accepting executions
nobody will ever run.

## Connections

This library does not pool anything. It takes an `Engine` and asks it for connections, the same
way db-scheduler takes a `DataSource` and asks that — HikariCP is a *test* dependency in
db-scheduler's own build. How connections are pooled, sized and shared is yours to decide.

That matters for one specific reason. The heartbeat is how a running execution proves it is
alive. If task bodies hold every connection in the pool, the heartbeat cannot write, the row's
timestamp stops moving, and another instance declares the execution dead and **runs it again**.

Measured with three workers each holding a connection:

| | executions run twice |
| --- | --- |
| scheduler shares the application's exhausted pool | **3 of 3** |
| scheduler has an engine of its own | 0 of 3 |

So the recommendation, in order of preference:

```python
# Best: the scheduler gets its own connections, and task bodies cannot starve it.
scheduler = create(create_engine(url, pool_size=13)).threads(10).build()
# ... while the application keeps using its own engine for everything else.
```

Failing that, size one shared pool above `threads(n)` — the scheduler's poll, heartbeat and
dead-execution threads each need a connection too, so `threads + 3` is the floor. `build()`
warns at startup when the pool cannot outnumber the workers, including with SQLAlchemy's default
pool of 15, where `threads(20)` looks perfectly reasonable and is not.

## Clocks

Dead-execution detection compares a heartbeat written by *another* instance against this one's
clock, so the fleet's clocks must agree to within the dead-threshold —
`heartbeat_interval * missed_heartbeats_limit`. Beyond that, an instance whose clock runs fast
declares a healthy execution dead and runs it again.

The default threshold is **thirty minutes** (5 min × 6), which no plausible NTP drift comes
near. It only becomes reachable if you tune the heartbeat down for faster recovery — so if you
do, keep the threshold comfortably above the worst skew you expect. db-scheduler makes the same
assumption; it is inherent to using wall-clock timestamps for liveness.

## Limits worth knowing

`task_data` is a `blob` on MySQL and MariaDB, as it is in every db-scheduler DDL file — so
payloads over ~64KB are **rejected** there while Postgres accepts them. The failure is loud
(`DataError`), never a silent truncation, but it is a way for something that works on a Postgres
laptop to fail on a MySQL production database. Keep task data small and point at the payload
rather than carrying it.

Task names and instance ids are `varchar(100)`. Postgres and MySQL reject anything longer;
SQLite does not enforce `varchar` lengths at all and stores the whole thing. Nothing truncates
silently on any of them.

## Self-contained

Its own `pyproject.toml`, its own dependencies, its own tests. It does not import from `ekbatan`
and `ekbatan` does not import from it. The table it writes is db-scheduler's `scheduled_tasks`,
so a Java service and a Python service coordinate through one row per task.

## Layout

    src/db_scheduler/          the library, mirroring the Java package structure
    tests/unit/                no database
    tests/integration/         Testcontainers: postgres, mysql, mariadb
    scripts/ledger.py          counts docs/FILE-LEDGER.md
    docs/FILE-LEDGER.md        every Java source file, and where it went
    docs/PORTING.md            the plan, the deliberate differences, the verification status
    docs/java-api-inventory.md   every public Java signature, machine-extracted
    docs/java-test-inventory.md  every Java test method, machine-extracted

## What is ported

All 159 Java source files. 133 ported, 26 recorded as not-applicable with a reason — mostly the
JDBC plumbing SQLAlchemy already is, and the Gson/Jackson bindings the JSON serializer replaces.
`python scripts/ledger.py` counts it.

## Running the tests

    pytest                          # sqlite, about three seconds
    pytest --dialects=postgres      # one container
    pytest --dialects=all           # sqlite, postgres, mysql, mariadb - the same assertions
    ruff check . && mypy

| dialect | result |
| --- | --- |
| SQLite | 750 passed, 14 skipped |
| PostgreSQL 17 | 760 passed, 4 skipped |
| MySQL 8.4 (LTS) | 760 passed, 4 skipped |
| MySQL 26.7 (current) | 760 passed, 4 skipped |
| MariaDB 11.4 | 758 passed, 6 skipped |

Every integration test runs twice per dialect, once for each timestamp shape - zone-less UTC
columns, and the zoned columns a Java-created table has. Reading a table in the wrong shape is
an hour's silent drift rather than an error, so both are exercised rather than assumed.

Times are stored exactly as given and read back as the driver returns them - nothing converts
them. The default clock gives the current UTC time with no zone attached, right for zone-less
columns; a zoned (`timestamptz`) table needs a clock that gives aware UTC, set with
`.clock(...)` on the builder.

The skips are lock-and-fetch on the dialects that have no `SELECT ... FOR UPDATE SKIP LOCKED`,
plus the test asserting the refusal on those — so the counts differ by exactly the dialects'
real capabilities.
