Metadata-Version: 2.1
Name: gpumanager
Version: 0.3.3
Summary: Lightweight NVIDIA GPU utilization sampler, Slack reporter, and systemd timer helper
Author: OpenAI Codex
License: MIT
Project-URL: Website, https://happilee12.github.io/gpu-util-webhook/
Keywords: gpu,nvidia,slack,systemd,monitoring,cli
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Science/Research
Classifier: Intended Audience :: System Administrators
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: System :: Monitoring
Classifier: Topic :: System :: Systems Administration
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: tomli >=1.1.0 ; python_version < "3.11"
Requires-Dist: backports.zoneinfo >=0.2.1 ; python_version < "3.9"

# gpumanager

`gpumanager` is a lightweight Python CLI tool for sampling NVIDIA GPU utilization, storing minute-by-minute CSV snapshots, aggregating utilization over a reporting window, and sending GPU-wise summaries to Slack. It is designed to work together with a Slack incoming webhook for notifications.

The installable Python distribution is named `gpumanager`. The CLI entrypoint is `gpumanager`.

Website: https://happilee12.github.io/gpu-util-webhook/
Pip Page: https://pypi.org/project/gpumanager/

## Features

- Samples NVIDIA GPU utilization with `nvidia-smi`
- Stores one CSV file per sample
- Aggregates average utilization by GPU UUID
- Sends reports to Slack via webhook
- Supports interactive configuration
- Installs system-wide `systemd` services and timers
- Uses minimal dependencies and stays close to the standard library

## Requirements

- Linux
- Python 3.8+
- NVIDIA GPU
- `nvidia-smi` in `PATH`
- `systemd` recommended

## Installation

Python 3.8 support uses small compatibility dependencies installed automatically by pip:

- `tomli` on Python < 3.11
- `backports.zoneinfo` on Python < 3.9

```bash
pip install .
# or
pipx install .
```

If you install with `pipx`, make sure the `pipx` binary path is added to your shell:

```bash
pipx ensurepath
source ~/.bashrc
```

After publishing:

```bash
pip install gpumanager
# or
pipx install gpumanager
```

After a published `pipx` install, run this once if needed:

```bash
pipx ensurepath
source ~/.bashrc
```

## Quick Start

```bash
gpumanager init
gpumanager install-systemd --enable-now
```

During `init`, the CLI shows the current server time and a few common cron examples so it is easier to enter `report.report_time`.

If you edit the config file manually after timers are installed, run `gpumanager reload` to apply the updated systemd timer settings.

## Troubleshooting

### 4. Test

After finishing the configuration, send a test report.

```bash
gpumanager test-sample
gpumanager test-report
```

`gpumanager test-report` sends every configured report, and asks for confirmation first when more than one is configured. Use `gpumanager test-report --report weekly` to send just one, or `--yes` to skip the question in a script. The installed timers always target a single report by name, so they never wait for an answer.

If the Slack message arrives normally, the setup is working.

If the message is delivered here but does not arrive at the scheduled time, `gpumanager install-systemd` may not have been run yet. In that case, run `gpumanager status` and check `sample_timer_installed`, `sample.next_trigger`, and the `timer_installed` / `next_trigger` fields of each entry in `reports`. The next scheduled runs are visible directly in status output:

```json
"sample.next_trigger": "Tue 2026-03-24 14:41:35 KST; 9s left",
"reports": [
  {
    "name": "daily",
    "timer": "gpumanager-report-daily.timer",
    "timer_installed": true,
    "next_trigger": "Tue 2026-03-24 14:42:00 KST; 33s left"
  }
]
```

These values are read by parsing the `Trigger:` line from `systemctl status <timer unit>`.

If systemd timers are already installed, `gpumanager init` automatically rewrites and reloads the installed timer files so schedule changes take effect immediately. If you edit the config file manually later, run `gpumanager reload`. 

## Commands

- `gpumanager init`
- `gpumanager test-sample`
- `gpumanager add-report [NAME] [--report-time CRON] [--interval WINDOW]`
- `gpumanager remove-report NAME [--yes]`
- `gpumanager test-report [--report NAME] [--yes]`
- `gpumanager delete-csv`
- `gpumanager status`
- `gpumanager install-systemd`
- `gpumanager uninstall-systemd`
- `gpumanager disable-sample`
- `gpumanager disable-report [--report NAME] [--yes]`
- `gpumanager reload`

## Configuration

The tool searches for configuration in this order:

1. Path passed with `--config`
2. `GPUMANAGER_CONFIG`
3. `~/.config/gpumanager/config.toml`
4. `/etc/gpumanager/config.toml`

Example:

```toml
[slack]
webhook_url = "https://hooks.slack.com/services/..."

[storage]
csv_dir = "/var/lib/gpumanager"

[sample]
interval = "1m"

[[report]]
name = "daily"
report_time = "0 9 * * *"
interval = "1d"

[[report]]
name = "weekly"
report_time = "0 9 * * 1"
interval = "7d"

[general]
timezone = "Asia/Seoul"
server_name = "AICA_H100"
```

Each `[[report]]` block is one schedule, and any number of them can be configured. `name` is required, must be unique, and may contain lowercase letters, digits, `-` and `_` only, because it becomes part of the installed systemd unit name (`gpumanager-report-<name>.timer`).

A single `[report]` table from older versions is still accepted and is read as one report named `default`.

To add a schedule, use `gpumanager add-report`. It appends the block, installs the new timer and starts it, without touching the schedules that are already there:

```bash
gpumanager add-report realtime --report-time "*/10 * * * *" --interval 1h
gpumanager add-report            # asks for name, cron and window
```

To delete a schedule, use `gpumanager remove-report <name>`. It drops the block from the config, disables the timer and deletes its unit files, so the report disappears from `gpumanager status` as well. Editing the config by hand and running `gpumanager reload` does the same thing. The last remaining report cannot be removed; use `uninstall-systemd` to remove everything.

To keep the block but stop the notification, use `gpumanager disable-report --report <name>` instead.

`interval` is the aggregation window, and takes two forms:

- a rolling duration counted back from send time: `30m`, `1h`, `12h`, `1d`, `7d`
- a calendar anchor with the `since:` prefix, aggregating from a boundary up to send time:
  - `since:day`, `since:week` (Monday), `since:month`, `since:quarter` (Jan/Apr/Jul/Oct 1st), `since:year`
  - `since:2026-01-01` for a fixed start date

A quarterly report that averages everything since the most recent quarter boundary:

```toml
[[report]]
name = "quarterly"
report_time = "0 9 1 1,4,7,10 *"   # 09:00 on Jan/Apr/Jul/Oct 1st
interval = "since:quarter"
```

Common `report_time` examples:

- Every day at 09:00: `0 9 * * *`
- Every hour: `0 * * * *`
- Every 10 minutes: `*/10 * * * *`

Sampling examples:

- Every 7 seconds: `7s`
- Every 30 seconds: `30s`
- Every 2 minutes: `2m`
- Every 15 minutes: `15m`
- Every hour: `1h`

## Recommended Setup

### 1. Realtime report

Check near-realtime GPU activity every 10 minutes.

```toml
[sample]
interval = "1m"

[[report]]
name = "realtime"
report_time = "*/10 * * * *"
interval = "1m"
```

### 2. Daily Average report

This matches the current default-style daily setup.

```toml
[sample]
interval = "1m"

[[report]]
name = "daily"
report_time = "0 9 * * *"
interval = "1d"
```

### 3. Weekly report

Send one summary per week and aggregate the last 7 days.

```toml
[sample]
interval = "1m"

[[report]]
name = "weekly"
report_time = "0 9 * * 1"
interval = "7d"
```

### 4. Quarterly report

Fires on the first day of each quarter and averages the whole quarter that just ended, instead of a fixed number of days.

```toml
[sample]
interval = "1m"

[[report]]
name = "quarterly"
report_time = "0 9 1 1,4,7,10 *"
interval = "since:quarter"
```

### 5. Several reports at once

Reports are independent, so a realtime ping and a daily summary can run side by side. Sampling is shared: one sampler feeds every report.

```toml
[sample]
interval = "1m"

[[report]]
name = "realtime"
report_time = "*/10 * * * *"
interval = "1m"

[[report]]
name = "daily"
report_time = "0 9 * * *"
interval = "1d"
```

## Before Running Reports

A few things must be prepared by the user before `gpumanager` can collect data and send Slack notifications through a Slack incoming webhook:

- `nvidia-smi` must work on the server
- A valid Slack incoming webhook URL must be configured
- The CSV storage directory must be writable
- If you want automatic collection and reporting, the system-wide `systemd` timers must be enabled

Slack incoming webhook setup reference:

- https://docs.slack.dev/messaging/sending-messages-using-incoming-webhooks/

Quick manual verification:

```bash
nvidia-smi
gpumanager status
gpumanager test-sample
gpumanager test-report
```

## Automatic Scheduling

`gpumanager` does not start background collection on its own. To run sampling every minute and reporting on the configured cron-style schedule, install and enable the system timers.

Install the unit files:

```bash
gpumanager install-systemd
```

Then enable the timers:

```bash
sudo systemctl enable --now gpumanager-sample.timer gpumanager-report-daily.timer
```

`gpumanager install-systemd --enable-now` enables the sample timer and every configured report timer without typing the unit names.

Check timer status or reload installed timers:

```bash
gpumanager status
gpumanager reload
```

`gpumanager status` shows the next scheduled sample time in `sample.next_trigger` and the next run of every report in `reports[].next_trigger`. These values are read by parsing the `Trigger:` line from `systemctl status <timer unit>`.

Disable only sampling:

```bash
gpumanager disable-sample
```

Disable reporting (all reports, or one by name):

```bash
gpumanager disable-report                 # asks before disabling every report
gpumanager disable-report --report weekly
```

Disabling stops the timer but keeps the `[[report]]` block, so `gpumanager reload` will not bring it back. Re-enable it with `sudo systemctl enable --now gpumanager-report-weekly.timer`.

Sampling is disabled the same way:

```bash
gpumanager disable-sample
```

Every report reads the same CSV files, so disabling sampling leaves all of them with nothing to aggregate. Re-enable with `sudo systemctl enable --now gpumanager-sample.timer`.


## Sampling

Each sample creates a CSV file named like:

```text
2026-03-22T16-21-00.csv
```

Each CSV contains one row per GPU:

```csv
timestamp,gpu_index,gpu_uuid,gpu_name,util_gpu
2026-03-22T16:21:00+09:00,0,GPU-aaa,NVIDIA A100,35
2026-03-22T16:21:00+09:00,1,GPU-bbb,NVIDIA A100,2
```

## Report Format

Reports use the configured `general.server_name` as the bracketed name prefix, followed by the report name as `[server/report]`. A report named `default` (what an older single-`[report]` config becomes) shows only the server name. Average GPU utilization is rounded to two decimal places.

Example:

```text
[AICA_H100/weekly] 2025.09.06 16:49:32 KST
Window: since 2025-07-01 00:00
GPU 0: 31.38%
GPU 1: 29.39%
GPU 2: 31.57%
GPU 3: 56.36%
GPU 4: 61.25%
GPU 5: 61.52%
GPU 6: 59.88%
GPU 7: 63.93%
```

## systemd

`gpumanager install-systemd` installs system services into `/etc/systemd/system/`:

- `gpumanager-sample.service`
- `gpumanager-sample.timer`
- `gpumanager-report-<name>.service` and `gpumanager-report-<name>.timer`, one pair per `[[report]]` block

`add-report`, `remove-report`, `install-systemd` and `reload` reconcile the installed units with the config file: timers for newly added reports are enabled and started, units for reports that are no longer configured are disabled and deleted, and the single unnamed `gpumanager-report.{service,timer}` pair from older versions is replaced by `gpumanager-report-default.*`.


## Notes

- Each `[[report]]` entry needs a unique `name`; it is used as the systemd unit name
- `report_time` uses a 5-field cron string such as `0 9 * * *`
- `sample.interval` controls how often GPU utilization is sampled and saved
- `interval` controls the aggregation window shown as `Window: last ...` for durations, or `Window: since ...` for `since:` anchors
- `since:` windows resolve against the report's own `general.timezone`, and start at 00:00 on the boundary day
- Missing samples are ignored during aggregation
- The README content is used as the package long description, so this setup guide will also be visible on package index web pages after publishing
