Metadata-Version: 2.4
Name: rdmatop
Version: 0.1.30
Summary: RDMA terminal monitor and PyTorch profiler counter tracks
License-Expression: Apache-2.0
Project-URL: Homepage, https://github.com/uccl-project/rdmatop
Project-URL: Issues, https://github.com/uccl-project/rdmatop/issues
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Rust
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: torch==2.9.0
Dynamic: license-file
Dynamic: requires-dist

# rdmatop

[![Crates.io](https://img.shields.io/crates/v/rdmatop)](https://crates.io/crates/rdmatop)
[![License](https://img.shields.io/crates/l/rdmatop)](LICENSE)

`htop`, but for RDMA traffic — a real-time TUI monitor for RDMA network interfaces.

<p align="center">
  <img src="https://raw.githubusercontent.com/uccl-project/rdmatop/main/images/rdmatop.gif" alt="rdmatop" width="800">
</p>

Monitors per-device throughput (Gbps, packets/s, drops), RDMA read/write counters,
retransmits, health events, and shows which processes are using each RDMA device —
all via RDMA netlink, the same interface used by [rdma statistic](https://github.com/iproute2/iproute2/blob/main/rdma/stat.c).

## Blogs

- [rdmatop: Cross-Provider htop for RDMA Traffic](https://uccl-project.github.io/posts/rdma-monitoring/) (2026-06-15)
- [NVSHMEM Multi-NIC Support with AWS EFA](https://www.pythonsheets.com/notes/appendix/nvshmem-multi-nic.html) (2026-03-27)

## Requirements

- **Linux** (netlink-based — macOS/Windows are not supported)
- RDMA-capable NICs (e.g., Mellanox/NVIDIA ConnectX, AWS EFA)

## Installation

### Ubuntu (PPA)

On Ubuntu 22.04 (jammy), 24.04 (noble), or 26.04 (resolute) — amd64 and arm64:

```bash
sudo add-apt-repository ppa:crazyguitar/rdmatop
sudo apt update
sudo apt install rdmatop
```

### Cargo

```bash
cargo install rdmatop
```

### From source

```bash
make         # cargo build
make install # cargo install
```

## Usage

```bash
rdmatop
```

## Perfetto recording

Press `r` in the TUI to start recording and `r` again to stop. rdmatop captures
every device's tx/rx Gbps and packets/s per interval and writes a Chrome-JSON
trace (`rdmatop-<unix_timestamp>.json`, in the current directory) you can drag into [ui.perfetto.dev](https://ui.perfetto.dev)
— each device/port becomes its own set of counter tracks. Timestamps are relative
to when you pressed `r`, so the trace spans exactly your record window.

<p align="center">
  <img src="https://raw.githubusercontent.com/uccl-project/rdmatop/main/images/perfetto.png" alt="rdmatop Perfetto recording" width="800">
</p>

## PyTorch profiler

rdmatop can run inside the training process as a Kineto child profiler, so
RDMA counter tracks land in the same trace `torch.profiler` writes:

```python
import torch
from torch.profiler import ProfilerActivity, profile

import rdmatop.kineto

rdmatop.kineto.enable()
with profile(activities=[ProfilerActivity.CPU, ProfilerActivity.CUDA]) as prof:
    train_step()
prof.export_chrome_trace("trace.json")
```

The shim links against the installed torch (>= 2.9), so it is built from a
checkout with `cargo` and a C++ compiler on `PATH`. PyTorch selects the required
C++ standard (C++17 or C++20, depending on its version). Rebuild the shim after
changing PyTorch versions:

```bash
pip install "setuptools>=64" "torch>=2.9"
pip install --no-build-isolation -e ./python
```

On older Kineto versions without native counters, `enable()` wraps
`torch.profiler.profile.export_chrome_trace()` to convert rdmatop's marked
events into counter tracks. This also supports gzip exports and
`tensorboard_trace_handler`; raw Kineto exports retain zero-duration events.
Newer versions emit native counters and need no export wrapper.

## Examples

Use `rdmatop` to monitor RDMA traffic while running GPU
communication benchmarks:

- [PyTorch](examples/pytorch/) — intranode NVLink/XGMI traffic
- [IB Perftest](examples/ib/) — two-node `ib_write_bw` benchmark
- [UCX Perftest](examples/ucx/) — two-node `ucx_perftest` bandwidth / latency
- [NCCL](examples/nccl/) — collective communication
- [NIXL](examples/nixl/) — point-to-point KV cache transfer
- [NVSHMEM](examples/nvshmem/) — one-sided GPU communication
- [PPLX Kernels](examples/pplx/) — MoE all-to-all dispatch/combine
- [UCCL](examples/uccl/) — DeepEP-compatible expert-parallel dispatch/combine
- [RDMA Statistics](examples/rdma/) — shell-based RDMA stats
- [Kubernetes](examples/kubernetes/) — DaemonSet deployment for Kubernetes

## How It Works

1. **Device enumeration** — `RDMA_NLDEV_CMD_GET` via netlink to discover all RDMA devices
2. **HW counters** — `RDMA_NLDEV_CMD_STAT_GET` per device/port, same as `rdma statistic show`
3. **Process detection** — `RDMA_NLDEV_CMD_RES_QP_GET` to map QPs → PIDs, enriched with `/proc` data
4. **Throughput** — Two snapshots per interval, delta / elapsed for rates

## Contributing

See [CONTRIBUTING.md](CONTRIBUTING.md) for build/test instructions,
design ground rules, and how to submit changes.

## License

Apache-2.0
