Metadata-Version: 2.4
Name: nsys2prv
Version: 0.6.0rc2
Summary: Translate a NVIDIA Nsight System trace to a Paraver trace
License: GPL-3.0-only
License-File: LICENSE
Author: Marc Clascà
Author-email: marc.clasca@bsc.es
Requires-Python: >=3.10,<4.0
Classifier: License :: OSI Approved :: GNU General Public License v3 (GPLv3)
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Programming Language :: Python :: 3.15
Requires-Dist: pandas (>=2.2.2,<3.0.0)
Requires-Dist: sqlalchemy (>=2.0.0,<3.0.0)
Requires-Dist: toml-rs (>=0.3.4,<0.4.0)
Requires-Dist: tqdm
Project-URL: Homepage, https://pm.bsc.es/gitlab/beppp/nsys2prv
Project-URL: Repository, https://pm.bsc.es/gitlab/beppp/nsys2prv
Description-Content-Type: text/markdown

<center>
<img src="docs/h-color.png" alt="logo" width="40%">

# nsys2prv: Translate NVIDIA Nsight Systems traces to Paraver traces

![PyPI - Version](https://img.shields.io/pypi/v/nsys2prv)
![Gitlab Pipeline Status](https://gitlab.pm.bsc.es/beppp/nsys2prv/badges/main/pipeline.svg
)
![PyPI - Downloads](https://img.shields.io/pypi/dm/nsys2prv)
</center>

_nsys2prv_ is a Python package that parses and interprets the exported data of an NVIDIA Nsight Systems[^1] report and converts it to Paraver semantics in order to browse the trace with Paraver.  Paraver is a parallel performance analysis tool by the Performance Tools team at BSC, and is a parallel trace visualization system allowing for large scale trace execution analysis. Paraver can be obtained at [https://tools.bsc.es/downloads](https://tools.bsc.es/downloads).

The Nsight Systems traces should include minimum GPU kernel activity. Apart from this, _nsys2prv_ can also translate information about CUDA runtime, OpenACC constructs, MPI runtime, GPU metrics and NVTX regions. In addition to the different programming model semantics, one of the main features of _nsys2prv_ is its ability to **merge different Nsight Systems reports into one trace, allowing for easier analysis of multi-node, large scale parallel executions.**

## How it works
This tool relies on the export functionality of `nsys`. The data collection consists of a mix of the `nsys stats` predefined scripts, and a manual parsing of the _.sqlite_ exported format data.  The following figure summarizes the translation workflow:
![translation workflow](docs/translate-workflow.png)

More details on the workflow and the data parsing logic can be read on the [wiki pages](https://pm.bsc.es/gitlab/beppp/nsys2prv/-/wikis/Home).

## Installation

_nsys2prv_ is distributed as a PyPI package and thus can be installed with `pip`. The following requirements for the package to work will be installed automatically by `pip`:
- python >= 3.10
- pandas > 2.2.2
- sqlalchemy
- tqdm

Additionally, it requires an installation of NIDIA Nsight Systems in your _PATH_ to extract the data. Alternatively, you can set the NSYS_HOME environment variable.  It is required that the version of Nsight Systems is always greater than the one used to obtain the trace. **For translating, a minumum version of 24.6 is required**.

To install the package just use `pip` globally or create a vitual environment:
```bash
pip install --global nsys2prv
# or
python -m venv ./venv
source ./venv/bin/activate
pip install nsys2prv
```

## Basic usage
To translate a trace from Nsight Systems you need the _.nsys-rep_ report file that `nsys profile` outputs.  The basic usage of _Nsight Systems_ to get a trace is the following:

```bash
nsys profile --gpu-metrics-devices=cuda-visible -t cuda,nvtx -o ./llm_all --capture-range=nvtx --env-var=NSYS_NVTX_PROFILER_REGISTER_ONLY=0 --nvtx-capture=RANGE_NAME python TestLLAMA.py
```

In this example, we ask the profiler to only trace during the “RANGE_NAME” NVTX range, to get a trace for our phase of interest.

This should output the _llm\_all.nsys-rep_ file, that serves as input to `nsys2prv`.
```bash
nsys2prv -t type1,type2 llm_all.nsys-rep output-paraver-name
```

Currently, the translator needs that the user manually specifies the different programming models information to translate using the `--trace,-t` flag. By default it always extracts kernel execution information, so it is mandatory that the nsys report contains GPU activity. Future releases will automatically detect the information that is available in the report and make this flag optional.  The accepted value for the flag is a comma-separated list with any of the following categories:  
- `cuda_api_trace`: CUDA API calls
- `nvtx_pushpop_trace`: The developer defined NVTX Push/Pop regions
- `nvtx_startend_trace`: The developer defined NVTX Start/End regions
- `gpu_metrics`: The sampling events of hardware metrics for the GPUs
- `cpu_raw_counters`: CPU raw performance-counter samples attributed to the
  translated thread scheduled on the sample's CPU
- `mpi_event_trace`: The MPI calls
- `openacc`: The OpenACC constructs
- `graphs`: If your trace includes CUDA Graphs, include this flag to make sure that all Graph activity is translated, wether it is Node tracing or Graph tracing. If you are not sure, you can still include it and it will be disabled if there are no CUDA Graphs.
- `nccl`: If your application has NCCL activity, you can add this flag to include the NCCL payloads (e.g. communication size, reduction operation, root rank, etc.) in the NCCL kernel events.
  Payloads are attached only for an unambiguous one-operation/one-launch/one-kernel mapping.
  Multi-operation NCCL groups are left without GPU payloads, even when the operation and
  launch counts agree: calls may be fused, so positional matching is unsafe. Ambiguous
  or incomplete groups produce warnings; their kernel timelines and CPU NVTX payloads
  are retained. Nested groups are resolved at the outermost group end, independently
  for each task and host thread.
- `osrt`: OS runtime calls (except `pthread` ones)
- `pthread`: The POSIX threads library calls

Finally, the `output-paraver-name.prv` trace can be opened with Paraver and analyzed.

For multi-report translation, simply add the `-m` flag and add all the _.nsys-rep_ files:
```bash
nsys2prv -t type1,type2 -m source_rank0.nsys-rep [source_rank1.nsys-rep [source_rank2.nsys-rep ...]] output-paraver-name

```

## Configuration

See [docs/example_nvtx.toml](docs/example_nvtx.toml) for a complete small example
using the converter's real settings. All tables and keys are optional; no file
is loaded automatically. For example:

```bash
nsys2prv --config docs/example_nvtx.toml input.nsys-rep output
nsys2prv --config docs/example_nvtx.toml --no-sort --time-limit-start 0 input.nsys-rep output
nsys2prv -t nvtx --domain-mode exclude --domain 'solver.step' 'enabled=true' input.nsys-rep output
nsys2prv -t nvtx --domain-default 'payload.attach=false' --domain 'solver.step' 'payload={attach=true,float_strategy="round"}' input.nsys-rep output
nsys2prv --config docs/example_nvtx.toml --print-config
```

Application settings live in `[app]`, with keys matching long CLI options but
using underscores (for example `thread_name`, `force_sqlite`, `time_limit_start`).
`trace`, `label`, `filter_cuda_calls_include_name`, and
`filter_cuda_calls_exclude_name` are arrays of strings in TOML. CLI `-t` and name
filters retain comma-separated syntax, and repeated `-L` flags build the CLI
label list. An empty CLI trace/name-filter argument supplies an empty list.
Boolean flags also accept `--no-...` forms. With no `trace` setting the existing
implicit kernel translation is used, rather than failing on an omitted `-t`.

Application precedence is **built-ins < explicit TOML < explicit CLI**. For each
NVTX domain it is **built-ins < TOML defaults < TOML named override < CLI defaults
< CLI named override**. Consequently a general CLI option beats a named TOML
option. CLI named options beat CLI defaults regardless of argument order.
Nested tables merge recursively by key; scalars and lists replace inherited
values, including empty lists. Empty tables add nothing. Repeated assignments
within a source/scope use the last value at each key; `-L` retains its existing
append behavior. Omitted CLI switches do not overwrite TOML.

NVTX translation settings live under the case-sensitive `[NVTX]` scope.
`[NVTX.defaults]` supplies shared domain options; `[NVTX.domains."name"]`
supplies settings for one domain. `[NVTX] mode = "include"` (the default)
processes every discovered domain unless its settings have `enabled = false`.
`mode = "exclude"` processes only
domains explicitly set to `enabled = true`; other options alone never enable a
domain. Mode itself resolves built-ins < TOML < CLI. A CLI mode change does not
erase explicit selections. `enabled` is only valid in named domain tables, not in
`NVTX.defaults`. Names are arbitrary strings: quote dotted TOML table names
and pass the name separately to `--domain NAME KEY=TOML_VALUE`. Assignment values
use TOML syntax, including booleans and inline tables, not Python literals.

Domain options are `event_type` (positive event ID; automatic when omitted),
`payloads`, `payload.attach`, `payload.float_strategy` (`scale`, `round`, `string`,
or `fraction`), and the existing `project` setting. `project` remains reserved:
it does not change projection behavior in the current converter.
`NVTX.base_event_type` controls automatic event IDs. Default/neSmiK/NCCL keep
their existing built-in event IDs. Payload scaling retains the existing factor
of one million. Unsupported NCCL payloads are disabled for the translation based
on the SQLite schema, without modifying the supplied configuration.

Selection also excludes a domain's rows from NVTX thread discovery and trace
duration. Payload IDs follow the domain's effective event ID and categories.
Automatic IDs move to avoid occupied category/payload IDs; overlapping explicit
IDs are reported after input domains/categories are discovered. Give explicitly
numbered domains enough non-overlapping IDs for their categories and payloads.
CUDA name filters compose: an empty include array selects no calls, while an
empty exclude array excludes none.

`--print-config` prints JSON and exits without reading reports. Its
`NVTX.domains` entries are **configured names**, not discovered input domains. After a
translation, `<output>.pmf` stores `translation_metadata.effective_configuration`,
including `NVTX.resolved_domains` for every discovered NVTX domain (enabled or not),
effective event IDs, and runtime payload restrictions. The Python API separates
application and capability settings: `Configuration(...).app` contains only
application options, `.nvtx.resolve_domain(name)` and `.nvtx.setup_table(names)`
resolve NVTX domains, and `.cuda_api_calls.options` contains CUDA API filters.
Root `.resolve_domain(name)` and `.setup_table(names)` remain compatibility
aliases. Returned dictionaries are independent copies.

Capability configuration is owned by its semantic module: `NVTXConfiguration`
in `semantics/nvtx_pushpop_simple_semantic.py` defines domain defaults, validation,
CLI registration/normalization, selection and resolution; `CudaApiCallsConfiguration`
in `semantics/cuda_api_calls_semantic.py` owns CUDA API filter defaults, validation,
CLI handling and query options. Shared `config.py` handles application options,
file loading and composition; `config_utils.py` supplies capability-independent
merge, TOML assignment and validation helpers. TOML and diagnostic JSON both
group NVTX settings under `NVTX`; CUDA filters remain in `[app]`, despite their
capability-owned Python API. Existing `--domain-mode`, `--domain-default` and
`--domain` flags are unchanged.

Unknown keys, invalid values (including disabled domains), unreadable explicit
files, and invalid TOML are errors with source/path diagnostics. Python 3.11+
uses `tomllib`; Python 3.10 uses the existing `toml-rs` dependency.

**TOML migration:** rename `[domains]` to `[NVTX]`, `[domains.defaults]` to
`[NVTX.defaults]`, and `[domains.overrides."name"]` to `[NVTX.domains."name"]`.
Diagnostic JSON follows the same rename, and the PMF's previously top-level
`resolved_domains` is now `NVTX.resolved_domains`. There are no legacy aliases:
old tables produce an error explaining the new paths, rather than being ignored
or merged ambiguously. For the older experimental `capabilities.nvtx` layout,
move its `default` to `NVTX.mode`, shared options to `NVTX.defaults`, named
domains to `NVTX.domains.<name>`, and rename `translate` to `enabled`.
Move `base_event_type` to `NVTX` and application options into `app`.
The old example's `multi` becomes
`app.multi_report`, and `nsys.force_export` becomes `app.force_sqlite`; `source`
is removed (only Nsight input is supported). Legacy TOML layouts are rejected
rather than silently ignored. Existing CLI aliases and trace-category aliases
remain supported; invalid/unknown trace categories are now rejected explicitly.

## How to analyze your trace
A predefined set of Paraver Config Files (CFGs) can be found in the `cfgs` directory of this repository. If you open these files with Paraver, you will see predefined windows showing the information available in the trace. Some of them are more advanced and will show analysis views.

Config files are grouped in 5 different sets:

| Folder       | Description                                                                                                   |
|--------------|---------------------------------------------------------------------------------------------------------------|
| _basics_     | Configurations that show the available raw events in timelines                                                |
| _views_      | Compound views to show different behaviors and kernel semantics                                               |
| _analysis_   | More complex configurations aimed to summarize insight in the form of efficiency metrics from derived metrics |
| _hwcounters_ | Specific views to show hardware counters obtained with "gpu-metrics" flag                                     |
| _model_      | Profile configurations intended to support POP3's efficiency model computation                                |

**As a starting point**, the _views/cuda\_activities.cfg_ configuration file will show a timeline aggregating all CUDA-related activities: CUDA API calls, kernel execution at the GPU and memory operations.

For documentation about trace analysis and config files (CFGs) provided, please refer to the [wiki pages](https://pm.bsc.es/gitlab/beppp/nsys2prv/-/wikis/Home).

## Bug reporting and contribution
A list of the current bugs and features targeted can be seen in the GitLab repository. The project is still currently under initial development and is expecting a major code refactoring and changes in the command line interface (CLI).  As it is a tool to support and enable performance analysts' work, new use cases or petitions for other programming model information support are always welcomed. Please contact marc.clasca@bsc.es or beppp@bsc.es for any of those requests, recommendations, or contribution ideas.

A list of common issues and errors has been compiled on the wiki page [Troubleshooting](https://gitlab.pm.bsc.es/beppp/nsys2prv/-/wikis/Troubleshooting).  

If you are a regular user and would like to receive updates on new important releases and bug fixes, you can subscribe to the users mailing list sending an email to nsys2prv-users-join@bsc.es.


[^1]: https://developer.nvidia.com/nsight-systems

