/** 
* @file:   README
* CVS:     $Id$
* @author: Asim YarKhan yarkhan@icl.utk.edu
* @author: Heike McCraw mccraw@icl.utk.edu
* @defgroup papi_components Components
* @brief Component Specific Readme file: CUDA
*/

/** @page component_readme Component Readme 

@section Component Specific Information

Known problems and limitations in early release of metric support
-----------------------------------------------------------------

* If creation/destruction of EventSets is repeated dozens of times,
  the AMD portion of the software refuses further creation. Perhaps
  a limit is reached, or we are not performing some necessary 
  housekeeping. 
   
* Only sets of metrics and events that can be gathered in a single
  pass are supported.

* All metrics are returned as long long integers.


General information
-------------------

The PAPI ROCM component is a hardware performance counter measurement
technology for the AMD ROCM platform which provides access to the hardware
counters inside the GPU. The component is an adapter to the ROCm profiling
library (RPL, 'ROC-profiler') which is included in standard ROCM release.

How to install PAPI with the ROCM component?
-------------------------------------------- 

PAPI requires one environment variable: PAPI_ROCM_ROOT, but there are
four more needed at runtime for operation by AMD software. These added
environment variables are typically set as follows, AFTER
PAPI_ROCM_ROOT has been exported. An example is provided below,
setting PAPI_ROCM_ROOT to its default value:

export PAPI_ROCM_ROOT=/opt/rocm
export ROCP_METRICS=$PAPI_ROCM_ROOT/rocprofiler/lib/metrics.xml 
export ROCPROFILER_LOG=1
export HSA_VEN_AMD_AQLPROFILE_LOG=1
export AQLPROFILE_READ_API=1

The first of these, ROCP_METRICS, must point at a file containing the
descriptions of metrics. The standard location is shown above, the
final three are above are fixed settings.

For a standard installed system, these are the only environment
variables that need to be set, for both compile and runtime.

The above example works on ICL's Caffeine system.

Within PAPI_ROCM_ROOT, we expect the following standard directories:
PAPI_ROCM_ROOT/include
PAPI_ROCM_ROOT/include/hsa
PAPI_ROCM_ROOT/lib
PAPI_ROCM_ROOT/rocprofiler/lib
PAPI_ROCM_ROOT/rocprofiler/include 

After the exports shown above, PAPI must be configured and built.
When papi is installed, there will be a papi/src directory. Navigate
to that, and execute the following:

> ./configure --with-components="rocm"
> make 

If you are rebuilding PAPI, then before the configure step, execute
> make clobber

TESTING the component is installed: Still from papi/src:
> utils/papi_component_avail

If the component is functional, it will show that. Otherwise it will
report it is disabled, and provide a reason why.

You can see what events are provided by a working component as
follows: 
> utils/papi_native_avail | grep -i "rocm:::"



------------------------UNUSUAL INSTALLATIONS------------------------

System configurations can vary. Some systems use Spack, a package
manager, to automatically keep paths straight. Others (like our own
ICL Saturn System) require "module load" commands to provide some
services, e.g. 'module load rocm', and these may also set environment
variables and change the LD_LIBRARY_PATH search order.

Users may require the help of sysadmin personnel to navigate these
facilities and gain access to the correct libraries.

For the ROCM component to be operational, it must find the dynamic
libraries libhsa-runtime64.so and librocprofiler64.so. 

If these are not found (or are not functional) then the
component will be listed as "disabled" with a reason explaining the
problem. If libraries were not found, then they are not in the
expected places. The component can be configured to look for each of
these libraries in a specific place, and using an alternate name if
desired. Detailed instructions are contained in the Rules.rocm file.
They are technical, users may wish to enlist the help of a sysadmin.
