/*
* @file:   README
* CVS:     $Id$
* @author: Heike Jagode (jagode@icl.utk.edu)
* @author: Tony Castaldo (tonycastaldo@icl.utk.edu) & 
           Jiali Li (jli111@vols.utk.edu)
* @defgroup papi_components Components
* @brief Component Specific Readme file: pcp
*/


General Information
-------------------

+++ This component interfaces PAPI and the Performance Co-Pilot;
    for the Performance Metrics Domain agent: pmda=perfevent
...
+++ Purpose: Enable monitoring of P9 'nest' events via PCP without
             elevated privilege.
...
+++ P9 nest counters: any number of counters can be monitored
                      at any time. Meaning, there is no limitation
                      similar to P9 core counter monitoring, which
                      is limited to the number of counters (on P9
                      there are 6 counters: 4 programmable and
                      2 fixed (cycles, instructions completed).
...
+++ Repository: bitbucket/icl/papi/src/components/pcp

This PAPI PCP component has been developed and tested using PCP
version 3.12.2.

How to install PAPI with the PCP component?
-------------------------------------------
There is ONE required environment variable: PAPI_PCP_ROOT. This is
required for both compiling, and at runtime. 

An example that works on ICL's Caffeine system (at this writing):
export PAPI_PCP_ROOT=/usr

Within PAPI_PCP_ROOT, we expect the following standard directories:
PAPI_PCP_ROOT/include #OR# PAPI_PCP_ROOT/include/pcp
PAPI_PCP_ROOT/lib64

For a standard installed system, this is the only environment variable
required for both compile and runtime. 

System configurations can vary. Some systems use Spack, a package
manager, to automatically keep paths straight. Others (like our own
ICL Saturn System) require "module load" commands to provide some
services, e.g. 'module load pcp', and these may also set environment
variables and change the LD_LIBRARY_PATH search order.

Users may require the help of sysadmin personnel to navigate these
facilities and gain access to the correct libraries.

Configure PAPI with PCP enabled. We presume you have navigated to the
directory papi/src, AND that you have exported PAPI_PCP_ROOT. 
In the papi/src directory:
    % ./configure --with-components="pcp"
    % make 

TESTING the component is installed: Still from papi/src:
> utils/papi_component_avail

For the PCP component to be operational, it must find the dynamic
library libpcp.so.

If it is not found (or is not functional) then the component will be
listed as "disabled" with a reason explaining the problem. If library
was not found, then it is not in the expected place.  The component
can be configured to look for the library in a specific place, and
using an alternate name if desired. Detailed instructions are
contained in the Rules.pcp file.  They are technical, users may wish
to enlist the help of a sysadmin.

To find a list of PCP supported events:
    % utils/papi_native_avail | grep -i PCP

Special Notes: PCP interfaces with a daemon (a background program
running on the machines). If you use a batch system (like SLURM) so
that your programs run on a different machine (node) than your login
node, then it is possible one machine can have the daemon installed
and the other machine does not. This is the case for ICL developers on
the Peak and Summit machines; the login nodes are not executing the
PCP daemon, and the work nodes are. 

Thus to test the PCP component on Peak or Summit, you must
>jsrun --np 1 someprogram 

All below this line is code specific information for developers
---------------------------------------------------------------

This code was tested on both Saturn and ORNL peak (Power9; P9).
------------------------------------------------------------------

The code handles an arbitrary number of PCP events; the necessary
tables are realloc()'ed as needed to accomodate any number of
events.   
------------------------------------------------------------------

The code is caching; because the communications with the PCP
daemon can be costly. This increases the complexity of the code
and memory footprint, but we strive to read event names, numerical
IDs, descriptors and domain data only once from the daemon, and
only as needed to support the PAPI user.
------------------------------------------------------------------

The P9 nest events contain both counters and "dutycycle" events.
The dutycycle events return instantaneous double precision
floating point values (between 0.0 and 1.0) that cannot be
'reset'. A PAPI_get_event_info() on these events will return, in
PAPI_event_info_t, a 'timescope' of PAPI_TIMESCOPE_POINT. Counters
will have a PAPI_TIMESCOPE_SINCE_START.  
------------------------------------------------------------------

The P9 PCP is installed without any "help text" descriptions of
the counters. The Saturn version of PCP does have help text.
------------------------------------------------------------------

This software translates all 32 bit counters (int, unsigned int,
float) into corresponding 64 bit counters (long long int, unsigned
long long, double) for return to PAPI. e.g. a float = -1.23 will
become a double = -1.23; an unsigned int of 0xFFFFFFFF will become
an unsigned long long of 0x00000000FFFFFFFF; a signed int of
0xFFFFFFFF (-1) becomes a long long of 0xFFFFFFFFFFFFFFFF (-1).
------------------------------------------------------------------

PAPI : PCP differences, and resolutions.

PCP STRING EVENTS AND OTHER EVENTS:-------------------------------
PCP has events that return arbitrarily length strings. There is no
char* data type in PAPI, so we exclude such events from the list
of events we support. The only one we excluded on P9 was a string
containing the version number of the PCP code in use.

We also exclude PCP aggregate types and some other types that
cannot be reasonably represented by PAPI's numerical types.

PCP ARRAYS OF VALUES: --------------------------------------------
A single PCP event can return an array of values; for example on
the Saturn test system nearly all events return an array of 64
values: 1 per Core. Others return 4 values; one per socket. This
includes a naming system for each of those values; e.g. cpu0 to
cpu63, or socket0 to socket63, etc. We call these multi-valued
events.

However, PAPI is designed to return exactly one value per event.
Our approach here is to 'explode' the multi-valued events, into
one PAPI event per value, and make the event names unique by
appending the PCP index name to the PCP event name with a ':'. 

e.g. PCP_EVENT_NAME -> PCP_EVENT_NAME:cpu0, PCP_EVENT_NAME:cpu1,
... PCP_EVENT_NAME:cpu63.

Internally we translate these names to a single event request from
PCP, and handle all the array indexing as needed; so to PAPI they
look like distinct events (including for zeroing or resets). 

This was an arbitrary coding choice that allows flexibility for
the user; an alternative would have been making these array
entries like '?' event mask parameters or options. 

PCP EVENT MASKS & OPTIONS:----------------------------------------
PCP doesn't have these. It returns a single value, or an array of
several named values.

PCP ROUND TRIP OPTIMIZATION:--------------------------------------
This interface communicates with a daemon; and on some systems
that can take significant time. On Saturn, every communication
costs 8.2 ms, so for ~10,000 events, a communication for each
could cost 82 seconds at initialization. For this reason the code
looks somewhat complex as we use batch queries that can move much
information at once; like pmTraversePMNS, pmLookup and pmFetch
that all take arrays of events at a time, and cost only one round
trip penalty. 

------------------------------------------------------------------
CODE OPTIONS for linux-pcp.

Near line 12: // #define DEBUG
To enable PAPI style debug messages; we don't use many, but there
are five SUBDBG (Substrate) messages in the _pcp_ctl() routine.

Near line 42: #define AGENT_NAME "perfevent"
PCP will only return events (during a pmTraversePMNS) that begin
with this string. At this writing 'nest' events are always within
'perfevents', but 'perfevents' is an umbrella that contains 
other events that are not 'nest' events. 

Near line 194: #define COUNT_ROUTINES 1
Will enable a macro 'mRtnCnt' which will print on stderr the entry
of each PAPI interfacing routine the first time it is executed;
and when the component is shutdown a list of how often each
interface routine was executed. This is used to ensure the test
program exercised each interface routine at least once.

Near line 239: #define _prog_fprintf
change to 'if (1)' to enable printing (to stderr) of various
progress messages in the initialization stage.

Near line 240,241: #defines _time_fprintf, _time_gettimeofday.
change to 'if (1)' to enable printing (to stderr) of various
timings of the initilization operations, like reading names,
fetching values, etc. 


------------------------------------------------------------------

The testPCP.c code (bitbucket/icl/papi/src/components/pcp/tests)

This is straightforward application level code that reads through
every event, and tests every interface function of the component
at least once. First it finds the pcp component, then enumerates
all the events. It creates an event set, resets it, adds and
subtracts the event by name, gets its code, gets its info, and so
on. The final test is to add all the events to a single event set,
clean it up, destroy it, and execute a PAPI_shutdown.

It will print (to stdout) a CSV style text, with a heading,
detailing all the symbols, units, text description, time scope,
type and a sample value for for each event. 
------------------------------------------------------------------

