Chapter 12. The rank Command
| Part III. Command-line Usage |

## Chapter 12. The rank Command

The rank command allows the user to specify a number of CDD files that have been created from the same DUT (presumably in a regression run -- but not necessarily) and rank them in order of coverage importance. This information can be useful in accomplishing the following tasks:

- Understand which CDD files (simulations) can be eliminated from regression due to their inability to add useful coverage information that selected CDD files already hit.

- Understand the best order to run simulations in full regression.

- It can help in creating a quick (confidence) regression which is meant to hit a good deal of the overall coverage points in a short period of time to verify that recent changes don't massively break a previously working DUT. This can be accomplished by selecting the first N simulation CDD files in the ranked list. The selection can easily be done by either using the simulation order, accumulated hit, hit percent or timestep column values as a means of selection.

## 12.1. Options

Table 12.1. Options to rank Command

Option | Description
--- | ---
-depth number | Specifies the minimum number of CDD files to hit each coverage point. The value of number should be a value of 1 or more. Default is 1.
-d directory | Directory to search for CDD files to include. This option is used in conjunction with the -ext option which specifies the file extension to use for determining which files in the directory are CDD files.
-ext extension | Used in conjunction with the -d option. If no -ext options are specified on the command-line, the default value of '.cdd' is used. Note that a period (.) should be specified.
-f filename | Name of file containing additional arguments to parse.
-h | Displays help information for the rank command.
-names-only | If specified, outputs only the needed CDD filenames that need to be run in the order they need to be run. If this option is not set, a report-style output is provided with additional information.
-o filename | Name of file to output ranking information to. Default is stdout.
-require-cdd filename | Name of CDD file to use as a required CDD file for ranking, regardless of whether it is necessary to achieve full coverage or not.
-require-list filename | Name of file containing list of CDD files which are required to be in the list of ranked CDDs to be run, regardless of whether they are necessary to achieve full coverage or not.
-v | Outputs verbose information during the run of the rank command. Useful for selection process understanding and command performance.
-weight-assert number | Specifies a relative weighting for assertion coverage used to rank non-unique coverage points.
-weight-comb number | Specifies a relative weighting for combinational logic coverage used to rank non-unique coverage points.
-weight-fsm number | Specifies a relative weighting for FSM state/state transition coverage used to rank non-unique coverage points.
-weight-line number | Specifies a relative weighting for line coverage used to rank non-unique coverage points.
-weight-memory number | Specifies a relative weighting for memory coverage used to rank non-unique coverage points.
-weight-toggle number | Specifies a relative weighting for toggle coverage used to rank non-unique coverage points.

## 12.2. Methods for specifying CDDs to rank

The rank command provides several mechanisms for specifying CDD files to rank. Feel free to use any combination of methods for providing this information to Covered as it makes sense to do in your environment. The following is the list of input methods:

- Specify each CDD file after the rank command options have been specified. This method is best when there are a relatively small number of CDD files to rank as the length of the command-line in a shell is typically limited to a certain number of characters.

- Create a file containing the list of CDD files to rank and pass this file to the rank command's -f option. Using this method, one can selectively specify a large number of CDD files. You may specify as many "-f filename" options as is necessary.

- Use shell commands to create a list of CDD files and pass them to Covered's standard input via command-line piping and the -f - command-line option (the - means use standard input). The following line is an example of this usage:

        ls foo*.cdd | covered rank -f -

  You may specify only one "-f -" option to the rank command.

- Use the -d directory and -ext extension options to tell Covered to retrieve all files found in the directory specified by directory that contain the file extension of extension (if the -ext option is not specified, the default extension of ".cdd" is used). The following is an example of this usage:

        covered rank -d /home/bubba/cdd_dir -ext .mycdd

  You may specify as many -d options on the command-line as is necessary; however, only one -ext option is allowed.

- Specify a single CDD file that MUST be included in the final ranking list via the -require-cdd option. You can specify as many -require-cdd options to the rank command as necessary.

- Create a file containing a list of CDDs that MUST be included in the final ranking list where each CDD filename is separated by a newline or whitespace. Pass this file to the -require-list option. You can specify as many -require-list options to the rank command as necessary.

## 12.3. Specifying metric weights

The rank command has several options that allow the user to weight any and all of the coverage metrics that Covered supplies. Weights are used to ascribe the importance of its metric in ranking the value of a CDD file.

Weight values are non-negative numbers whose values are relative to each other. A value of 0 effectively disables the associated weight from consideration in ranking. All other values are considered relative to each other. In other words, if the line weight is set to a value of 5 while the toggle weight is set to 1, line coverage is considered 5 times more important than toggle coverage in ranking.

By default, each metric is given an equal weight of 1. Adjust each of the metrics as necessary to potentially modify which CDD files are chosen and what order those CDD files are given in the ranked list of diagnostics to run.

## 12.4. Understanding the rank command output information

After the rank command has been executed, the output will either be displayed to standard output (if no -o option was specified) or it will be output to a file. If the -names_only option was specified, the output will be a list of CDD filenames (one per line) which are output in the order in which they should be run to achieve full coverage in the shortest amount of time. If the -names_only option was not specified, the output will be in the form of a table (similar to reports generated with the report command) which describes not only the order that CDD files should be run but also various statistical data and the names of CDD files which are considered redundant in terms of coverage information. The following example shows an example of this report output.

Example 12.1. Rank Output With No Required Files

                                               ::::::::::::::::::::::::::::::::::::::::::::::::::::
                                               ::                                                ::
                                               ::     Covered -- Simulation Ranked Run Order     ::
                                               ::                                                ::
                                               ::::::::::::::::::::::::::::::::::::::::::::::::::::

    * Reduced 3 CDD files down to 2 needed to maintain coverage ( 33% reduction,   1.5x improvement)
    * Reduced 9 timesteps down to 6 needed to maintain coverage ( 33% reduction,   1.5x improvement)

    -----------+-------------------------------------------+----------------------------------------------------------
               |                ACCUMULATIVE               |                             CDD
    Simulation |-------------------------------------------+----------------------------------------------------------
    Order      |        Hit /      Total     %   Timesteps |  R  Name               Hit /      Total     %   Timesteps
    -----------+-------------------------------------------+----------------------------------------------------------

             1            9           17   53%           3       rank1c.cdd           9           17   53%           3
             2           13           17   76%           6       rank1b.cdd           8           17   47%           3

    ---------------------------------------  The following CDD files add no additional coverage  ----------------------------------------------

             3           13           17   76%           9       rank1a.cdd           8           17   47%           3

After the header information, there is a summary of the ranking reduction (if one occurred) specified in both number of files and number of simulation timesteps. Example 12.2, “Rank Output With Required Files” shows what this information looks like when no file reduction has occurred.

After the summary information, there is the rank output table which displays both the individual CDD information and the accumulative coverage information. Each row of the table contains one CDD file where the first row contains the first CDD that should be run in regression for maximum coverage impact. Each successive row contains CDD files that add less and less coverage to the overall coverage gained. If there were CDD files that were found to not add any coverage information to the above CDD files, a line is drawn and these CDD files displayed below that line. These CDD files can be excluded from regression because they don't add any useful coverage information over-and-above the CDD files that are selected to be run in regression.

The first major column is the simulation order. This value increments by one for each successive row. If you would like to create a shorter running regression that is limited to a certain number of simulations, this row information can be useful for figuring out which simulations to include in these type of regressions.

The second major column is the accumulative coverage statistics which include the following:

- Hit

  The first row contains the number of coverage points that the first CDD file hit. All successive rows add the unique coverage points hit by the CDD file listed on that line to the above hit total.

  For example, in Example 12.1, “Rank Output With No Required Files” "rank1c.cdd" contains 9 hit coverage points (accumulative hit count = 9). The second highest ranked file, "rank1b.cdd", contains 8 hit coverage points but only 4 of those coverage points are different than the coverage points hit by "rank1c.cdd"; therefore, the accumulative coverage hit count is 13. The third coverage file, "rank1c.cdd", contains 8 coverage points but none of them add to the coverage points hit by the first two coverage files; therefore, it is considered unnecessary for regression.

- Total

  The "accumulative" total represents the total number of coverage points in each CDD file. Since this number must be the same for each CDD file, this value will be the same for each row.

- %

  This column represents the hit percentage and is a simple calculation of `hit / total` for each row.

- Timesteps

  This column represents the number of accumulated simulation timesteps. This value represents the number of timesteps that need to transpire to gain the amount of accumulated coverage. One of the goals of the ranking algorithm is to gain the most amount of coverage in the least amount of simulation time.

The third major column is the individual coverage information for the CDD file listed in the row which includes the following information:

- R

  This column represents whether the current CDD file was a required CDD file by the user (a file listed via the -require-list or -require-cdd options). If the file is a required CDD file, an asterisk (*) is listed in this column. If the file was not required, nothing will be listed in this column. Example 12.2, “Rank Output With Required Files” provides an example of what this output looks like.

- Name

  Filename of the CDD file that was selected for this row.

- Hit

  This column represents the number of coverage points hit by the CDD file.

- Total

  This column represents the total number of coverage points within the CDD file that could be hit.

- %

  This column represents the hit percentage (a calculation of `hit / total`.

- Timesteps

  This column represents the number of simulation timesteps that the CDD file took to run during simulation. This value does not necessarily represent the ending simulation time.

Example 12.2. Rank Output With Required Files

                                               ::::::::::::::::::::::::::::::::::::::::::::::::::::
                                               ::                                                ::
                                               ::     Covered -- Simulation Ranked Run Order     ::
                                               ::                                                ::
                                               ::::::::::::::::::::::::::::::::::::::::::::::::::::

    No reduction occurred

    -----------+-------------------------------------------+------------------------------------------------------------
               |                ACCUMULATIVE               |                              CDD
    Simulation |-------------------------------------------+------------------------------------------------------------
    Order      |        Hit /      Total     %   Timesteps |  R  Name                 Hit /      Total     %   Timesteps
    -----------+-------------------------------------------+------------------------------------------------------------

             1            9           17   53%           3       rank1.1c.cdd           9           17   53%           3
             2           13           17   76%           6    *  rank1.1b.cdd           8           17   47%           3
             3           13           17   76%           9    *  rank1.1a.cdd           8           17   47%           3

## 12.5. Description of ranking algorithm

This section describes the overall algorithm that is used to rank the specified CDD files. The algorithm is being presented in this document for the purposes of providing an explanation for how diagnostics are selected for ranking and how they are ranked. Knowing this information is not necessary for using the rank command or understanding its results.

The algorithm makes use of three different types of structures. The first structure is a bit-compressed structure where each bit corresponds to one coverage point in the associated CDD file. The purpose of this structure is to store a large amount of information as concisely and immediately available as possible. In addition to coverage point information, each structure also contains the following information:

- Name of CDD file that the structure is associated with (for reporting purposes only).

- The number of timesteps that occurred during the CDD simulation. This is used for calculating the average number of coverage points hit during a single clock period -- used for ranking selection.

- The total number of coverage points hit by the CDD. This is used for calculating the average number of coverage points hit during a single clock period -- used for ranking selection.

- The total number of coverage points that were hit by this CDD that were not hit by any other CDD listed in the rank command-line. This information is used to figure out which CDDs are absolutely necessary to be included in the ranked list.

- The current score associated with the CDD file. This value is updated for each selection iteration.

Each of these structures are maintained in an array that is iterated on during the ranking process.

The second structure is an array of 16-bit values where each value corresponds to a single coverage point. Its value is a sum of the number of unranked CDD files that have hit that coverage point. Each value is decremented by one when a CDD file is moved from the unranked list to the ranked list and that CDD file hits the given coverage point. If a coverage point value is a value of 1 and a CDD file has its corresponding coverage bit set, we know that the CDD is the only one that hits the coverage point.

The third structure is similar to the second and works similarly, except that it stores the number of CDD files in the ranked list that hit a particular coverage point.

The specified CDD files are ranked in a series of steps, some of which are reiterated for each CDD that is selected to be ranked. Here is the basic algorithm (in pseudo-code):

      calculate the size of the compressed coverage point structure

      foreach CDD in CDD_list {
        convert CDD to compressed coverage point structure
      }

      foreach CDD in CDD_list {
        if this CDD contains uniquely hit coverage points {
          move the CDD file into the ranked list
        }
      }

      foreach CDD in unranked list {
        foreach CDD in unranked list {
          calculate score (((total_num_cov_points_hit / num_timesteps) * 100) * metric_weight_value)
          if any unranked elements are less than the -depth option value and are hit by this CDD {
            set unique_found to true
          }
          if our score > highest score and a "unique" coverage point was found {
            set highest_score to our score
          }
        }
        move the CDD file indexed by highest_score from the unranked list to the ranked list
      }

      resort ranked list from most unique to least unique (if there is a tie, select the CDD that hit the most coverage
        points in the least amount of time)

Chapter 11. The report Command |  | Chapter 13. The exclude Command