code coverage Tool

The code coverage tool provides software developers with a view of how much application code is exercised when a specific workload is applied to the application. To determine which code is used, the code coverage tool uses Profile-guided Optimization (PGO) options and optimizations. The major features of the code coverage tool are:

The information about using the code coverage tool is separated into the following sections:

The tool analyzes static profile information generated by the compiler, as well as dynamic profile information generated by running an instrumented form of the application binaries on the workload. The tool can generate the in HTML-formatted report and export data in both text-, and XML-formatted files. The reports can be further customized to show color-coded, annotated, source-code listings that distinguish between used and unused code.

The code coverage tool is available on IA-32, Intel® 64, and IA-64 architectures on Linux* and Windows*. The tool is available on IA-32 and Intel® 64 architectures on Mac OS* X.

You can use the tool in a number of ways to improve development efficiency, reduce defects, and increase application performance:

code coverage tool Requirements

To run the code coverage tool on an application, you must have following items:

See Understanding Profile-guided Optimization and Example of Profile-guided Optimization for general information on creating the files needed to run this tool.

Using the Tool

The tool uses the following syntax:

Tool Syntax

codecov [-codecov_option]

where -codecov_option is one or more optional parameters specifying the tool option passed to the tool. The available tool options are listed in code coverage tools Options (below). If you do not use any additional tool options, the tool will provide the top-level code coverage for the entire application.

See PGO Sample for information on using the tool with the supplied sample file.

In general, you must perform the following steps to use the code coverage tool:

  1. Compile the application using -prof-genx (Linux) or /Qprof-genx (Windows).
    This step generates an instrumented executable and a corresponding static profile information (pgopti.spi) file.

  2. Run the instrumented application.
    This step creates the dynamic profile information (.dyn) file Each time you run the instrumented application, the compiler generates a unique .dyn file either in the current directory or the directory specified in by prof_dir.

  3. Use the profmerge tool to merge all the .dyn files into one .dpi (pgopti.dpi) file.
    This step consolidates results from all runs and represents the total profile information for the application, generates an optimized binary, and creates the dpi file needed by the code coverage tool.

    You can use the profmerge tool to merge the .dyn files into a .dpi file without recompiling the application. The profmerge tool can also merge multiple .dpi files into one .dpi file using the profmerge -a option. Select the name of the output .dpi file using the profmerge -prof_dpi option.

Caution

The profmerge tool merges all .dyn files that exist in the given directory. Make sure unrelated .dyn files, which may remain from unrelated runs, are not present. Otherwise, the profile information will be skewed with invalid profile data, which can result in misleading coverage information and adverse performance of the optimized code.

  1. Run the code coverage tool. (The valid syntax and tool options are shown below.)
    This step creates a report or exported data as specified. If no other options are specified, the code coverage tool creates a single HTML file (CODE_COVERAGE.HTML) and a sub-directory (CodeCoverage) in the current directory. Open the file in a web browser to view the reports.

Note

Windows* only: Unlike the compiler options, which are preceded by forward slash ("/"), the tool options are preceded by a hyphen ("-").

The code coverage tool allows you to name the project and specify paths to specific, necessary files. The following example demonstrates how to name a project and specify .dpi and .spi files to use:

Example: specify .dpi and .spi files

codecov -prj myProject -spi pgopti.spi -dpi pgopti.dpi

The tool can add a contact name and generate an email link for that contact at the bottom of each HTML page. This provide a way to send an electronic message to the named contact. The following example demonstrates how to add specify a contact and the email links:

Example: add contact information

codecov -prj myProject -mname JoeSmith -maddr js@company.com

This following example demonstrates how to use the tool to specify the project name, specify the dynamic profile information file, and specify the output format and file name.

Example: export data to text

codecov -prj test1 -dpi test1.dpi -txtbcvrg test1_bcvrg.txt

code coverage tool Options

The tool uses the options listed in the table:

Option

Default

Description

-bcolor color

#FFFF99

Specifies the HTML color name for code in the uncovered blocks.

-beginblkdsbl string

 

Specifies the comment that marks the beginning of the code fragment to be ignored by the coverage tool.

-ccolor color

#FFFFFF

Specifies the HTML color name or code of the covered code.

-comp file

 

Specifies the file name that contains the list of files being (or not) displayed.

-counts

 

Generates dynamic execution counts.

-demang

 

Demangles both function names and their arguments.

-dpi file

pgopti.dpi

Specifies the file name of the dynamic profile information file (.dpi).

-endblkdsbl string

 

Specifies the comment that marks the end of the code fragment to be ignored by the coverage tool.

-fcolor color

#FFCCCC

Specifies the HTML color name for code of the uncovered functions.

-help, -h

 

Prints tool option descriptions.

-icolor color

#FFFFFF

Specifies the HTML color name or code of the information lines, such as basic-block markers and dynamic counts.

-maddr string

Nobody

Sets the email address of the web-page owner

-mname string

Nobody

Sets the name of the web-page owner.

-nopartial

 

Treats partially covered code as fully covered code.

-nopmeter

 

Turns off the progress meter. The meter is enabled by default.

-onelinedsbl string

 

Specifies the comment that marks individual lines of code or the whole functions to be ignored by the coverage tool.

-pcolor color

#FAFAD2

Specifies the HTML color name or code of the partially covered code.

-prj string

 

Sets the project name.

-ref

 

Finds the differential coverage with respect to ref_dpi_file.

-spi file

pgopti.spi

Specifies the file name of the static profile information file (.spi).

-txtbcvrg file

 

Export block-coverage for covered functions as text format. The file parameter must be in the form of a valid file name.

-txtbcvrgfull file

 

Export block-coverage for entire application in text and HTML formats. The file parameter must be in the form of a valid file name.

-txtdcg file

 

Generates the dynamic call-graph information in text format. The file parameter must be in the form of a valid file name.

-txtfcvrg file

 

Export function coverage for covered function in text format. The file parameter must by in the form of a valid file name.

-ucolor color

#FFFFFF

Specifies the HTML color name or code of the unknown code.

-xcolor color

#90EE90

Specifies the HTML color of the code ignored.

-xmlbcvrg file

 

Export the block-coverage for the covered function in XML format. The file parameter must by in the form of a valid file name.

-xmlbcvrgfull file

 

Export function coverage for entire application in XML format in addition to HTML output. The file parameter must be in the form of a valid file name.

-xmlfcvrg file

 

Export function coverage for covered function in XML format. The file parameter must be in the form of a valid file name.

Visually Presenting Code Coverage for an Application

Based on the profile information collected from running the instrumented binaries when testing an application, the Intel® compiler will create HTML-formatted reports using the code coverage tool. These reports indicate portions of the source code that were or were not exercised by the tests. When applied to the profile of the performance workloads, the code coverage information shows how well the training workload covers the application's critical code. High coverage of performance-critical modules is essential to taking full advantage of the profile-guided optimizations.

The code coverage tool can create two levels of coverage:

Top Level Coverage

The top-level coverage reports the overall code coverage of the modules that were selected. The following options are provided:

By default, the code coverage tool generates a single HTML file (CODE_COVERAGE.HTML) and a subdirectory (CodeCoverage) in the current directory. The HTML file defines a frameset to display all of the other generated reports. Open the HTML file in a web-browser. The tool places all other generated report files in a CodeCoverage subdirectory.

If you choose to generate the html-formatted version of the report, you can view coverage source of that particular module directly from a browser. The following figure shows the top-level coverage report.

The coverage tool creates a frame set that allows quick browsing through the code to identify uncovered code. The top frame displays the list of uncovered functions while the bottom frame displays the list of covered functions. For uncovered functions, the total number of basic blocks of each function is also displayed. For covered functions, both the total number of blocks and the number of covered blocks as well as their ratio (that is, the coverage rate) are displayed.

For example, 66.67(4/6) indicates that four out of the six blocks of the corresponding function were covered. The block coverage rate of that function is thus 66.67%. These lists can be sorted based on the coverage rate, number of blocks, or function names.  Function names are linked to the position in source view where the function body starts. So, just by one click, you can see the least-covered function in the list and by another click the browser displays the body of the function. You can scroll down in the source view and browse through the function body.

Individual Module Source View

Within the individual module source views, the tool provides the list of uncovered functions as well as the list of covered functions. The lists are reported in two distinct frames that provide easy navigation of the source code. The lists can be sorted based on:

Setting the Coloring Scheme for the Code Coverage

The tool provides a visible coloring distinction of the following coverage categories: covered code, uncovered basic blocks, uncovered functions, partially covered code, and unknown code. The default colors that the tool uses for presenting the coverage information are shown in the tables that follows:

Category

Default

Description

Covered code

#FFFFFF

Indicates code was exercised by the tests. You can override the default color with the -ccolor tool option.

Uncovered basic block

#FFFF99

Indicates the basic blocks that were not exercised by any of the tests. However, these blocks were within functions that were executed during the tests. You can override the default color with the -bcolor tool option.

Uncovered function

#FFCCCC

Indicates functions that were never called during the tests. You can override the default color with the -fcolor tool option.

Partially covered code

#FAFAD2

Indicates that more than one basic block was generated for the code at this position. Some of the blocks were covered and some were not. You can override the default color with the -pcolor tool option.

Ignored code

#90EE90

Indicates code that was specifically marked to be ignored. You can override this default color using the -xcolor tool option.

Information lines

#FFFFFF

Indicates basic-block markers and dynamic counts. You can override the default color with the -icolor tool option.

Unknown

#FFFFFF

Indicates that no code was generated for this source line. Most probably, the source at this position is a comment, a header-file inclusion, or a variable declaration. You can override the default color with the -ucolor tool option.

The default colors can be customized to be any valid HTML color name or hexadecimal value using the options mentioned for each coverage category in the table above.

For code coverage colored presentation, the coverage tool uses the following heuristic: source characters are scanned until reaching a position in the source that is indicated by the profile information as the beginning of a basic block. If the profile information for that basic block indicates that a coverage category changes, then the tool changes the color corresponding to the coverage condition of that portion of the code, and the coverage tool inserts the appropriate color change in the HTML-formatted report files.

Note

You need to interpret the colors in the context of the code. For instance, comment lines that follow a basic block that was never executed would be colored in the same color as the uncovered blocks.

Dynamic Counters

The coverage tool can be configured to generate the information about the dynamic execution counts. This ability can display the dynamic execution count of each basic block of the application and is useful for both coverage and performance tuning.

The custom configuration requires using the -counts option. The counts information is displayed under the code after a "^" sign precisely under the source position where the corresponding basic block begins.

If more than one basic block is generated for the code at a source position (for example, for macros), then the total number of such blocks and the number of the blocks that were executed are also displayed in front of the execution count. For example, line 11 in the code is an IF statement:

Example

11   IF ((N .EQ. 1).OR. (N .EQ. 0))

     ^ 10 (1/2)

12      PRINT N

        ^ 7

The coverage lines under code lines 11 and 12  contain the following information:

In certain situations, it may be desirable to consider all the blocks generated for a single source position as one entity.  In such cases, it is necessary to assume that all blocks generated for one source position are covered when at least one of the blocks is covered. This assumption can be configured with the -nopartial option. When this option is specified, decision coverage is disabled, and the related statistics are adjusted accordingly. The code lines 11 and 12  indicate that the print statement in line 12 was covered. However, only one of the conditions in line 11 was ever true. With the -nopartial option, the tool treats the partially covered code (like the code on line 11) as covered.

Differential Coverage

Using the code coverage tool, you can compare the profiles from two runs of an application: a reference run and a new run identifying the code that is covered by the new run but not covered by the reference run. Use this feature to find the portion of the applications code that is not covered by the applications tests but is executed when the application is run by a customer. It can also be used to find the incremental coverage impact of newly added tests to an applications test space.

Generating Reference Data

Create the dynamic profile information for the reference data, which can be used in differential coverage reporting later, by using the -ref option. The following command demonstrate a typical command for generating the reference data:

Example: generating reference data

codecov -prj Project_Name -dpi customer.dpi -ref appTests.dpi

The coverage statistics of a differential-coverage run shows the percentage of the code exercised on a new run but missed in the reference run. In such cases, the tool shows only the modules that included the code that was not covered. Keep this in mind when viewing the coloring scheme in the source views.

The code that has the same coverage property (covered or not covered) on both runs is considered as covered code. Otherwise, if the new run indicates that the code was executed while in the reference run the code was not executed, then the code is treated as uncovered. On the other hand, if the code is covered in the reference run but not covered in the new run, the differential-coverage source view shows the code as covered.

Running Differential Coverage

To run the code coverage tool for differential coverage, you must have the application sources, the .spi file, and the .dpi file, as described in the code coverage tool Requirements section (above).

Once the required files are available, enter a command similar to the following begin the process of differential coverage analysis:

Example

codecov -prj Project_Name -spi pgopti.spi -dpi pgopti.dpi

Specify the .dpi and .spi files using the -spi and -dpi options.

Excluding Code from Coverage Analysis

The code coverage tool allows you to exclude portions of your code from coverage analysis. This ability can be useful during development; for example, certain portions of code might include functions used for debugging only. The test case should not include tests for functionality that will unavailable in the final application.

Another example of code that can be excluded is code that might be designed to deal with internal errors unlikely to occur in the application. In such cases, not having a test case lack of a test case is preferred. You might want to ignore infeasible (dead) code in the coverage analysis. The code coverage tool provides several options for marking portions of the code infeasible (dead) and ignoring the code at the file level, function level, line level, and arbitrary code boundaries indicated by user-specific comments. The following sections explain how to exclude code at different levels.

Including and Excluding Coverage at the File Level

The code coverage tool provides the ability to selectively include or exclude files for analysis. Create a component file and add the appropriate string values that indicate the file and directory name for code you want included or excluded.  Pass the file name as a parameter of the -comp option. The following example shows the general command:

Example: specifying a component file

codecov -comp file

where file is the name of a text file containing strings that ask as file and directory name masks for including and excluding file-level analysis. For example, assume that the following:

Once you have a component file, enter a command similar to the following:

Example

codecov -comp myComp.txt

In this example, individual files name including the string "source" (like source1.f and source2.f) and files in directories where the name contains the string "source" (like source/file1.f and source2\file2.f ) are include in the analysis.

Excluding files is done in the same way; however, the string must have a tilde (~) prefix. The inclusion and exclusion can be specified in the same component file.

For example, assume you want to analyze all individual files or files contained in a directory where the name included the string "source", and you wanted to exclude all individual file and files contained in directories where the name included  the string "skip". You would add content similar to the following to the component file (myComp.txt) and pass it to the -comp option:

Example: inclusion and exclusion strings

source

~skip

Entering the codecov -comp myComp.txt command with both instructions in the component file will instruct the tool to include individual files where the name contains "source" (like source1.f and source2.f) and directories where the name contains "source" (like source/file1.f and source2\file2.f ), and exclude any individual files where the name contains "skip" (like skipthis1.f and skipthis2.f) or directories where the name contains "skip" (like skipthese1\debug1.f and skipthese2\debug2.f).

Excluding Coverage at the Line and Function Level

You can mark individual lines for exclusion my passing string values to the -onelinedsbl option. For example, assume that you have some code similar to the following:

Sample code

print*, "ERROR: n  = ", n  ! NO_COVER

print*, "       n2 = ", n2  ! INF IA-32 architecture

 If you wanted to exclude all functions marked with the comments NO_COVER or INF IA-32 architecture, you would enter a command similar to the following.

Example

codecov -onelinedsbl NO_COVER -onelinedsbl "INF IA-32 architecture"

You can specify multiple exclusion strings simultaneously, and you can specify any string values for the markers; however, you must remember the following guidelines when using this option:

An entire function can be excluded from coverage analysis using the same methods. For example, the following function will be ignored from the coverage analysis when you issue example command shown above.

Sample code

subroutine dumpInfo (n)

integer n ! NO_COVER

...

end subroutine

Additionally, you can use the code coverage tool to color the infeasible code with any valid HTML color code by combining the -onelinedsbl and -xcolor options. The following example commands demonstrate the combination:

Example: combining tool options

codecov -onelinedsbl INF -xcolor lightgreen

codecov -onelinedsbl INF -xcolor #CCFFCC

Excluding Code by Defining Arbitrary Boundaries

The code coverage tool provides the ability to arbitrarily exclude code from coverage analysis. This feature is most useful where the excluded code either occur inside of a function or spans several functions.

Use the -beginblkdsbl and -endblkdsbl options to mark the beginning and end, respectively, of any arbitrarily defined boundary to exclude code from analysis.  Remember the following guidelines when using these options:

For example assume that you have the following code:

Sample code

integer n, n2

n = 123

n2 = n*n

if (n2 .lt. 0) then

! /* BINF */

  print*, "ERROR: n  = ", n

  print*, "       n2 = ", n2

! // EINF

else if (n2 .eq. 0) then

  print*, "zero: n = ", n, " n2 = ", n2

else

  print*, "positive: n = ", n, " n2 = ", n2

endif

end

The following example commands demonstrate how to use the -beginblkdsbl option to mark the beginning and the -endblkdsbl option to mark the end of code to exclude from the sample shown above.

Example: arbitrary code marker commands

codecov -xcolor #ccFFCC -beginblkdsbl BINF -endblkdsbl EINF

codecov -xcolor #ccFFCC -beginblkdsbl "BEGIN_INF" -endblkdsbl "END_INF"

Notice that you can combine these options in combination with the -xcolor option.

Exporting Coverage Data

The code coverage tool provides specific options to extract coverage data from the dynamic profile information (.dpi files) that result from running instrumented application binaries under various workloads. The tool can export the coverage data in various formats for post-processing and direct loading into databases: the default HTML, text, and XML. You can choose to export data at the function and basic block levels.

There are two basic methods for exporting the data: quick export and combined export. Each method has associated options supported by the tool

These export methods provide the means to quickly extend the code coverage reporting capabilities by supplying consistently formatted output from the code coverage tool. You can extend these by creating additional reporting tools on top of these report files.

Quick Export

The profile of covered functions of an application can be exported quickly using the -xmlfcvrg, -txtfcvrg, -xmlbcvrg, and -txtbcvrg options. When using any of these options, specify the output file that will contain the coverage report. For example, enter a command similar to the following to generate a report of covered functions in XML formatted output:

Example: quick export of function data

codecov -prj test1 -dpi test1.dpi -xmlfcvrg test1_fcvrg.xml

The resulting report will show how many times each function was executed and the total number of blocks of each function together with the number of covered blocks and the block coverage of each function. The following example shows some of the content of a typical XML report.

XML-formatted report example

<PROJECT name = "test1">

   <MODULE name = "D:\SAMPLE.F">

      <FUNCTION name="f0" freq="2">

         <BLOCKS total="6" covered="5" coverage="83.33%"></BLOCKS>

      </FUNCTION>

      ...

   </MODULE>

   <MODULE name = "D:\SAMPLE2.F">

      ...

   </MODULE>

</PROJECT>

In the above example, we note that function f0, which is defined in file sample.f, has been executed twice. It has a total number of six basic blocks, five of which are executed, resulting in an 83.33% basic block coverage.

You can also export the data in text format using the -txtfcvrg option. The generated text report, using this option, for the above example would be similar to the following example:

Text-formatted report example

Covered Functions in File: "D:\SAMPLE.F"

"f0"    2       6       5        83.33

"f1"    1       6       4        66.67

"f2"    1       6       3        50.00

...

In the text formatted version of the report, the each line of the report should be read in the following manner:

Column 1

Column 2

Column 3

Column 4

Column 5

function name

execution frequency

line number of the start of the function definition

column number of the start of the function definition

percentage of basic-block coverage of the function

Additionally, the tool supports exporting the block level coverage data using the -xmlbcvrg option. For example, enter a command similar to the following to generate a report of covered blocks in XML formatted output:

Example: quick export of block data to XML

codecov -prj test1 -dpi test1.dpi -xmlbcvrg test1_bcvrg.xml

The example command shown above would generate XML-formatted results similar to the following:

XML-formatted report example

<PROJECT name = "test1">

  <MODULE name = "D:\SAMPLE.cF90">

    <FUNCTION name="f0" freq="2">

    ...

      <BLOCK line="11" col="2">

        <INSTANCE id="1" freq="1"> </INSTANCE>

      </BLOCK>

      <BLOCK line="12" col="3">

        <INSTANCE id="1" freq="2"> </INSTANCE>

        <INSTANCE id="2" freq="1"> </INSTANCE>

      </BLOCK>

In the sample report, notice that one basic block is generated for the code in function f0 at the line 11, column 2 of the file sample.f90. This particular block has been executed only once. Also notice that there are two basic blocks generated for the code that starts at line 12, column 3 of file. One of these blocks, which has id = 1, has been executed two times, while the other block has been executed only once. A similar report in text format can be generated through the -txtbcvrg option.

Combined Exports

The code coverage tool has also the capability of exporting coverage data in the default HTML format while simultaneously generating the text- and XML-formatted reports.

Use the -xmlbcvrgfull and -txtbcvrgfull options to generate reports in all supported formatted in a single run. These options export the basic-block level coverage data while simultaneously generating the HTML reports. These options generate more complete reports since they include analysis on functions that were not executed at all. However, exporting the coverage data using these options requires access to application source files and take much longer to run.

Dynamic Call Graphs

Using the -txtdcg option the tool can provide detailed information about the dynamic call graphs in an application. Specify an output file for the dynamic call-graph report. The resulting call graph report contains information about the percentage of static and dynamic calls (direct, indirect, and virtual) at the application, module, and function levels.