Carnival is a performance measurement and analysis tool under development at the University of Rochester that meets the requirements outlined previously. In particular, it supports hierarchical abstraction in the presentation of performance data, maintains links between dynamic measurements and the source code, and automates cause-and-effect analysis of performance phenomena.
Our initial implementation of Carnival is targeted to data-parallel applications running on distributed memory machines using message passing for communication. It is implemented using the Sage++ package from Indiana [9] and Tcl/Tk [12]. The implementation runs on the IBM SP2 [3] and supports parallel code generated by the Forge APR HPF compiler [2].
The compilation of HPF programs in this environment is a two-step process. Initially, the HPF code is translated by the Forge xhpf preprocessor into SPMD Fortran 77 code augmented with calls to a runtime library, which performs data exchanges and synchronization operations. This F77 code is compiled by the IBM Fortran Compiler - xlf - generating executable code that runs in parallel.
Carnival exploits this two-step compilation process to insert instrumentation into the code. Each call to a library function generates an event that records the processor id, a scope identifier, the processing category (or processor state), and time of occurrence. These events are stored in a file for further processing.
During execution, the instrumented executable utilizes the UTE trace facility [13] (which is implemented on top of IBM's AIX trace facility [1]) to generate a trace file containing information about the execution. The level of detail in the UTE traces is enormous (and for the most part beyond our control), so the overhead of tracing in this environment is quite large, increasing execution time by a factor of 2-3 in many cases. It is not unusual to see 100 MB trace files on the SP2.
It is also possible to analyze hand-instrumented code using Carnival. We have implementations of the tracing library for networks of Sun workstations and the SGI Challenge that assume hand-instrumented PVM code. In those environments we generate only those events used in our analysis, resulting in many fewer trace events. As a result, tracing is much less intrusive (on the order of 5% execution time overhead) and produces much smaller trace files (by one to two orders of magnitude) than on the SP2.
Figure 1: Overview of Carnival
Figure 1 presents the various steps and files involved when using Carnival on the SP2. Rectangles represent files; arrows represent the steps that process files. The two main tasks performed by the tool are instrumentation and analysis.
During instrumentation, Carnival inserts calls to the trace library and collects static information that is used for post-processing. Both tasks are performed automatically by tools that use the code manipulation facilities provided by Sage++. The instrumentation's goal is to expose the semantic structure of the code and parallelization details that are not explicit in the original HPF code.
Carnival measures and presents performance profiles for several different categories of execution time including:
The result is a Tcl/Tk [12] interface that presents both static and dynamic data. This visualization interface is discussed in more detail in Section 5.