A typical parallel performance analysis session begins with an attempt to evaluate overall performance using global metrics such as speedup or efficiency. If this evaluation suggests poor performance, various breakdowns of performance phenomena (e.g., procedure profiles, processor profiles) are used to help identify the cause. Eventually the focus moves to low-level performance phenomena, such as synchronization loss, where the programmer seeks to explain the observed phenomena in terms of the implementation. Additional experiments may be performed to gather more performance data, to confirm hypotheses about the cause of performance phenomena, and to evaluate the efficacy of changes to the application.
From this typical scenario, we can glean some requirements for a parallel performance tool. First, it should be possible to examine application performance at various abstract levels, from a global perspective based on speedup or efficiency, to a more detailed perspective based on code profiles. It must also be possible to link specific performance phenomena with source code; if a performance measurement suggests a bottleneck, the tool should identify the corresponding source code. In addition, there should be some support for relating effects (observed phenomena) to underlying causes. This facility is particularly important in cases where the symptom (e.g., waiting time) and the underlying cause (e.g., load imbalance in some prior loop) are far apart in time or are widely separated in the source code.
There are many tools that present performance data and metrics, giving the user the flexibility to determine what data to observe and how to view the data. For example, predicate profiling [5] measures and displays categories of inefficiency that are meaningful to the user, offering a high-level view of any performance problems. Paradyn [8] goes into greater detail, identifying individual performance bottlenecks in an execution and their location in the source code. The integrated Fortran D/Pablo tool [10] adds static information (e.g., data dependences, communication requirements) to the dynamic measurements, providing more context for the analysis. All of these tools assist the user in understanding performance phenomena, but do not automate the inference process linking an observation to a specific cause. Having identified a bottleneck (such as excessive waiting at a barrier), we would like to know why the bottleneck exists.
Carnival is a performance measurement and visualization tool for SPMD message-passing programs that automates the cause-and-effect inference process for waiting time. The tool uses detailed event traces to gather performance information, which it presents both as global summary statistics and as localized performance profiles, facilitating top-down performance analysis. The user interface presents performance information together with the source code, creating a link between the observed phenomena and the code. Most important, Carnival supports waiting time analysis, an automatic inference process that explains each source of waiting time, instead of simply identifying where it occurs.
We first describe Carnival, and then introduce waiting time analysis, and explain how we capture cause-and-effect relationships using this technique. We then present the user interface, and illustrate its use with examples.