There are many tools that allow observations at the level of individual execution events. These tools provide one or more visualizations of detailed performance data, leaving to the user the task of choosing displays that are most appropriate for the specific analysis at hand. We describe three representative tools below.
Upshot (and its successor, Nupshot) [Herrarte and Lusk1991] displays execution information as a time line for each processor. Each time line consists of processor states (e.g., sending, receiving, idle, processing) and their duration. We can understand the performance of parallel programs by browsing execution events, identifying interactions among processors, and analyzing the duration of specific operations. Although it is possible to understand an application's performance using this approach, the effort involved can be prohibitive for large applications, where the level of detail is overwhelming. Also, as the programmer focuses on individual performance phenomena, it is difficult for him to generalize his understanding across the execution, especially for applications that do not exhibit regular behavior.
The PPUTTs toolkit [LeBlanc et al. 1990] provides a graphical view of the program behavior based on a directed acyclic graph representation of processes and communication. Nodes in this acyclic graph represent synchronization or user-defined events, and arcs represent precedence relations between events. This representation can be manipulated in various ways providing multiple views of the program execution. Like Upshot, PPUTTs presents detailed information that can overwhelm the user, however it has some features that make performance understanding easier, such as the ability to select the events to be displayed, and the explicit presentation of precedence relations. Also, the PPUTTs capability of creating and integrating user-defined categories allows inspection of the dynamic behavior of data structures and other fine-grain components.
Paragraph [Heath and Etheridge1991] is another tool that provides multiple visualizations of execution events. These visualizations range from detailed dynamic information about the program (called communication and task displays) to graphical summaries of program performance (called utilization displays). These graphical summaries are visual representations of the summary statistics discussed in Section 2.1.1. As in Upshot and PPUTTS, analysis of detailed displays for performance understanding can be arduous and error-prone, because of the amount of data to be inspected, summarized, and explained. Also, the variety of displays only facilitates the detection of performance phenomena, leaving to the user not only the inference process that leads to explanations, but also the task of choosing among displays.
In general, performance understanding at the level of execution events is laborious, error-prone, and dependent on user knowledge of the parallel system. This difficulty is caused by characteristics of the cause-effect relations that explain the performance behavior of an application. For example, an effect may have more than one set of causes that explain it, and these sets can be completely different, as a consequence of the irregular behavior of an application. This variety of causes makes it difficult to determine cause-effect relations and understand the overall performance of the application.
Also, note that the parallel programmer is responsible for most of the performance understanding process, since none of these tools presents information about an application's execution in a way that explains its behavior.