next up previous contents
Next: Execution Events Up: Observation Tools and Techniques Previous: Observation Tools and Techniques

Summary statistics

 

Summary statistics are high-level metrics that represent the average behavior of the program (or parts of it) across its execution.

Many summary statistics provide an overview of system performance as a whole. Speedup, efficiency, and serial fraction [Kumar and Gupta1994] are examples of metrics that can be used in evaluating the performance of parallel applications. However, these metrics only describe the overall utilization of parallel resources, and at most tell us whether our application is performing well.

Source code profiling (e.g., prof, gprof) provides summary statistics for the various application components (usually procedures) by accumulating the time spent in them across the execution. We can use this information to identify the most important portions of code. However, knowing which procedures are more expensive does not help in understanding the causes of poor performance, since it does not suggest why they have a high cost and what are the interactions among them.

Critical path analysis (CPA) [Yang and Miller1988] analyzes data collected during a program execution and identifies the execution path that dominates application performance. This information can be useful for improving the performance of the application, since improvements in portions of the dominant execution path are likely to enhance application performance. Although this technique gives us information about the dynamic behavior of the application, it does not provide information for explaining that behavior, since the differences in execution that define the dominant path are not considered. In SPMD programs, for example, it can highlight the whole program, failing to identify the application's bottlenecks.

Synthetic perturbation tuning (SPT) [Lyon et al. 1994] inserts artificial delays in the program and measures the impact on execution time. Using experimental design techniques, SPT identifies those delays that have the most impact on performance and therefore are potential bottlenecks in the application. This technique identifies the portions of code that dominate the execution, like CPA. Note that CPA and SPT use different strategies for acquiring performance data, but they have the same limitations for performance understanding, which are the lack of information about non-dominant execution paths and support for examining differences among execution paths. One caveat of SPT is that the number of experiments necessary for acquiring performance information increases exponentially with the number of inserted delays. Thus, for large programs, the cost of SPT can be prohibitive.

The integration of Fortran D with the Pablo toolkit [Adve et al. 1994] offers summary statistics augmented by static information (e.g., data dependences, communication requirements) associated with source code. The assumption here is that the availability of static information simplifies the task of understanding and analyzing an application's behavior. Nonetheless, the programmer is still fully responsible for the inference process.

Summary statistics are inadequate for performance understanding because they provide information at an abstract level, which hides relevant details about the application's behavior. They may be used for identification of performance phenomena, but their lack of detail prevents the generation of accurate explanations.


next up previous contents
Next: Execution Events Up: Observation Tools and Techniques Previous: Observation Tools and Techniques

Wagner Meira
Wed Feb 28 15:37:47 EST 1996