We presented waiting time analysis, an automated technique to determine the cause of execution delays in parallel programs. We used several example applications to illustrate the value of WT analysis in performance diagnosis and modeling. We showed how WT analysis can be used to track down the causes of waiting time in parallel programs, and to suggest program modifications that improve performance. In this regard, it complements other techniques that measure, but do not explain, waiting time. We also showed how WT analysis can be used to derive models for waiting time as the composition of models for the source code that caused the waiting time.
Although WT analysis is based on execution traces provided by the Carnival framework, it does not require storage of full trace files, or access to an entire trace at one time. Its memory requirement is proportional to the source code length times the number of processors used. Our experience with this technique confirms that WT analysis compresses dynamic execution data by orders of magnitude, while maintaining our ability to study and understand a program's dynamic behavior.