When porting applications from one environment to another, performance may be severely affected by the change in costs of basic operations. In particular, lock operations in hardware shared memory and in software distributed shared memory have very different costs. Waiting time analysis is a valuable technique for analyzing and understanding these costs, and the implications for the synchronization behavior of the application.
Water, from the Splash benchmark suite [11], is an example of an application whose performance changed drastically when it was ported from hardware shared memory to Treadmarks. On hardware shared memory, waiting time was not a significant part of the overall execution time and yet on Treadmarks, waiting time accounted for 60% of the execution time. Waiting time analysis showed that most of this waiting time was due solely to the high cost of lock operations, rather than an imbalance in the workload. Furthermore, the time spent waiting at barriers could be indirectly attributed to the high cost of lock operations that occurred earlier in the execution. Our analysis suggested the need to reduce the frequency of lock operations, and so we changed the granularity of synchronization by moving a lock outside a loop and using temporary variables within the loop. This change, which wasn't necessary on the hardware shared memory where the program was developed originally, but which became essential on software distributed shared memory, improved the execution time by a factor of 17.