In this talk, I will present three main contributions to cluster scheduling and resource management. I will first show how we can boost system utilization to as high as 95% for workloads running at supercomputing centers (primarily scientific applications). Next, I will present a novel suite of light-weight cluster scheduling mechanisms for providing better response times to interactive workloads, which are better suited to commercial applications (databases and e-commerce) and web services. The third contribution is in scheduling support for applications having different Quality-of-Service (QoS) demands on the underlying system. Finally, I will summarize our ongoing work in providing systems support for a commercial database server on our Linux cluster.
These studies have required extensive simulations and experimental evaluations. I will also present details on our work in developing the first set of analytical models for these complicated systems. There has been extensive collaboration with IBM Research in this thesis, and the research has been supported in part by an IBM fellowship.