In this talk, I will first elaborate on the nature of the challenges in knowledge discovery, and then summarize some aspects of my work in which I developed techniques that can be used to address them. In particular, I will present two of the major components of my thesis: document clustering and mining temporal patterns. First, I will present a new document clustering algorithm which explicitly models the subspace structure associated with each cluster. The clustering algorithm simultaneously performs two tasks: data reduction by assigning data points to clusters and subspace identification of each cluster. The tasks are carried out via an iterative optimization procedure that alternates between identification of the subspace structure from current cluster partitions and update of cluster partitions based on new subspace structures. I will present the theoretical foundations of the algorithm as well as experimental results that demonstrate its efficiency and effectiveness.
Next, I will discuss the problem of discovering temporal patterns in event data. This is formulated as the problem of comparing two probability distributions of inter-arrival times. I will present a technique for discovering significant, but infrequent, patterns. It has been proven that this technique provides a meaningful characterization of patterns, and that it is robust against noise. I will present the theoretical underpinnings of the technique and the results obtained by applying the technique to production network data. I will conclude the talk with a brief summary of other research topics I conducted.