Login
Computer Science @ Rochester
Friday, March 21, 2003
11:00 AM
CSB 209
Filippo Menczer
U. Iowa
Mining, Mapping, Modeling and Crawling the Web
Can we model the scale-free distribution of Web links under realistic assumptions about the behavior of page authors? Can a Web crawler efficiently locate an unknown relevant page? These questions are receiving much attention due to their potential impact for understanding the structure of the Web and for building better search engines. This talk will discuss the semantic maps obtained by analyzing the connection between similarity functions based on text, link and semantic cues across a massive number of page pairs. These maps uncover some striking relationships. For example, link probability displays a phase transition between a region where it is not determined by content and one where it decays with textual distance according to a power law. This relationship suggests a novel Web growth model that is shown to accurately predict the distribution of page degree, based on textual content and assuming only local knowledge of degree for existing pages. A similar phase transition is found between link probability and semantic distance, and both results indicate that efficient paths can be discovered by Web crawling algorithms based on textual and/or categorical cues. I will conclude by surveying a number of applications of these findings to the evaluation and design of more efficient, effective, and scalable search engines and crawlers.