Scalability is increasingly crucial as the availability of digital imagery increases. The internet hosts billions of images which can potentially serve as rich sources of training data for recognition. Our work will explore this option through the following contributions. 1. A new method for collecting large, clean image sets of specific object categories from web image search. Our approach will use metadata including web page text to reduce the need for supervision, and it will exploit properties of web images for automatic segmentation and discovery of contextual information. 2. Creation of the first large-scale image dataset with clean labels, segmentation masks, and context descriptors. 3. A novel probabilistic model for visual recognition to efficiently combine object and contextual information from large-scale datasets. We will thoroughly evaluate our approach by testing recognition performance on known datasets, using our processed web images for training.