The goal of the thesis is to develop algorithms for learning from labeled and unlabeled data. We first propose a simple and efficient multi-class classification approach via generalized discriminant analysis. Second, we study the methods for automatically generating hierarchical structures to facilitate classification. Third, we introduce the concept of co-updating, a semi-supervised learning algorithm from different information sources. Finally, we summarize our previous work that is closely related to the proposal and discuss future research.