Big data has presented us with unprecedented opportunities to model and understand massive visual and textual contents generated by the explosion of a wide variety of digital applications ranging from social media to healthcare. With the developments of machine learning and artificial intelligence, large-scale and rich resourced general knowledge can be used to infer information for various applications. However, for some specific domains there only exist limited resources based on which we are aiming to achieve robust performances, such as healthcare where conventional machine learning approaches perform hardly as effective. We argue that the incorporation of domain knowledge with machine learning approaches is capable of alleviating the complexity and insufficiency of utilizing general knowledge to solve such domain-specific problems.
In this thesis, we first present our preliminary studies on healthcare about disease detection and classification on textual data, and visual sentiment analysis, in order to showcase the major challenges in healthcare analysis. We then propose a novel and versatile framework to extract coarse-grained and fine-grained domain knowledge with minimum supervision in the form of a knowledge graph. The effectiveness of domain knowledge incorporation has been validated in the tasks of multimodal sentiment analysis and automatic radiology report generation where we take the advantages of the rich semantics conveyed in the domain knowledge. Furthermore, we explore the feasibility of utilizing social multimedia for analyzing human health and well-being including user behavioral and psychological analysis.
Advisor: Prof. Jiebo Luo (Computer Science)
Committee: Prof. Chenliang Xu (Computer Science), Prof. Zhiyao Duan (Electrical and Computer Engineering), Dr. Timothy D. Dye (SMD)
Reception to follow at 12:00pm in third-floor atrium of Wegmans Hall