Robust Image Sentiment Analysis Using Progressively Trained and Domain Transferred Deep Networks

Quanzeng You and Jiebo Luo, University of Rochester

Hailin Jin and Jianchao Yang, Adobe Research

AAAI 2015

Abstract

Sentiment analysis of online user generated content is important for many social media analytics tasks. Researchers have largely relied on textual sentiment analysis to develop systems to predict political elections, measure economic indicators, and so on. Recently, social media users are increasingly using images and videos to express their opinions and share their experiences. Sentiment analysis of such large scale visual content can help better extract user sentiments toward events or topics, such as those in image tweets, so that prediction of sentiment from visual content is complementary to textual sentiment analysis. Motivated by the needs in leveraging large scale yet noisy training data to solve the extremely challenging problem of image sentiment analysis, we employ Convolutional Neural Networks (CNN). We first design a suitable CNN architecture for image sentiment analysis. We obtain half a million training samples by using a baseline sentiment algorithm to label Flickr images. To make use of such noisy machine labeled data, we employ a progressive strategy to fine-tune the deep network. Furthermore, we improve the performance on Twitter images by inducing domain transfer with a small number of manually labeled Twitter images. We have conducted extensive experiments on manually labeled Twitter images. The results show that the proposed CNN can achieve better performance in image sentiment analysis than competing algorithms.

Paper[PDF]

Citing us

If you make use of this dataset, please refer to the following paper:

Quanzeng You, Jiebo Luo, Hailin Jin and Jianchao Yang, "Robust Image Sentiment Analysis using Progressively Trained and Domain Transferred Deep Networks", the Twenty-Ninth AAAI Conference on Artificial Intelligence (AAAI), Austin, TX, January 25-30, 2015.

Data

We submit two batches to AMT, which consist the total 1269 Twitter images used in our experiments. We provide both the original batch results from AMT as well as the process data file.

You can simply combine the results from the two batch files to obtain all the labels for the total 1269 images or simply download the Processed Data.

Batch_1544751_batch_results.csv

Batch_1545310_batch_results.csv

Processed Data (0: negative and 1: positive)

At least three agree [txt]

At least four agree [txt]

At least five agree [txt]

Images

All 1269 images are available for download. [ZIP] [TAR]

Top Ranked Images (green solid box: correctly predicted samples, red dashed box: wrongly predicted samples)

Since we cannot gurantee that there are no duplicate images in the 1269 images, it is possible that there are some images that look the same. We manually remove those duplicate images in the top ranked examples to give more diverse examples of each algorithm.

Top ranked negative images by prediction score from top to bottom in a decreasing order
PCNN CNN Sentribute Sentibank GCH LCH GCH+BoW LCH+BoW


Top ranked postive images by prediction score from top to bottom in a decreasing order
PCNN CNN Sentribute Sentibank GCH LCH GCH+BoW LCH+BoW

URCS People | URCS Home Page