Building a Large Scale Dataset for Image Emotion Recognition: The Fine Print and The Benchmark

Quanzeng You and Jiebo Luo, Department of Computer Science, University of Rochester

Hailin Jin, Adobe Research

Jianchao Yang, Snapchat Inc




Psychological research results have confirmed that people can have different emotional reactions to different visual stimuli. Several papers have been published on the problem of visual emotion analysis. In particular, attempts have been made to analyze and predict people's emotional reaction towards images. To this end, different kinds of hand-tuned features are proposed. The results reported on several carefully selected and labeled small image data sets have confirmed the promise of such features. While the recent successes of many computer vision related tasks are due to the adoption of Convolutional Neural Networks (CNNs), visual emotion analysis has not achieved the same level of success. This may be primarily due to the unavailability of confidently labeled and relatively large image data sets for visual emotion analysis. In this work, we introduce a new data set, which started from 3+ million weakly labeled images of different emotions and ended up 30 times as large as the current largest publicly available visual emotion data set. We hope that this data set encourages further research on visual emotion analysis. We also perform extensive benchmarking analyses on this large data set using the state of the art methods including CNNs.

Cite us


Quanzeng You, Jiebo Luo, Hailin Jin and Jianchao Yang. "Building a Large Scale Dataset for Image Emotion Recognition: The Fine Print and The Benchmark", the Thirtieth AAAI Conference on Artificial Intelligence (AAAI), 2016.

We submitted different tasks to AMT. Both the processed data and the raw AMT results are provided.


1. Aggregated Results (
    Each line in the csv file have 4 columns: emotion, image, num_of_disagrees, num_of_agrees.
    (Note: num_of_disagrees + num_of_agrees should equal 5. However, some (very small number of them) may have less than 5 or more than 5.
     We simply use majority vote to select the ground truth.)
2. Raw batch results from AMT (
    Directly downloaded from AMT, including details on each task.

If you have any questions, please let me know.