Image Mood Classification Guide

A model example for compound image classification for emotional impact and metadata tagging.

Background

This model relies on a first pass with image classification features and then learns a secondary classification layer. To use this model within Acumos, one would need to use a model connector available through the design studio. Using classification outputs from image classifier, predict the mood of an image into classes (here Amusement, Awe, Contentment, Excitement, Anger, Disgust, Fear, and Sad).

Adapting a task of image mood classification, this model learns a set of labels on top of the image-classifier output features (some 1000 classification tags). The training data comes from the “art photos” collection from this academic work also posted on the authors’ publication website:

Jana Machajdik and Allan Hanbury. 2010. *Affective image classification using features
inspired by psychology and art theory*. In Proceedings of the 18th ACM international
conference on Multimedia (MM '10). ACM, New York, NY, USA, 83-92.
DOI: https://doi.org/10.1145/1873951.1873965

Additionally, the images used to train this model came from a dataset available here.

Following original experimental definitions, the model is evaluated by separating the data into a training and test set using K-fold Cross Validation (K = 5). Original performance metrics comparing methods on this data are shown below, as found in the original publication.

Usage

Input to the model is an array of multiple classification scores for each image. Specifically, an image index, a class name, and the probability of that class are expected as inputs. Outputs of the model will contain a similar format except that they will be created in the domain of the mood classifier classes: image index, mood class name, and probability score. The probabilities are normalized to sum to 1.0 over all values so that they can utilized as relative confidence scores.

A web demo is included with the source code, available via the Acumos Gerrit repository or the mirrored Acumos Github repository. It utilizes a protobuf javascript library and a few image examples to classify and display the top N detected mood scores, as illustrated in the model image.

Once deployed, you can quickly jump to the default webhost page and point to your model for a demo; see demonstration_image_mood.

Performance

The model has a few training modes, but the algorithm below (random forest) seemed to perform the best between non neural-net computations. The training and evaluation is completed wholly though scikit-learn to demonstrate the utility of simple pipeline wrapping for an Acumos model.

(Random Forest - 300 estimators); (the default)

Class	precision	recall	f1-score	sample count
Amusement	0.29	0.27	0.28	22
Anger	0.25	0.10	0.14	10
Awe	0.37	0.37	0.37	19
Contentment	0.50	0.45	0.48	11
Disgust	0.21	0.27	0.24	11
Excitement	0.38	0.31	0.34	26
Fear	0.38	0.55	0.44	22
Sad	0.29	0.29	0.29	41
avg / total	0.33	0.33	0.33	162

More Information

Future work for this model may include increased sample count, additional learning algorithm evaluation, and the inclusion of raw features in addition to image classes. The image classes alone lack some deeper information (e.g. color, texture, etc.) that is typically associated with (and causes variations in) mood. Additionally, there are some performance differences detected as the resolution of the input image is varied, likely due to the original image classifier performing poorly with small-resolution objects.

Source Installation

This section is useful for source-based installations and is not generally intended for catalog documentation.

Package dependencies

Package dependencies for the core code and testing have been flattened into a single file for convenience. Instead of installing this package into your your local environment, execute the command below.

Note: If you are using an anaconda-based environment, you may want to try installing with conda first and then pip.

conda install --yes --file requirements.txt  # suggested first step if you're using conda

Installation of the package requirements for a new environment.

pip install -r requirements.txt

Usage

This package contains runable scripts for command-line evaluation, packaging of a model (both dump and posting), and simple web-test uses. All functionality is encapsulated in the classify_image.py script and has the following arguments.

usage: classify_image.py [-h] [-p PREDICT_PATH] [-i INPUT] [-C CUDA_ENV]
                         [-l LABELS] [-m {svm,rf}] [-f] [-n] [-a PUSH_ADDRESS]
                         [-A AUTH_ADDRESS] [-d DUMP_MODEL] [-s SUMMARY]

optional arguments:
  -h, --help            show this help message and exit

main execution and evaluation functionality:
  -p PREDICT_PATH, --predict_path PREDICT_PATH
                        Save predictions from model (model must be provided
                        via 'dump_model')
  -i INPUT, --input INPUT
                        Absolute path to input training data file. (for now
                        must be a header-less CSV)
  -C CUDA_ENV, --cuda_env CUDA_ENV
                        Anything special to inject into CUDA_VISIBLE_DEVICES
                        environment string

model creation and configuration options:
  -l LABELS, --labels LABELS
                        Path to label one-column file with one row for each
                        input
  -m {svm,rf}, --model_type {svm,rf}
                        specify the underlying classifier type (rf
                        (randomforest), svc (SVM))
  -f, --feature_nomask  create masked samples on input
  -n, --add_softnoise   do not add soft noise to classification inputs
  -a PUSH_ADDRESS, --push_address PUSH_ADDRESS
                        server address to push the model (e.g.
                        http://localhost:8887/upload)
  -A AUTH_ADDRESS, --auth_address AUTH_ADDRESS
                        server address for login and push of the model (e.g.
                        http://localhost:8887/auth)
  -d DUMP_MODEL, --dump_model DUMP_MODEL
                        dump model to a pickle directory for local running
  -s SUMMARY, --summary SUMMARY
                        summarize top N image classes are strong for which
                        label class (only in training)

Sample image examples can be found in the web_demo directory.

Example Images

For the purpose of testing the classifier a few sample images are provided. These example content files are intended to demonstrate the performance and functionality of the model included within the Acumos platform. See the source link for more info on the specific license of each image.

• example_excitement_2.jpg
• example_awe_1.jpg
• example_excitement_1.jpg
• example_sad_1.jpg

Example Usages

Please consult the Tutorial directory for usage examples or jump right to Deployment: Wrapping and Executing Image Mood Classifier Models to get started.

Release Notes

The Image Mood Classifier Release Notes catalog additions and modifications over various version changes.