State of Facial Recognition (Azure Face API et al) & Sentiment Analysis

Sam just wrote a precis on Why Facial Recognition Is the Next Big Thing in Marketing which outlines how brands are / can use the facial recognition to increase engagement, and therefore sales. From a machine learning and data science perspective, building algorithms which understand what one's face is really saying i.e. performing emotion analysis to find insight into purchasing patterns is an emerging area marred by privacy concerns. Discovery true feelings lurking behind facial expressions while keeping the users anonymous is challenging less because of eigen values, and more because of respective omnipresent 'creep' factor.

Sentiment Analysis as a service is not something new; Google cloud vision, Microsoft Cognitive Services, Microsoft Azure Media Analytics, and AWS marketplace all offer some form of image Sentiment Analysis and Vision API which can analyze emotional facial attributes in images. These features are then provided as part of programmable objects such as the JSON output below.

JSON:
[
  {
    "faceRectangle": {
      "left": 488,
      "top": 263,
      "width": 148,
      "height": 148
    },
    "scores": {
      "anger": 9.075572e-13,
      "contempt": 7.048959e-9,
      "disgust": 1.02152783e-11,
      "fear": 1.778957e-14,
      "happiness": 0.9999999,
      "neutral": 1.31694478e-7,
      "sadness": 6.04054263e-12,
      "surprise": 3.92249462e-11
    }
  },

Targeted advertising system based on an audience recognition, especially in a streaming manner would allow marketers access to a wide stream of data points to the sentiments such as surprise, joy, anger, sorrow, disgust etc; combining this information with merchandise (object detection) and the corresponding product logo (the causal inference), so you can assess how clientele feels about a particular product.

The methods and systems for dynamic advertising on mobile and stationary platforms is implied in Gartner’s Hype Cycle for Emerging Technologies in 2015-2016 (Source Gartner)

Facial recognition privacy woes aside, the applications for determining and responding to User Sentiments (i.e. beyond the facial recognition) during their shopping experience, or during viewed media contents (yes your smart TV might be watching you), provide tremendous value, and are bound to happen. Mandatory privacy joke here:

....Privacy is a very modern concept,...

so is sanitation!

But I digress. So the idea goes beyond facial ‘recognition’ venturing into sparse matrix factorization for anonymized ‘sentiment analysis’ i.e. recognition of excitement, surprise, like/dislike and other features which can be associated to the eventual outcome for the purchase. As a machine learning enthusiast, an upcoming technology to keep an eye on are context-enriched services; i.e. set of services which proactively anticipate the user’s needs, and serve up the content, product or service most appropriate to the user are essentially the future of shopping experience, both online and in-store. Merging that with streaming analytics and viola, you have a dynamic shopping card based on wishful looks of the clients.

Integrating with big data repositories which provide customer demographics, behavior, sentiment analysis and using machine learning to augment the experience, the ROI/business Impact of such services is tremendous. Virtual assistants can also be employed to reduce the ‘awkwardness’ factor in case of human-human coordination especially for sensitive purchases. Video analytics can also help to enhance the customer experience and drive revenue generation by obtaining a better understanding of customers’ “reality experience.” The technology is ideally suited to the retail industry, but other industries (including travel and entertainment) will also benefit from tailored and customized deployments.

References & Further Reading