Skip to main content

Microsoft Project Oxford: Machines That Recognize Human Emotion

Microsoft Project Oxford

One of the things that separate humans from machines is that we're able to recognize and distinguish between different emotions when we interact with others. With advances in Artificial Intelligence (AI), Machine Learning and computing power in general, apps and programs are being created that are able to recognize speech, identify faces and now even distinguish between different emotions!

Microsoft Project Oxford is an evolving portfolio of APIs and SDKs that allow developers to add intelligent services to their website or build an app to provide a better experience for their users by taking advantage of Microsoft's machine learning capabilities. There are three groups of tools within the portfolio - Vision, Speech and Language.

Microsoft Project Oxford Emotion Recognition

The Emotion APIs released for public beta this week, using Microsoft's cloud-based emotion recognition algorithms, identify emotions in the faces of a photograph. The eight emotions detected are anger, contempt, disgust, fear, happiness, neutral, sadness, and surprise which are understood to be cross-culturally and universally communicated with particular facial expressions.

Once the photograph has been uploaded, the emotion recognition tool automatically identifies the faces in the photograph and then assigns a score to each of the eight emotions, with the total adding up to 1. Now that we know how the tool works, we put it to the test with some iconic images from recent history. Let's see how the tool does, shall we?

Exhibit A: President Obama Re-election Speech

President Obama Re-election Speech

 Let's start with this image from President Obama's re-election speech in November, 2012. I think it's fair to say that most of us would say he's pretty happy. Well, the Emotion Recognition Tool thinks the same way too, assigning President Obama a "happiness" score of 85%.

President Obama Re-election Speech Analyzed

OK, so that was fairly easy. What about Mitt Romney who lost the election vote to President Obama in 2012? We thought he cut a rather sad figure, but interestingly, his score for sadness was rather non-existent. Instead, he was rated by the Emotion Recognition tool as being a mix of Neutral and Happy. Maybe he was just happy the campaign was finally over! 

Mitt Romney Concession Speech Analyzed

Exhibit B: 48 Hours After The Lehman Brothers Crash

Lehman Brothers collapse Hank Paulson  and Tim Geithner

Here's an image of Hank Paulson, Ben Bernanke and Tim Geithner 48 hours after the collapse of Lehman Brothers in 2008. Ben Bernanke was almost 100% neutral but we found Tim Geithner and Hank Paulson's scores more interesting. As you can see from the above picture, Tim Geithner looked pretty angry and his scores show, with a 40% "angry" rating. Hank Paulson on the other hand had the highest "fear" rating of the 3, coming in at 10%.

Exhibit C: Afghan Girl - National Geographic Cover 

Afghan Girl National Geographic Cover

For our final image test, we decided to mix it up a little and test the iconic "Afghan Girl" image by journalist Steve McCurry, featured on the June 1985 cover of National Geographic. To us, she looked fearful and sad, with a tinge of anger in her eyes. Let's see how the Emotion Recognition Tool scores the image.

Afghan Girl National Geographic Cover Emotions Analyzed

According to the tool, her expression is mostly neutral after which "sadness" and "anger" scored the next highest, which matches our initial assessment. 


We see two main areas where the Emotion Tool can have a big impact. The first is in security - where security cameras can not only recognize faces on screen, but also identify their emotions and assess whether they pose any threat. From our experiment above, it's clear that a photograph only captures the emotions at a point in time. If the Emotion Tool could be extended to a video, this would provide more samples to draw from, allowing for a more complete assessment of the subject's emotions. 

The other area which could use the Emotion APIs is Artificial Intelligence that interacts with humans. We previously wrote about Gmail's new Smart Reply feature that could parse through the content of an email you received and based on the words and sentences, come up with a short, intelligent reply. Words form only a small part of our communication and usually more importance is placed on non-verbal cues such as our facial expressions. It would really be the next step in building smart robots if they could decode not just our words and sentence structure, but also our facial expression.

Rounding Up    

Check out the Emotion Tool and the rest of Project Oxford for some cool next-generation technology. Follows us for more interesting tech stories, cool sites and gadget buying guides!


Popular posts from this blog

Best Cables for 144Hz Monitors - DisplayPort

144Hz monitors have a few different inputs, like DisplayPort , Dual-link DVI and HDMI to choose from, but which is the most recommended one? We've definitely come a long way from the days when monitors offered only a single VGA port. This article will explain which is the recommended port and type of cable to use for your 144Hz monitor.

Big Home Routers | Best WiFi Routers for a 4000 sq ft House

One of the great frustrations of living in a large or multi-story house is that a single Wi-Fi router isn't able to cover the entire house. Even homes around 2,000 square feet in size are likely to be plagued by Wi-Fi dead spots. If you'd like to say goodbye to terrible Wi-Fi, this guide will explain what Wi-Fi mesh is - one of the most exciting router technologies to emerge in recent years and also highlight what are some of the best mesh routers on the market today.

Choose The Right DVI Cable | Can DVI Support 144 Hz?

Can DVI cables support video display at 144 Hz? Dual-link DVI-D and DVI-I cables can support Full-HD display at 144 Hz but the confusion arises because there are so many different types of DVI cables and not all of them are able to support the high bandwidth required. In this article, we'll cover some of the DVI basics like connector types, single-link vs dual-link and connecting to other display standards like VGA and HDMI as well as recommend some popular DVI cables.