Sentiment Analysis: When Humans Are Not More Reliable

Jason Falls‘ recent post on new analytics entrant Sentiment360 and the human advantage that they bring to the table resulted in a flurry of comments from experts within the social media analytics industry. Sentiment360 is definitely on to something and industry insiders are beginning to take note.

But questions remain as to why it makes sense (which I believe it does, by the way) to hire humans to analyze sentiment as opposed to relying solely on computer algorithms which have been shown to be less reliable than humans.

One insider asking some very good questions – and providing intriguing answers too – is Zoe Siskos, who, in a comment on Jason’s blog, referenced a recent study she co-authored with five of her colleagues from Syncapse Corporation‘s Measurement Science Team. I’m not going to go into all the details here (it was a brilliantly conceived, well-designed and highly scientific study) but here are the conclusions I came to (slightly different than what the authors came to):

1. Ambiguity in the instructions given from CMOs to human analysts need to be crystal clear to prevent tonality scores that are competely inaccurate.

2. Unambiguous instructions will be much easier to formulate if your enterprise has clear business goals that tie directly into what Jeremiah Owyang of Altimeter Group insightfully referred in a special report as the 5Ms and the 18 Social Uses Cases.

CMOs and Service Providers: What do you think: Are human analysts worth hiring for sentiment analytics or not? How is your company dealing what sentiment analytics? Are you using only auto sentiment analysis? Only human? A combination? Not at all?

Photo Credit: Second Life Resident Torley LindenVisit Ambleside.

Reblog this post [with Zemanta]

8 thoughts on “Sentiment Analysis: When Humans Are Not More Reliable

  1. In full agreement on the need for clarity in the instructions to analysts. One advantage that human analysis can provide (as witnessed by Sentiment360's monitoring of an advertising campaign for a major government anti-drug effort) is the ability of analysts to connect disparate bits of chatter and discover trends. This same effort also showed the ability of analysts to discern complex videos and imagery and develop not only truer sentiment scoring but also provide meaningful context to the conversation by understanding the range of emotions involved.

  2. I had the opportunity to comment on Jason's post, and I'm glad I stumbled upon yours.There are so many reasons why human review becomes crucial to the analysis and measurement phases. Video/audio and images (which make up approx 3% of the SM source mix – higher if you include their shared use on social networks sites like Facebook and MySpace) themselves present enough reason for clips to be reviewed by a reader.This is why we've continued to include human review in all our firms offerings. However we also provide our users an optional dashboard view into monitoring real-time mentions which are auto assigned a sentiment score.From the standpoint of using monitoring as an advanced warning system to mitigate risk, auto assigned sentiment creates a decent windsock approach to assist with following the direction of trends/topics, however only human review and the interpretation of risk/context (enabled through the actual reading of each incident) makes it possible to proceed to the “what next” stage.Thanks for advancing the discussion!Joseph | RepuMetrix Inc.@RepuTrack

  3. Thank you for the reference to the Syncapse study and the shoutout,Indeed, clarity in the instructions should reduce error to an extent. If you go to great lengths defining what is considered 'positive', and those instructions are executed faithfully, there is less margin for error. A much more fundamental problem, systemic in marketing, is that the consumers of the analysis don't necessarily know what the specific definitions are – so when their perception collides with what is scored – dissatisfaction ensues and legitimacy is questioned. The problem has persisted for 18 years in web analytics and will persist in social for much longer. Communication can take us pretty far in solving the problem. To your second point: yes, it should always begin with alignment to actual business goals.Thanks again for the shoutout,

  4. Great point on multimedia content. I don't know of any “technology” that is able to assess sentiment of say, a youtube video. To my knowledge (and correct me please if I am wrong), this is a HUGE blind spot for social media monitoring technology right now. So a services model makes perfect sense.

  5. Joseph, Thanks so much for joining the conversation abnd good point on multimedia. I agree, human analysis is key to the “what next” stage. Are you aware of any additional stats by chance, on the SM source format mix trends from Nielsen, Edison, Forrester or others? Also, here's an Edison report that I thought was quite interesting on social media platform trends:http://www.edisonresearch.com/home/archives/201

  6. Christopher – Kudos to you on a terrific study. It really made me think. As you can tell. In response to your comment: In your view, what steps should consumers and providers of the marketing data take to ensure that they are both on the same page? What is getting in the way of the clear communication that is vital to this whole process? Is there perhaps an instruction format that might help to eliminate this problem? Should instructions perhaps always include examples of positively scored vs. negatively scored mentions?

  7. Joseph, Thanks for sharing this. I'm curious: Does this breakdown reflect the requests of your clients or the %s you observed without instructions from clients to focus on or not focus on multimedia? In other words, I've found some companies are not concerned about multimedia content monitoring. Therefore, one wouldn't cover that content because of client instructions, not necessarily because of a lack of such content. Can you clarify? Thanks!

Leave a Reply

Your email address will not be published. Required fields are marked *

*

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>