Sentiment Tagging: Some Guidelines
Sunday, July 26, 2009 | 9 Comments
Tags: radian6, sentiment, Social Media, social media measurement, Social Media Monitoring
Posted in: Platform, Social Media, Social Media Monitoring
Sentiment is a tricky beast.
We all crave the technology that can automatically tell us whether a post we read and track is positive/negative/neutral (and the holy grail would be something that could make next step recommendations). Truth is, we’re not there yet. Technology isn’t perfectly up to the challenge so far, and the complexities and subtleties of human use of language will always render this a difficult task.
For now, and until/unless automated sentiment reaches unquestionable accuracy (not likely anytime soon), the human factor in analysis for sentiment is absolutely critical, and irreplaceable. So, I’m offering a few guidelines here about how *I* judge sentiment on the posts I track. I’ll say that on average, about half the posts I encounter get marked as neutral, so don’t be afraid of that.
A couple of things to note here: this is subjective. Period. And it should make sense in the context of YOUR business. Consistency in application is really the key; once you determine that a certain type of post is classified a certain way, stick with it to ensure that your analysis down the road is sound, and helps provide guidelines for others.
I’d love to hear your additions, questions, and suggestions in the comments.
What I mark as positive:
- Blatant and direct compliments or recommendations, without competitors mentioned. Can include product compliments or positive statements about service and support.
- Posts that contain superlatives in direct reference to our company or product (good, great, awesome)
- Reviews that are clearly complimentary, even if they contain a few improvements we could make
- If the post is a Digg, Stumble, or Delicious (someone found it valuable enough to vote on or bookmark)
- Retweets or links to any of the above posts
Somewhat positive:
- Retweets of our events or publicity (implied endorsement)
- Posts that announce/feature our inclusion in a list, ranking, or otherwise, including along with competitors
- Posts that recommend us alongside competitors
- Inquiries about getting a demo and/or trialing the product (implies good enough impression to ask to see more)
- Retweets or links to any of the above types of posts
Neutral
- Any tweets that are company outreach (from our employees). This helps to not sway the snapshot of what our community is saying, for better or worse
- Links to our website with no commentary at all
- Passing mentions of us in conversation unless they meet pos/neg criteria
- Statements like “checking out Radian6″ without other commentary
- Factual information about our product/brand without reaction or comment (including retweets)
- Links or retweets to our blog, events, etc. that don’t include commentary.
- Troubleshooting inquiries that are simply technical in nature
Somewhat Negative:
- Retweets or links from the community to third-party posts that contain criticism (passive endorsement of the negative content)
- Posts that contain criticisms of our product or service coupled with compliments or positive statements, if the negative seems to outweigh the positive
- Sarcastic comments that allude to a negative experience but without a blatant callout
- Troubleshooting inquiries that include statements of frustration
Negative:
- Clear criticisms or complaints about our product or service. These are usually pretty obvious.
So is this helpful? What would you add, and what else can I help answer? There are no perfect answers, but it sure helps if everyone shares some of their input. What do you think?
9 Responses to “Sentiment Tagging: Some Guidelines”
Stuart Foster on July 27th, 2009 at 10:05 am
Awesome. Love to see the transparency here on how Radian6 scores sentiment. Definitely useful in gaining more adoption…
Dave Weinberg on July 27th, 2009 at 10:05 am
Great post. Have you had a chance to check out OpenAmplify yet? We do a great job of sentiment analysis using out open API.
Let me know what you think.
http://community.openamplify.com
Dave Weinberg
Community Manager
OpenAmplify
Chris Near on July 27th, 2009 at 11:29 am
The article says the definitions relate to, \"how *I* judge sentiment on the posts I track.\" Does that mean that other employees at Radian6 are coding using different definitions? If so, shouldn\’t they all be standardized to maintain consistency? That statement caught my attention (If I\’m interpreting it correctly). I don\’t necessarily agree with all the above definitions, but I do like the transparency here.
Amber Naslund on July 27th, 2009 at 11:36 am
Chris – thanks, should clarify. “I” should really mean “our small team”. I’m doing most of the sentiment tagging, but other folks on our team use the same guidelines when they’re tagging stuff.
And not agreeing is perfectly okay. It’s about what works for YOU. This is what works for us, for now. And if and when it doesn’t, we shift.
stop dreaming start action on August 3rd, 2009 at 5:24 am
complete guide, thank you
dominiq on September 1st, 2009 at 11:55 pm
Excellent article. Thanks.
I\\\’d like to add a couple of things.
1- Sentiment is not discrete, it\\\’s a continuum. And you\\\’ve different degree in negative sentiment.
So your clients (and our clients) have some decision to make upfront on how they treat mildly negatives, negatives with facts, strong negative, or even legal threat. I really like your answer to Chris and the fact that you already have 5 different flavor s of rating.
2- There is \\"social\\" in social media and when consumers say they want brands to participate, I doubt that they want their messages to be read by an algorithm. I want Amber to react or Marcel or David, not an algorithm (which by the way is wrong 30% of the time).
Best
Felix on November 17th, 2009 at 10:06 am
Hello,
great conversation about transparency.
But due to the fact that I am looking for a tool to monitor the “social-media-voice” I have some questions:
E.g. I am interested in how the people like my new Company –> “XY Pizza Delivery Service ”
Assumed I use the radian6 tools, than I could type in my company name (XY Pizza Delivery Service).
How do you deliver the result (and sentiment results), especialy in terms of positive/negative/neutral?
Probably I get the name of the forums (or other source) and the number of accuarate threads with my keyword within a nice design, isn´t?
But how do I get the information if the thread is positive/negative or neutral. Do I get it directly, like in real time?
Due to the fact that you have written that sentiment results are impossible do deliver automaticaly there has to be the described manual work. Would explain it to us, how does that work to deliver the results in real time?
Thanks for your information and thanks for the transparency of this conversation.
Regards
Felix
Stacy on November 18th, 2009 at 1:39 pm
What about questions? Do you put them into their own category? The brand I work for generates a lot of questions on how the product works. The tone of the comments are usually neutral.
Hannah Del Porto on July 27th, 2009 at 9:50 am
I completely agree that consistency is the most important part of sentiment analysis. The bias of individual mentions is far less important than the ability to accurately compare groups of mentions, and detect trends over time.
An important component of that consistency for me is defining a point-of-view. Most often, I imagine that I am an “informed outsider” – someone who understands my client’s industry but isn’t an employee, competitor, analyst or PR rep.
Any of those perspectives are a legitimate way to view media mentions, but I think it’s essential to pick one at the outset of monitoring and make sure you are always analyzing mentions from the same perspective.
Thanks for bringing up such an important aspect of social media monitoring!