July 26, 2009
Sentiment Tagging: Some Guidelines
By: Amber Naslund
Sentiment is a tricky beast.
We all crave the technology that can automatically tell us whether a post we read and track is positive/negative/neutral (and the holy grail would be something that could make next step recommendations). Truth is, we’re not there yet. Technology isn’t perfectly up to the challenge so far, and the complexities and subtleties of human use of language will always render this a difficult task.
For now, and until/unless automated sentiment reaches unquestionable accuracy (not likely anytime soon), the human factor in analysis for sentiment is absolutely critical, and irreplaceable. So, I’m offering a few guidelines here about how *I* judge sentiment on the posts I track. I’ll say that on average, about half the posts I encounter get marked as neutral, so don’t be afraid of that.
A couple of things to note here: this is subjective. Period. And it should make sense in the context of YOUR business. Consistency in application is really the key; once you determine that a certain type of post is classified a certain way, stick with it to ensure that your analysis down the road is sound, and helps provide guidelines for others.
I’d love to hear your additions, questions, and suggestions in the comments.
What I mark as positive:
- Blatant and direct compliments or recommendations, without competitors mentioned. Can include product compliments or positive statements about service and support.
- Posts that contain superlatives in direct reference to our company or product (good, great, awesome)
- Reviews that are clearly complimentary, even if they contain a few improvements we could make
- If the post is a Digg, Stumble, or Delicious (someone found it valuable enough to vote on or bookmark)
- Retweets or links to any of the above posts
Somewhat positive:
- Retweets of our events or publicity (implied endorsement)
- Posts that announce/feature our inclusion in a list, ranking, or otherwise, including along with competitors
- Posts that recommend us alongside competitors
- Inquiries about getting a demo and/or trialing the product (implies good enough impression to ask to see more)
- Retweets or links to any of the above types of posts
Neutral
- Any tweets that are company outreach (from our employees). This helps to not sway the snapshot of what our community is saying, for better or worse
- Links to our website with no commentary at all
- Passing mentions of us in conversation unless they meet pos/neg criteria
- Statements like “checking out Radian6″ without other commentary
- Factual information about our product/brand without reaction or comment (including retweets)
- Links or retweets to our blog, events, etc. that don’t include commentary.
- Troubleshooting inquiries that are simply technical in nature
Somewhat Negative:
- Retweets or links from the community to third-party posts that contain criticism (passive endorsement of the negative content)
- Posts that contain criticisms of our product or service coupled with compliments or positive statements, if the negative seems to outweigh the positive
- Sarcastic comments that allude to a negative experience but without a blatant callout
- Troubleshooting inquiries that include statements of frustration
Negative:
- Clear criticisms or complaints about our product or service. These are usually pretty obvious.
So is this helpful? What would you add, and what else can I help answer? There are no perfect answers, but it sure helps if everyone shares some of their input. What do you think?
Great post. Have you had a chance to check out OpenAmplify yet? We do a great job of sentiment analysis using out open API.
Let me know what you think.
http://community.openamplify.com
Dave Weinberg
Community Manager
OpenAmplify
I completely agree that consistency is the most important part of sentiment analysis. The bias of individual mentions is far less important than the ability to accurately compare groups of mentions, and detect trends over time.
An important component of that consistency for me is defining a point-of-view. Most often, I imagine that I am an “informed outsider” – someone who understands my client’s industry but isn’t an employee, competitor, analyst or PR rep.
Any of those perspectives are a legitimate way to view media mentions, but I think it’s essential to pick one at the outset of monitoring and make sure you are always analyzing mentions from the same perspective.
Thanks for bringing up such an important aspect of social media monitoring!
I completely agree that consistency is the most important part of sentiment analysis. The bias of individual mentions is far less important than the ability to accurately compare groups of mentions, and detect trends over time.
An important component of that consistency for me is defining a point-of-view. Most often, I imagine that I am an "informed outsider" – someone who understands my client's industry but isn't an employee, competitor, analyst or PR rep.
Any of those perspectives are a legitimate way to view media mentions, but I think it's essential to pick one at the outset of monitoring and make sure you are always analyzing mentions from the same perspective.
Thanks for bringing up such an important aspect of social media monitoring!
Great post. Have you had a chance to check out OpenAmplify yet? We do a great job of sentiment analysis using out open API.
Let me know what you think.
http://community.openamplify.com
Dave Weinberg
Community Manager
OpenAmplify
Chris – thanks, should clarify. “I” should really mean “our small team”. I’m doing most of the sentiment tagging, but other folks on our team use the same guidelines when they’re tagging stuff.
And not agreeing is perfectly okay. It’s about what works for YOU. This is what works for us, for now. And if and when it doesn’t, we shift.
Chris – thanks, should clarify. "I" should really mean "our small team". I'm doing most of the sentiment tagging, but other folks on our team use the same guidelines when they're tagging stuff.
And not agreeing is perfectly okay. It's about what works for YOU. This is what works for us, for now. And if and when it doesn't, we shift.
Excellent article. Thanks.
I'd like to add a couple of things.
1- Sentiment is not discrete, it's a continuum. And you've different degree in negative sentiment.
So your clients (and our clients) have some decision to make upfront on how they treat mildly negatives, negatives with facts, strong negative, or even legal threat. I really like your answer to Chris and the fact that you already have 5 different flavor s of rating.
2- There is "social" in social media and when consumers say they want brands to participate, I doubt that they want their messages to be read by an algorithm. I want Amber to react or Marcel or David, not an algorithm (which by the way is wrong 30% of the time).
Best
Hello,
great conversation about transparency.
But due to the fact that I am looking for a tool to monitor the “social-media-voice” I have some questions:
E.g. I am interested in how the people like my new Company –> “XY Pizza Delivery Service ”
Assumed I use the radian6 tools, than I could type in my company name (XY Pizza Delivery Service).
How do you deliver the result (and sentiment results), especialy in terms of positive/negative/neutral?
Probably I get the name of the forums (or other source) and the number of accuarate threads with my keyword within a nice design, isn´t?
But how do I get the information if the thread is positive/negative or neutral. Do I get it directly, like in real time?
Due to the fact that you have written that sentiment results are impossible do deliver automaticaly there has to be the described manual work. Would explain it to us, how does that work to deliver the results in real time?
Thanks for your information and thanks for the transparency of this conversation.
Regards
Felix
Hello,
great conversation about transparency.
But due to the fact that I am looking for a tool to monitor the "social-media-voice" I have some questions:
E.g. I am interested in how the people like my new Company –> "XY Pizza Delivery Service "
Assumed I use the radian6 tools, than I could type in my company name (XY Pizza Delivery Service).
How do you deliver the result (and sentiment results), especialy in terms of positive/negative/neutral?
Probably I get the name of the forums (or other source) and the number of accuarate threads with my keyword within a nice design, isn´t?
But how do I get the information if the thread is positive/negative or neutral. Do I get it directly, like in real time?
Due to the fact that you have written that sentiment results are impossible do deliver automaticaly there has to be the described manual work. Would explain it to us, how does that work to deliver the results in real time?
Thanks for your information and thanks for the transparency of this conversation.
Regards
Felix
…
LGL Funny, I used to be discussing this thing with my older sister the other day, now I’ll have 1 more argument in my hand when it’ll arrive to confrontation when yet again. -YN….
Great post. I completely agree.
But what are the "Mixed" sentiment? I have few in my dashboards but don't get what they are reffering to.
Thank you,
Great post. I completely agree.
But what are the "Mixed" sentiment? I have few in my dashboards but don't get what they are reffering to.
Thank you,
[...] Radian6の過去の記事に手動でセンチメントを測定する際に、判定する人が持つべきカイドラインが記載されていたので妙訳しておく。 [...]