With just about every major brand and celebrity on Facebook and Twitter, the line between social media and popular culture is becoming harder to distinguish by the day. What is evident is the incredible power social media has in uncovering popular sentiment and the way information is being shared and disseminated. The sheer amount of data amassed on any social network on any given day holds an incredible amount of intelligence and insight into where the collective sentiment of the masses lies.

The numbers speak for themselves: Twitter grew 1,444 percent last year with more than 50 million tweets sent each day, and Facebook now has over 400 million active users. Every minute, 600 new blog posts are published, 34,000 tweets are sent, and 240,000 pieces of content are shared on Facebook.

The question now is how businesses can benefit from all this data and what, if any, tangible intelligence can be extracted from it.

In fact, the aggregation of data from social media networks and forums can become an incredibly powerful tool for gathering real-time intelligence and making more informed and strategic business decisions. Companies can also analyze the data to better understand customer sentiment and satisfaction, market trends and competitive threats, to name just a few. From a cultural standpoint, the data can also provide insight into the collective opinions of the masses on a number of relevant issues, including political elections, social crises (like the BP oil spill) and yes, even reality television shows.

Case in point: At Kapow Technologies we created Reality Buzz, a social media analysis project that examined whether real-time analysis of social media conversations could predict the outcome of popular shows like American Idol and Dancing with the Stars. Reality Buzz semantically extracted tens of thousands of tweets, comments and discussions about contestants on both programs each week and applied sentiment analysis to the data. In the end, we were able to draw some clear, data-driven conclusions to predict the contestants who would be eliminated each week.

Although some may dismiss the prediction of reality television shows as frivolous, it demonstrates some crucial lessons about social media sentiment analysis that can be applied to the business world:

1. Add structure to unstructured data

Data on the Web is usually found as unstructured information on Web pages, meaning that it is not semantically tagged. However, adding semantic rules to structure the data makes it far more valuable for analytics. For example, if you pull data from a Web forum, you want to extract information about the author, time-stamp, tags, subject and so on. Any analytics done after it has been extracted will benefit tremendously from the added semantic information, providing meaningful business intelligence, such as customer sentiment trends or shopping behavior patterns.

2. Measure in real-time

Although Web data and the potential sentiment to glean from it are endless, the way companies approach and analyze this data requires a measured approach.  Most notably for business – relevancy is key. And what’s most relevant is what’s happening right now – not a week or a month in the past. On American Idol and Dancing With the Stars, for instance, post-performance data carries more weight than the prior five days.

3. Determine goals in advance

Companies need to clearly define their goals before analyzing social media data. There are differing degrees of sentiment, and not all translate equally well.  Most sentiment analysis tools begin by separating data into positive and negative groups, yet there is still a wide range of variability within each group.  In the world of reality TV as one example, there is a huge difference between a viewer liking a contestant and voting for them once and loving him or her so much they vote 100 times. Companies also need to consider how to weigh one tweet versus a Facebook comment versus a blog post – which is more meaningful? These are conversations that companies should have before – not after – a social media sentiment analysis project begins.

4. Filter out the noise

Even with the right analytic technology in place, it’s still important to understand that changes in momentum, volume of traffic and ratio of positive to negative sentiment has an effect on the data’s overall reliability. The more activity and conversation around a particular topic – whether that be a reality show, a company, a product or something else– the more “noise” is inevitable in the data’s finding. This is why it’s so important to understand how to filter and transform the data to maintain its quality and relevance.

From Reality TV to Business Intelligence and Everything in Between

Social media sentiment is a very real and measurable way for any organization, regardless of sector or affiliation, to predict future behavior and ensure their business strategies are aligned with what customers and other company stakeholders are openly revealing to them on the Web. Semantic technologies will only become increasingly prominent and important for these types of applications.

In the case of Reality Buzz, the result of the sentiment analysis and its accuracy in making eliminations predictions was of course dependent on good text analytics, but what was even more important was the extraction of semantically well-organized and “noise-free” data. No analysis is better than the data behind, and for text analytics this is even truer. This would be accurate for other applications outside of reality TV as well. In order to analyze the tweets, Facebook comments/likes/dislikes, the right information and semantic structures need to be extracted from the Web. An emerging class of technologies known as Web data services can help organizations to reliably collect data from unstructured web data sources and automatically apply rules to create the semantic structures essential for accurate text and sentiment analysis. The extreme ease in which content is now accessible through the Web makes data access easy, and with  Web data services in place, the information can now also be structured and made programmatically available as feeds or SQL data.

There’s no question that the impact of social media on business intelligence is growing exponentially and we are just beginning to decipher how impactful one is to each other. The early adopters of Web data access technologies are already gaining a competitive edge with their real-time customer and market understanding from applying semantic structures and text analytics to Web social media.

About the Author(s)