Join the Dark Side(Dark Data), Now!
Thank you Manas Ranjan Kar for allowing us to repost your blog post from March 13, 2015. I read it and thought the reference to “dark data” struck a chord with databahn’s focus on mining unstructured content in the enterprise.
Unstructured data is growing significantly faster than structured data. Organizations nowadays have little awareness of the volume, composition, risk and business value of their unstructured data.
So, what is dark data exactly? This representation showcases it wonderfully
Dark Data occurs mostly in textual format - emails, customer feedback, social mentions and what not. IDC states that up to 90% of Big Data is Dark Data.
So why is it called "dark"? Because even though it lies in our servers and is in a human readable format, it is often not indexed and managed. Call it inactive or orphaned data if you must.
What can you do with it? Since most of it is in textual format, Text Analytics becomes all the more important. Using relevant techniques and algorithms you can
- ANALYSE COMPETITION – Map competitive landscape, research competitors and emerging trends in business
- DESIGN PRODUCTS – Mine social media , call logs, customer feedback to understand latent needs and help in product development
- OPTIMIZE OPERATIONS – Analyse logs, operational data to understand pain points
- TARGET & RETAIN CUSTOMERS – Combine unstructured data with transactional data to identify needs & preferences, tailor marketing campaigns and improve customer experience
Still confused how? One of our clients from the service industry had the same question too - "how" ?
We proposed a 4-step solution for him to improve customer experience using text analytics;
- Understand your customer feedback for voice of customer (VOC)
- Classify the feedback over time for event detection
- Scour the web for brand perception and answer questions about your services
- Provide personalized offers and customer recommendations
Dealing with dark data has multiple implications on the techniques involved as well, like, unsupervised learning becomes important as the data is unlabeled, which means a foray into deep learning.
While we are learning as we go along, there is no doubt that dark data analytics is going to become more relevant for small and large organizations alike. The proactive organizations can expect to reap immediate results, with a proper ecosystem to transfer the insights through out their hierarchy.
Manas Ranjan Kar is a Business Intelligence at Smart Mandate. He is an analytics professional helping his clients make sense of their data and build powerful cases for business change using analytics in their respective companies. He also likes to generate new ideas and devise feasible solutions to broadly relevant problems. You can follow Manas on Linkedin.com by visiting his profile page.
