Electronic devices generate data every millisecond they are in operation. This data is vast, complex and contains a wealth of useful information. Network-based service providers — such as utility companies, mobile providers and broadcast networks — are capturing, storing and analyzing machine-generated data to help them measure and improve customer experience, provide proactive support, predict and prevent service outages, and drive impactful product roadmaps. For utility companies, sensor data from smart meters provides environmental information and corresponding resource usage for every point on the grid. For mobile providers, call detail records (CDRs) contain the details of each call or event that passes through a switch. As networks and devices continue generating more data at shorter intervals, and as user bases continue to grow, harnessing the volume, variety and velocity of this data presents a challenge.

Traditional Ways of Analyzing Customer Usage and Experience are Slow and Cumbersome

Network-based service providers have traditionally relied on two mechanisms to understand customer usage and to identify areas for improvement:

• 1. Direct customer feedback: Usually offered through routine surveys or client outreach for support, this data is valuable yet often skewed because participation is self selected. Example: If a cable company receives several calls from clients enrolled in the “family package” who want to add a sports channel to their plan, the company might notice a trend and decide to offer both sports and family channels in a single, bundled package. But they may be missing other cross-sell opportunities if those clients haven’t made calls to customer support.
• 2. Data collected from machines, typically on a weekly or monthly basis: Utilities and mobile or broadcast network providers must measure network usage, primarily to ensure correct billing. Example: Utility companies measure energy consumption at customer accounts by checking each meter on a monthly basis. However, if there was an error on the customer’s device in Week One of the billing cycle, it might not be identified for another three weeks, resulting in the utilities provider losing revenue because of an inability to accurately bill for energy consumption during the entirety of that month.

From a technology perspective, network-based service providers have traditionally relied largely on online analytical processing (OLAP) using relational database management systems (RDBMS) to collect and analyze this information. Today operators increasingly face demands to capture more machine-generated information at a higher granularity and with greater frequency than before. This demand is driven by business needs to better understand system operation as well as customers’ behaviors and actions. Herein lies the challenge: traditional data warehouse environments struggle to capture and analyze
machine-generated data with such high volume, velocity and variety.