Large national utilities company: optimizing smart meters and predictive analytics to drive business decisions and improve the customer experience

The world's largest utility companies rely on real-time data to keep operations running smoothly, 24/7, for customers and their communities. Around the world, utility infrastructure is becoming “smart” with built-in processing, connectivity, and sensing capabilities. As data is generated the volume grows exponentially each year with more smart devices and technologies being deployed, and the need for managing it becomes a critical factor. The challenge for this utility company resulted from lack of scalability and handling data queues which created maintenance issues. 

Challenge

Processing real-time streams of multi-structured information, as smart meters collect interval data in 15 minutes, the company needed to scale to serve more than 2 million customers. They chose Cloudera and IBM, a premier partner, to help manage data, and eliminate analytics silos across the organization. As massive volumes of machine-generated data from smart meters become readily available, they are leveraging this to detect and predict patterns in energy consumption. Through improved analytics, they can optimize energy distribution making the most efficient use of their resource pool. 

Additionally, the company wanted to use the data to predict and personalize the customer experience by leveraging the advanced metering infrastructure (AMI) — the two-way communicating smart meters. The technology improves billing accuracy and eliminates the need for a meter reader to enter onto a customer’s property, and offer its customers time-of-use pricing, automated demand response, near-real-time energy data feedback to better link them to the true cost of electricity and encourage them to save. This drives utility improvements, customer targeting, and optimize efficiency spending. 

For this utility company, meter data management significantly increased as data was growing and was no longer scalable outside of core usage and validations. Originally they created a data warehouse in a relational database which could not scale and handle the data load. As a result, issues arose managing data and updates — posing challenges and impacting data quality.

Solution

Cloudera’s enterprise data platform including DataFlow, NiFi, and IBM services was critical to gather all types of IoT data. Data collected from the meters includes not just consumption information, but also health and usage statistics that may indicate fraud or revenue leakage. For their engineering team, managing the data pipeline enabled data scientists to do the analytics - Spark based jobs - for daily health analysis. They also used Kafka to identify issues helping to reduce maintenance costs and increase customer satisfaction related to minimizing service disruption. With Cloudera's platform, CDF and IBM's industry insight, the company was able to not only stand up the platform to address their scalability challenges, but also realized value from their data faster.

Data analysis and optimization allows utility companies to establish a real-time, 360-degree view of their customers, creating individual customer interactions including providing customers with real-time utility consumption. For example, a customer portal allows for the viewing of usage of energy consumption. This data also helps companies sales and marketing teams target customers with the right information, at the right time, with product offerings tailored to their needs. By understanding how every customer's consumption needs are different, the company was able to deliver the best customer experience through proactive recommendations to help reduce the customer’s monthly utility costs.

Results

The utility company is now able to tackle the complete range of issues utilizing the enterprise data platform. End-to-end analytics flows that previously took 16 hours can now be completed in 5 hours, a 30% increase, which has consistently met its SLAs. Additionally, the business now has access to new data faster — what previously took 24 hours can now be done in 4 hours. Improved runtime performance, better scalability and reliability of AMI Interval Data has also been valuable. Data scientists and load researchers can now perform analyses not previously possible such as circuit level roll-up and load profile aggregation.

Insight to an individual customer’s value is key to the company's business objectives and core business values. They are able to develop a complete view of historical and real-time customer information to proactively monitor consumption, identify and eliminate issues related to overloads on their entire grid as they can better predict capacity and grid utilization. 

IBM, a premier Cloudera partner, provided the services of the utility company in their deployment and into production. This was a huge benefit as the bundling of Cloudera and IBM enabled the company to capitalize on the cost savings.