What is Streams Messaging?

The Streams Messaging capabilities offered within Cloudera DataFlow (CDF) and Cloudera Data Platform (CDP) enable enterprises to buffer and scale massive volumes of data ingests to serve the data needs of other enterprise and cloud applications that require access to such real-time data.

Primarily powered by Apache Kafka, the Streams Messaging capabilities include an entire ecosystem of tools to support your streaming needs.

Read why Cloudera offers the best Kafka ecosystem today

Use cases

  • Production Optimization
  • Market Monitoring
  • Customer Experience Management

Production optimization

Fine-tune operational processes, improve productivity and gain predictive insights. 

Some of the most innovative manufacturing companies are combining streaming data with historical data to predict and automatically change processes across different operating lines that will increase production.  Cloudera DataFlow’s Streams Messaging capabilities can realize the streaming data requirements and enable predictive use cases to automate manufacturing processes successfully.

Market monitoring

Handle millions of trades a second and scale to petabytes of financial information.

Financial stock exchanges face challenges with customer demands for real-time reporting and faster SLA requirements. Yet, petabytes of data must be processed to deliver these services. Cloudera DataFlow’s Streams Messaging can easily stream high volumes of data so stock exchanges can quickly create market-driven real-time analytics and meet the increasingly demanding SLAs.

Bombay Stock Exchange: World’s fastest stock exchange leverages real-time analytics to improve data governance and industry SLA’s

95% reduction in operational costs.

Read the case study

Customer experience management

Enhance customer experience with 360-degree data and real-time insights.

Better customer experience is knowing your customer segments and the types of products and services they desire. Cloudera DataFlow’s Streams Messaging capabilities can ingest high-volume, high-velocity data from multiple streams and stores that deliver customer data. It enables the holistic view of the customer via data lakes, operational stores, or analytic engines.

Globe Telecom: Enabling the digital lifestyle of mobile customers with a modern analytic environment

600PB of mobile data volume managed.

Read the case study

Key features

Supports millions of messages per second with low latency and high throughput, scaling elastically and transparently without downtime. Addresses a wide range of streaming data initiatives, enabling enterprises to keep up with customer demand, provide better services, and proactively manage risk.

Streams Messaging Manager provides a single pane of glass view with end-to-end visibility into how data moves across Kafka clusters—among producers, brokers, topics, and consumers—allowing you to track data lineage and governance from edge to cloud. It also simplifies troubleshooting of Kafka environments with intelligent filtering and sorting.

Streams Replication Manager, based on Mirrormaker 2, offers fault-tolerant, scalable, and robust cross-cluster Kafka topic replication, as well as replication monitoring and metrics at the cluster and topic levels. Delivers high availability, disaster recovery, cloud migrations, geo-proximity, and many others.

Schema Registry lets you manage, share, and support the evolution of all producer and customer schemas in a shared schema repository that allows applications to flexibly interact with each other across the Kafka landscape. Safely mitigate interruptions that occur due to schema mismatches.

Cruise Control lets you manage and load-balance large Kafka installations, as well as automatically detect and remediate anomalies. Address hard problems such as frequent hardware/virtual machine failures, cluster expansion/reduction, and load skew among brokers.

Cloudera SDX offers centralized security, control policies, governance, and data lineage across all components. They are set once and automatically enforced and are vendor-agnostic, allowing you to confidently embrace multi-cloud and hybrid cloud strategies. Supports the four main pillars of security: Identity, access, data protection, and visibility.

Deployment options

Buffer, scale, and deliver massive volumes of data across on-premises, public cloud, and hybrid cloud environments.

Streams Messaging for Data Hub

  • Extends on-premises Apache Kafka clusters to the public cloud with Streams Messaging for Data Hub
  • Choose your cluster size (S/M/L) and launch it across AWS, Azure or GCP
  • Extend the same on-premises user experience into the public cloud

CDP Private Cloud Base

  • Comprehensive Kafka ecosystem to deliver the best end-to-end streaming architecture
  • Manage, monitor and replicate your Kafka clusters with SMM & SRM
  • Secure, govern and protect your Kafka clusters with tight Cloudera SDX integration

Getting started

Product documentation

Read technical specifications, architecture, tutorials, and how-to articles about Apache Kafka.

Learn more

CDP Data Hub pricing

Evaluate CDP Public Cloud pricing for Data Hub across various instance types and cloud providers.

Get the details

Kafka on the cloud

Extend your streams messaging capabilities to the cloud with CDP Data Hub.

Watch a quick intro video

Best Kafka ecosystem

Learn why Cloudera offers the best Apache Kafka ecosystem in the industry today.

Read the whitepaper

Cloudera Community on Kafka

Connect with your peers, ask questions, troubleshoot, and learn more about Apache Flink.

Explore now

Streams Messaging Resources

Get all the key assets to learn how streams messaging will transform the way you manage streaming data.

Read now


Kafka PowerChat: Challenges in streaming architectures


Manage, monitor, and replicate Apache Kafka with Cloudera Data Platform


Advanced messaging and streams management

Solution Brief

Data-in-motion philosophy: A blueprint for enterprise-wide streaming data architecture

World-class training, support, & services

Your form submission has failed.

This may have been caused by one of the following:

  • Your request timed out
  • A plugin/browser extension blocked the submission. If you have an ad blocking plugin please disable it and close this message to reload the page.