We’re thrilled to announce the launch of Cloudera Surveyor for Apache Kafka, a powerful new tool designed to simplify the management and monitoring of your Kafka clusters! Previously, managing Kafka meant navigating complex command-line interfaces (CLIs) or dealing with outdated tools, resulting in a lack of a cohesive view of your environment due to disparate components or accumulated technical debt. That's why we built Surveyor with a modern architecture focused on intuitive design, scalability, and robust security, providing an out-of-the-box, unified management experience.
Surveyor is engineered to bring a centralized, intuitive, scalable, and secure platform for managing Apache Kafka clusters efficiently through a graphical interface, reducing the reliance on the Kafka CLI. What sets Surveyor apart is its broad compatibility: it can connect to and manage any Kafka distribution that provides an API compatible with Apache Kafka 2.4.1 or higher. This includes Kafka clusters deployed with the Strimzi Cluster Operator in Cloudera, Streams Messaging Kubernetes Operator, Kafka clusters running in Cloudera on-premises and Cloudera on cloud, as well as third-party Kafka distributions.
Figure 1: Visualizing your Kafka clusters with ease: The Cloudera Surveyor UI provides a clear overview and streamlined management capabilities.
The Surveyor Advantage: Overcoming Kafka Management Hurdles
Surveyor is engineered to address key challenges faced by Kafka users and administrators, designed specifically to fill the management gap with a truly modern solution:
Intuitive graphical interface: Say goodbye to the command line for everyday tasks. Surveyor provides a user-friendly UI for managing your Kafka clusters, topics, and consumer groups, making administration tasks much more efficient.
Multi-cluster support: Manage all your Kafka clusters from a single, centralized interface. Surveyor allows you to add, remove, and switch between clusters with ease, and even assign arbitrary tags for better organization and filtering.
Actionable health and performance insights: Gain deep visibility into your Kafka environment. Surveyor offers real-time insights into cluster health, performance, and key metrics, helping you quickly identify and address issues across brokers, topics, partitions, replicas, and consumer groups.
Enhanced security: Security is paramount. Surveyor adopts a secure-first approach, with TLS enabled by default for encrypted communication and robust LDAP authentication support. It also enforces Kafka ACLs to ensure users only perform actions they are authorized to do.
Scalability for enterprise deployments: Engineered for large-scale production environments, Surveyor can handle hundreds of Kafka clusters, each with hundreds of brokers and thousands of topics, ensuring it grows with your needs.
Broad Kafka compatibility: Manage diverse Kafka deployments from a single pane of glass, whether they’re on-premises, in the cloud, or on Kubernetes.
Insights in Action: An Example of Surveyor Success
Consider the following use case: Sarah is a Kafka administrator who manages over 20 diverse Kafka clusters deployed across on-premises data centers, public clouds, and Kubernetes environments. While reviewing her centralized Surveyor dashboard, she notices a “Critical” health status flagged on her primary production cluster.
Instead of Sarah needing to sift through countless logs spread across different environments or running multiple CLI commands for each cluster, Surveyor highlights an “Offline Partition” alert. This directs Sarah to check the broker's logs and address a disk failure. A quick drill-down, guided by Surveyor's intuitive visual indicators, reveals the specific topic and partition experiencing issues, along with the affected broker.
Thanks to Surveyor's real-time, centralized insights and clear visual cues, Sarah quickly pinpoints the problem, takes corrective action, and restores service much faster than before, significantly minimizing the impact on her organization's critical data streams.
Core and Out-of-Box Component of Cloudera Streams Messaging
Surveyor is a key component of the newly released Cloudera Streams Messaging - Kubernetes Operator 1.4.0. With this release, we’re integrating Surveyor directly into your Kubernetes environments, enhancing your ability to deploy and manage Kafka and related components on existing, shared Kubernetes infrastructure, completely eliminating the need for dedicated setups. This delivers flexible, agile, and rapid deployment and scaling for variable workloads, standardization on existing Kubernetes infrastructure, and operational efficiency through simple upgrades and swift cluster creation.
Cloudera Streams Messaging is deployed primarily via the Cloudera platform and the Cloudera Streams Messaging Operator on Kubernetes. Cloudera provides a managed, integrated platform with Cloudera Streams Messaging as a core service, ideal for comprehensive data lifecycle management. The Cloudera Streams Messaging - Kubernetes Operator, on the other hand, allows native deployment and management of Kafka components directly on Kubernetes, offering a more agile, cloud-native approach for those prioritizing Kubernetes-based workflows.
Beyond messaging with Kafka, Cloudera's broader Data in Motion portfolio includes robust offerings for data ingestion and transformation with Apache NiFi and powerful stream processing and analytics with Apache Flink.
Ready to Experience Seamless Kafka Management?
Cloudera Surveyor for Apache Kafka is proprietary software developed by Cloudera, ensuring it meets our high standards for performance, reliability, and support. We’re incredibly excited about Surveyor and its potential to transform how you manage your Kafka deployments.
Dive deeper into its capabilities and see it in action:
Explore the documentation: Learn more about deploying, configuring, and using Cloudera Surveyor for Apache Kafka by visiting our official documentation.
Connect with the community: Join the conversation, ask questions, and share your experiences with Cloudera Surveyor on our Community announcement post.
Get ready for a more efficient, intuitive, and secure Kafka experience!
