Overview
What is Edge and Flow Management?
Cloudera DataFlow (CDF) is a real-time streaming data platform for managing your data from edge to cloud. One of the key tenets of CDF is Edge and Flow Management.
The Flow Management capabilities of CDF, powered by Apache NiFi, deliver high-scale data ingestion, transformation, and management to enterprises from any-to-any environments. These address key enterprise and hybrid cloud use cases like data movement; log data ingestion; and acquisition of all types of streaming data including social, mobile, clickstream, and IoT data.
The Edge Management capabilities of CDF are made up of edge agents (MiNiFi) and an edge management hub called Edge Flow Manager. It manages, controls, and monitors edge agents to collect data from edge devices and push intelligence back to the edge. This addresses IoT use cases such as predictive maintenance, fleet management, and asset tracking.
Use cases
Predictive maintenance
Patient monitoring
Data movement
Predictive maintenance
Lower costs and reduce downtime with predictive maintenance.
Predictive Maintenance is a data-driven approach to analyze IoT and sensor data from connected equipment to effectively predict when and how an asset might fail, detect variances, understand warning signals, and quickly identify patterns that might indicate a potential breakdown. Cloudera DataFlow’s Edge and Flow Management capabilities modernize and simplify data ingestion from hundreds of connected assets to enhance predictive maintenance.
Patient monitoring
Capture real-time feeds from patient-monitoring devices to detect anomalies.
Biometric and telemetric devices are used in healthcare organizations to monitor post-surgery or high-risk patients. Ingesting sensor data from these devices about various patient vitals helps detect abnormalities or concerning patterns. Cloudera DataFlow’s Edge and Flow Management helps capture patient-monitoring data and delivers them to stream-processing engines for insights.
Data movement
Connect, integrate, and move massive volumes of data across hybrid and multi-cloud environments.
Traditional ETL processes are for use cases where data must move from one database to another. Modern enterprises transfer data from on-premises to cloud or cloud-to-cloud, moving petabytes of information in a matter of just hours. Cloudera DataFlow’s Flow Management capabilities are purpose-built for such use cases.
Hundreds of prebuilt processors are available to connect with a range of data sources, devices, and protocols. The user interface allows you to build sophisticated data flow pipelines with drag-and-drop ease.
Understand the origin and attribution of data as it moves throughout the enterprise, empowering the governance team to explain how any data point is affected by any system. Data lineage information is generated for everything it does at a fine-grained level, even when records change before and after an event.
Ingest, capture, and deliver data in real-time from any streaming source, including clickstreams, social media, mobile, or IoT devices. Enable actionable insights by easily connecting, transforming, managing, and monitoring them using complex data flow applications with NiFI’s 450+ processors and a custom-build monitoring dashboard.
Handle any throughput by moving petabytes of data from one data center to another in just a few hours or move data from your on-premises environment to the cloud or vice versa. Enable a multi-cloud model with a cloud-vendor-agnostic approach to managing data.
Adopt a DevOps-style data flow development lifecycle with NiFi Registry to deliver your flow applications faster and deploy them easily from one environment to another. Enable your development team to version their data flows and set up promotion schemes across environments.
Enable edge management at scale with command, control, and monitoring of hundreds of thousands of agents with minimal footprint to collect, filter, and process data. Allow end-to-end machine learning algorithms at the edge with automated learning loops.
Deployment options
Ingest and deliver data from the edge to the cloud across edge, on-premises, public cloud, and hybrid cloud environments.
Flow Management for Data Hub
- Spin up NiFi and NiFi Registry into your public cloud with Flow Management for Data Hub
- Choose your cluster size (S/M/L) and launch it across AWS, Azure or GCP
- Extend the same on-premises user experience into the public cloud
DataFlow for the Public Cloud
- Monitor all NiFi flows across multiple cloud clusters from a single dashboard
- Boost productivity by leveraging pre-built NiFi flows from a gallery of ReadyFlows
- Optimize the infrastructure sizing by allowing the DataFlow experience to auto-scale the NiFi flows
Get started
Product documentation
Read about technical specifications, architecture, tutorials, and how-to articles about Apache NiFi.
CDP Data Hub pricing
Evaluate CDP Public Cloud pricing for Data Hub across various instance types and cloud providers.
Logging Modernization
Learn more about one of the common use cases of flow management— Logging Modernization.
NiFi on the cloud
Extend your flow management capabilities to the cloud with CDP Data Hub.
Cloudera Community on NiFi
Connect with your peers, ask questions, troubleshoot, and learn more about Apache NiFi.
Training
Book a three day hands-on training course on Apache NiFi fundamentals and more.
