Training
DOPS-243: NiFi 2.0: Data Flow Migration and AI Integration
Overview
Apache NiFi 2.0 is revolutionizing data flow automation. This course provides in-depth knowledge of NiFi 2.0's enhanced capabilities, empowering you to build intelligent, efficient, and governed data pipelines for today's most demanding data challenges. You'll explore AI integration, advanced flow management features, and streamlined Python scripting, enabling you to create data solutions that are not only powerful but also adaptable and maintainable. Learn to build more intelligent and efficient NiFi workflows leveraging the new GenAI and Python processors along with the flow governance tools.
What you'll learn
During this course, you learn how to:
- Integrate Generative AI with Apache NiFi 2.0, including practical use cases.
- Understand the key changes in NiFi 2.0 compared to NiFi 1.x, such as deprecated processors and their new replacements.
- Clarify the differences between open-source NiFi 2.0 and Cloudera CFM4 with NiFi 2.0, with a focus on Cloudera-specific processors for GenAI.
- Perform a migration from NiFi 1.x to NiFi 2.0 and effectively monitor your new NiFi 2.0 instance.
Who Should Take This Course?
This course is designed for developers, data engineers, administrators, and others with an interest in learning NiFi's innovative no-code, graphical approach to data ingest. Although programming experience is not required, basic experience with Linux is presumed, and previous exposure to big data concepts and applications is helpful.
Prerequisites
Participants should have a foundational understanding of data flow concepts and experience with Apache NiFi. Familiarity with Python programming is beneficial but not mandatory.
Book the course
Course Details
NiFi 2.0: Core Concepts & New Features
- Architectural Overview: Key changes and enhancements in NiFi 2.0.
- AI-Powered Data Flows: Introduction to the GenAI processor suite.
- Flow Governance: Deep dive into NiFi's new flow analysis rules and their enforcement.
- Python Processor Enhancements: Simplified custom processor development with Python.
- NiFi & Cloud: Working with NiFi in cloud environments.
- Hands-on Lab : Getting started with NiFi 2.0.
Building Intelligent Data Flows with GenAI
- Generative AI & LLM Overview
- Using RAG to enhance LLM capabilities
- RAG Architecture & NiFi 2.0 for RAG
- GenAI with NiFi 2.0
- NiFi GenAI Use Cases
- Hands-on Lab: Create a GenAI Pipeline in NiFi using Amazon Bedrock
- Hands-on Lab: From Flow to GenAI: Building RAG pipeline with NiFi and Cloudera AI
Enforcing Flow Governance with Analysis Rules
- Understanding Flow Analysis Rules: Configuration, scope, and enforcement.
- Compliance and Auditing: Leveraging analysis rules for regulatory compliance.
- Hands-on Lab: Creating custom flow analysis rules for data quality and security.
Building custom python processor in NiFi 2.0
- Why python in NiFi 2.0
- Python Processor Deep Dive: Developing custom processors with Python.
- Managing dependencies
- Hands-on Lab: Building a Python-based processor in NiFi 2.0
NiFi 2.0 Upgrade and Migration Strategies
- Why migrate to Cloudera Flow Management 4
- Upgrade Planning: Assessing the impact and identifying potential issues.
- Cloudera Flow Management Migration tool Overview
- Flow Migration Tool: Automating the conversion of NiFi 1.x flows to NiFi 2.0.
- Demo - Variables Migration
- Demo - Templates Migration
- Demo - Component Migration
Appendix: Monitoring for NiFi 2.0
- Monitoring needs of NiFi 2.0
- Prometheus grafana Integration
- Datadog & ELK stack integration
- Performance Management.
- Alerting and Metric threshold.