X

Cloudera Tutorials

Optimize your time with detailed tutorials that clearly explain the best way to deploy, use, and manage Cloudera products. Login or register below to access all Cloudera tutorials.

Cloudera named a leader in 2022 Gartner® Magic Quadrant™ for Cloud Database Management Systems Get the report
    • Cloudera Cloudera
    • X
    Call us at(888) 789-1488
    International:+1 (650) 362-0488
    Email Sales
    Support Portal

    Ready to Get Started?

     

    NOTICE

     

    As of January 31, 2021, this tutorial references legacy products that no longer represent Cloudera’s current product offerings.

    Please visit recommended tutorials:

     

    Introduction

    Security Breaches are common problem for businesses with the question of when it will happen? One of the first lines of defense for detecting potential vulnerabilities in the network is to investigate the logs from your server. You have been brought on to apply your skills in Data Engineering and Data Analysis to acquire server log data, preprocess the data and store it into reliable distributed storage HDFS using the dataflow management framework Apache NiFi. You will need to further clean and refine the data using Apache Spark for specific insights into what activities are happening on your server, such as most frequent hosts hitting the server and which country or city causes the most network traffic with your server. You will then visualize these events using the data science notebook Apache Zeppelin to be able to tell a story to about the activities occurring on the server and if there is anything your team should be cautious about.

    Big Data Technologies used to develop the Application:

     

    Goals and Objectives

    • Learn about server log data, log data analysis, how it works, the various use cases and best practices
    • Learn to build a NiFi dataflow to acquire server log data
    • Learn to clean the data for filtering down to messages that can tell users about the activities happening on their servers using Spark
    • Learn to visualization your finding after cleaning the data using Zeppelin visualization

    Prerequisites

    Outline

    The tutorial series consists of the following tutorial modules:

    1. Application Development Concepts: Covers what is server log data, log data analysis, how log data analysis works, various use cases and some best practices that can be used in server log analysis.

    2. Setting up the Development Environment: You will perform any configurations on software services and/or install dependencies for software services that are needed to develop the application.

    3. Acquiring NASA Server Log Data: You will learn to build a NiFi dataflow that acquires 2 months worth of NASA log data, preprocesses the data and stores it into HDFS

    4. Cleaning the Raw NASA Log Data: You will learn to create a Zeppelin Notebook for cleaning the NASA log data and use Zeppelin's Spark Interpreter to clean the data and gather any valuable insight about the activities going on with the server.

    5. Visualizing NASA Log Data: You will create another Zeppelin Notebook whose purpose will be to visualize the key points you found when cleaning the data with Spark. Your data visualization will illustrate from the NASA log data, the Most Frequent Hosts - count per IP address of hosts hitting the server, Response Codes - count per response code in association with the server, Type of Extensions - count of the type of file formats being transferred between devices interacting with the server, Network Traffic per Location - location on where the server hits are coming from.



    Your form submission has failed.

    This may have been caused by one of the following:

    • Your request timed out
    • A plugin/browser extension blocked the submission. If you have an ad blocking plugin please disable it and close this message to reload the page.