Cloudera Tutorials

Optimize your time with detailed tutorials that clearly explain the best way to deploy, use, and manage Cloudera products. Login or register below to access all Cloudera tutorials.

First Name

Last Name

Job Title

Business Email

Company

Phone

Yes, I would like to be contacted by Cloudera for newsletters, promotions, events and marketing activities. Please read our privacy and data policy.

Yes, I consent to my information being shared with Cloudera's solution partners to offer related products and services. Please read our privacy and data policy.

I agree to Cloudera's terms and conditions.

Back to main tutorial page

Fast Start Gen AI with a 15% discount

Introduction

Experience the benefits of having access to a hybrid cloud solution, which provides us to access many resources, including NVIDIA GPUs. Explore how you can leverage NVIDIA's RAPIDS framework using Cloudera Machine Learning (CML), on the Cloudera Data Platform (CDP). Harness the GPU power and see significant speed improvements compared to commonly used machine learning libraries such as pandas, NumPy and Sklearn in both data preprocessing and model training.

Prerequisites

Have access to Cloudera Data Platform (CDP) Public Cloud
Have created a CDP workload User
Ensure proper CML role access
- MLUser: ability to run workloads
- MLAdmin: ability to create and delete workspaces

Outline

Watch Video
Download Assets
Setup Cloudera Machine Learning
Create Project
Create Session
Run Program in Jupyter Notebook
Summary
Further Reading

Watch Video

The video below provides a brief overview of what is covered in this tutorial:

Download Assets

There are two (2) options in getting assets for this tutorial:

Download a ZIP file

It contains only necessary files used in this tutorial. Remember its location. No need to unzip the file.

Clone our GitHub repository

It provides assets used in this and other tutorials; organized by tutorial title.

Setup Cloudera Machine Learning

Provision ML Workspace

If your environment doesn’t already have a Machine Learning Workspace provisioned, let’s provision it.

Select Machine Learning from Cloudera Data Platform (CDP) home page:

In the ML Workspaces section, select Provision Workspace.

Two simple pieces of information are needed to provision an ML workspace - the Workspace name and the Environment name. For example:

Workspace Name: cml-tutorial
Environment: <your environment name>
Select Provision Workspace

NOTE: You may need to activate GPU usage in Advanced Options

Create Resource Profile

Resource profiles define how many vCPUs and how much memory Cloudera Machine Learning will reserve for a particular workload (for example, session, job, model). You must have MLAdmin role access to create a new resource profile.

Let’s create a new resource profile.

Beginning from the ML Workspaces section, open your workspace by selecting its name, cml-tutorial.

In the Site Administration section, select Runtime/Engine.

Create a new resource profile using the following information:

vCPU: 4

Memory (GiB): 32

Select Add

Create Project

Beginning from the ML Workspaces section, open your workspace by selecting its name, cml-tutorial.

Select New Project.

Complete the New Project form using:

Project Name: Fare Prediction
Project Description:
A project showcasing speed improvements using RAPIDS framework.
Initial Setup: Local Files
Upload or Drag-Drop tutorial-files.zip file you downloaded earlier
Select Create Project

Create Session

Beginning from the Projects section, select the project name, Fare Prediction.

Select New Session and complete the session form:

Session Name: cml-rapids
Editor: JupyterLab
Kernel: Python 3.7
Edition: RAPIDS
Resource Profile: 4 vCPU / 32 GiB Memory, 1 GPU
Select Start Session

Run Program in Jupyter Notebook

Let’s open the file gpu_fare_prediction.ipynb by double-clicking the filename.

The default, mode = ‘cpu’, will run the program using only the CPU.

Without making any changes to the program, run all the cells by selecting Kernel > Restart Kernel and Run All Cells..., then click Restart.

The majority of the time is spent calculating the Haversine formula to determine the great-circle distance between pickup/drop-off locations.

Let’s make a simple change to run the program and run it using GPUs - change mode = ‘gpu’. Run all the cells by selecting Kernel > Restart Kernel and Run All Cells..., then click Restart.

Notice the time spent calculating the Haversine formula is now insignificant due to the multi-threaded functionality of the Rapids framework.

What changed? If you look at the import statements, located in the Program Initialization section, you’ll notice that we bound GPU libraries to the same name as we bound the CPU libraries.
For example, instead of import pandas as pd we used import cudf as pd when mode = ‘gpu’.
The syntax between the two libraries are so similar, that in most cases you can simply switch out your common machine learning libraries and achieve up to 60x performance improvements.

The instances where the syntax is different between the CPU libraries and the RAPIDS GPU libraries can be found where there are if statements in the notebook. There are only two of them. The first is when importing date type columns from a CSV. The second is when applying a user-defined function.

Summary

Congratulations on completing the tutorial.

As you’ve now experienced, the NVIDIA’s RAPIDS framework has a familiar look and feel to other common machine learning libraries. This is great because no major code re-write is needed to obtain the benefits of the RAPIDS libraries. It allowed us to harness the GPU power and see significant speed improvements compared to other commonly used machine learning libraries.