Creating and Using your own Repository

Custom repositories leverage package management solutions, which were designed to provide a convenient, efficient means to install software on many computers. This topic describes how to create a local package repository and then how to direct machines in your environment to use that repository.

To create a repository, you simply put the RPMs or DEBs you want to host in one directory. Then complete tasks with createrepo or reprepro, and then publish the resulting repository on a website.

Step 1: Download Installation Files

Creating a custom repository requires RPM or DEB files. If you already have these files, proceed to the next step. Otherwise, download the appropriate versions of Cloudera Manager. Available versions can be found at Cloudera Manager Version and Download Information.

Step 2: Prepare the RPM or DEB Files

Move the RPMs or DEBs to a directory that you will use for your repository. Suppose you have downloaded the Oracle JDK and want to host it internally to ease installation on various machines. After you have unpacked the RPMs or DEBs, you would have a collection of those files in your directory. For example, for a CentOS system, you might have:

$ls -l
total 79392
-rw-r--r-- 1 user group 70530937 Mar 13 14:42 jdk-6u24-linux-amd64.rpm
-rw-r--r-- 1 user group   499375 Mar 13 14:42 sun-javadb-client-10.6.2-1.1.i386.rpm
-rw-r--r-- 1 user group    14627 Mar 13 14:42 sun-javadb-common-10.6.2-1.1.i386.rpm
-rw-r--r-- 1 user group  4080625 Mar 13 14:42 sun-javadb-core-10.6.2-1.1.i386.rpm
-rw-r--r-- 1 user group   969861 Mar 13 14:42 sun-javadb-demo-10.6.2-1.1.i386.rpm
-rw-r--r-- 1 user group  4865183 Mar 13 14:42 sun-javadb-docs-10.6.2-1.1.i386.rpm
-rw-r--r-- 1 user group   201273 Mar 13 14:42 sun-javadb-javadoc-10.6.2-1.1.i386.rpm

Step 3: Create a Repository

You can using createrepo or reprepro to create a repository. If you don't have createrepo or reprepro installed, install it using the following command:

  • RedHat/CentOS: yum install createrepo
  • SLES: zypper install createrepo
  • Debian/Ubuntu: apt-get install reprepro

Creating a CentOS/RHEL/Oracle/SLES Repository

Having expanded the RPMs to a directory and ensured createrepo is installed, you can now create a repository. When you run createrepo in the directory that contains the files you will use to create a repo, the program creates an extra directory with some XML files that describing the repository. For example:

$ createrepo .
7/7 - sun-javadb-javadoc-10.6.2-1.1.i386.rpm                                    
Saving Primary metadata
Saving file lists metadata
Saving other metadata
$ ls -l
total 79400
-rw-r--r-- 1 user group 70530937 Mar 13 14:42 jdk-6u24-linux-amd64.rpm
drwxr-xr-x 2 user group     4096 Mar 13 14:45 repodata/
-rw-r--r-- 1 user group   499375 Mar 13 14:42 sun-javadb-client-10.6.2-1.1.i386.rpm
-rw-r--r-- 1 user group    14627 Mar 13 14:42 sun-javadb-common-10.6.2-1.1.i386.rpm
-rw-r--r-- 1 user group  4080625 Mar 13 14:42 sun-javadb-core-10.6.2-1.1.i386.rpm
-rw-r--r-- 1 user group   969861 Mar 13 14:42 sun-javadb-demo-10.6.2-1.1.i386.rpm
-rw-r--r-- 1 user group  4865183 Mar 13 14:42 sun-javadb-docs-10.6.2-1.1.i386.rpm
-rw-r--r-- 1 user group   201273 Mar 13 14:42 sun-javadb-javadoc-10.6.2-1.1.i386.rpm
$ ls -l repodata/
total 64
-rw-r--r-- 1 user group 32997 Mar 13 14:45 filelists.xml.gz
-rw-r--r-- 1 user group   482 Mar 13 14:45 other.xml.gz
-rw-r--r-- 1 user group  2429 Mar 13 14:45 primary.xml.gz
-rw-r--r-- 1 user group   951 Mar 13 14:45 repomd.xml

After this command completes, the specified RPMs have been added to your private repository.

  Note:

Over time, you may need to install multiple versions Cloudera software. To achieve this, you may need to update repository information. Do not create a situation where there are multiple repo files that package management tools could use to install Cloudera software. Either overwrite existing repo files with the new information or create a new repo file and delete the old one. If your machines have multiple repo files, installations may not complete as expected.

Creating a Debian/Ubuntu Repository

Having expanded the DEBs to a directory and ensured reprepro is installed, you can now create a repository.

To create a Debian/Ubuntu repository:

Create a directory that will host the repository and creating a configuration directory and file. The repo directory can have any location and name. The configuration directory must be named conf and be at the root level of the repository directory you create. For example:

$ mkdir /tmp/repo
$ mkdir /tmp/repo/conf

Create a configuration file that contains basic information about the repository. For example, after creating such a file, you could check its contents as follows:

$ cat /tmp/repo/conf/distributions
Origin: Cloudera
Label: Cloudera
Suite: stable
Codename: cloudera
Version: 0.1
Architectures: i386 amd64 source
Components: contrib
Description: Cloudera

Run reprepro on DEB files. If you had expanded the DEBs to the /tmp/repo/ directory, you might use the following command:

$ find /tmp/repo -name \*.deb -exec reprepro -Vb repo includedeb cloudera {} \;

After this command completes, the specified DEBs have been added to your private repository.

Step 4: Install a Web Server

The repository is typically hosted using HTTP on a machine inside your network. If you already have a web server in your organization, you can move the repository directory, which will include both the RPMs and the repodata/ subdirectory, to some a location hosted by the web server. If you are able to use an existing web server, then note the URL and skip to Modifying Clients to Find Repos.

An easy web server to install is the Apache HTTPD.

To install Apache HTTPD:

Install Apache HTTPD. You may need to respond to some prompts to confirm you want to complete the installation. For RedHat/CentOS:

[root@localhost yum.repos.d]$ yum install httpd

For SLES:

[root@localhost zypp]$ zypper install httpd

For Debian/Ubuntu:

[root@localhost apt]$ apt-get install httpd

Start Apache HTTPD: For RedHat

[root@localhost tmp]$  service httpd start
Starting httpd:                                            [  OK  ]

For SLES

[root@localhost tmp]$ service apache2 start
Starting httpd:                                            [  OK  ]

For Debian/Ubuntu

[root@localhost tmp]$ service apache2 start
Starting httpd:                                            [  OK  ]

Step 5: Publish Repository Files

Move your files to the web server directory and modify file permissions. For example, you might use the following commands:

[root@localhost tmp]$ mv /tmp/repo /var/www/html
[root@localhost tmp]$ chmod -R ugo+rX /var/www/html/repo

After moving files and changing permissions, visit http://<hostname>:80/repo to verify that you see an index of files. Note that Apache may have been configured to not show indexes, which is also acceptable.

Modifying Clients to Find Repos

Having established the repository, modify the clients so they find the repository.

For RedHat/CentOS systems: Create files on client systems with the following information and format, where hostname is the name of the web server you created in the previous step:

[myrepo]
name=myrepo
baseurl=http://hostname/repo
enabled=1
gpgcheck=0

See man yum.conf for more details. Put that file into /etc/yum.repos.d/myrepo.repo on all of your host machines to enable them to find the packages that you are hosting.

For SLES systems: Use the zypper utility to update client system repo information by issuing the following command:

$ zypper addrepo http://hostname/repo alias

For Debian/Ubuntu systems: Add a new list file to /etc/apt/sources.list.d/ on client systems. For example, you might create the file /etc/apt/sources.list.d/my-private-cloudera-repo.list. In that file, create an entry to your newly created repository. For example:

$ cat /etc/apt/sources.list.d/my-private-cloudera-repo.list
deb http://hostname/repo cloudera

After adding your .list file, ensure apt-get uses the latest information by issuing the following command:

$ sudo apt-get update

After completing these steps, you have established the environment necessary to install a previous version of Cloudera Manager or install Cloudera Manager to machines that are not connected to the Internet. Proceed with the installation process, being sure to target the newly created repository with your package management tool.