Sentry Policy File Authorization
Sentry enables role-based, fine-grained authorization for HiveServer2, Cloudera Impala and Cloudera Search.
For more information on installing, upgrading and configuring policy file authorization, see:
- CDH 4.3.0 or later.
- HiveServer2 and the Hive Metastore running with strong authentication. For HiveServer2, strong authentication is either Kerberos or LDAP. For the Hive Metastore, only Kerberos is considered strong authentication (to override, see Securing the Hive Metastore).
- Impala 1.2.1 (or later) running with strong authentication. With Impala, either Kerberos or LDAP can be configured to achieve strong authentication. Auditing of authentication failures is supported only with CDH 4.4.0 and Impala 1.2.1 or later.
- Implement Kerberos authentication on your cluster. This is to prevent a user bypassing the authorization and gaining direct access to the underlying data.
- An object is an entity protected by Sentry's authorization rules. The objects supported in the current release are server, database, table, and URI.
- A role is a collection of rules for accessing a given Hive object.
- A privilege is granted to a role to govern access to an object. Supported privileges are:
Valid privilege types and the objects they apply to Privilege Object INSERT SEVER, DB, TABLE SELECT DB, TABLE ALL SERVER, TABLE, DB, URI
- A user is an entity that is permitted by the authentication subsystem to access the Hive service. This entity can be a Kerberos principal, an LDAP userid, or an artifact of some other pluggable authentication system supported by HiveServer2.
- A group connects the authentication system with the authorization system. It is a collection of one or more users who have been granted one or more authorization roles. Sentry allows a set of roles to be configured for a group.
- A configured group provider determines a user’s affiliation with a group. The current release supports HDFS-backed groups and locally configured groups.
- Allows any user to execute show function, desc function, and show locks.
- Allows the user to see only those tables and databases for which this user has privileges.
- Requires a user to have the necessary privileges on the URI to execute HiveQL operations that take in a location. Examples of such operations include LOAD, IMPORT, and EXPORT.
- Privileges granted on URIs are recursively applied to all subdirectories. That is, privileges only need to be granted on the parent directory.
For more information, see Authorization Privilege Model for Hive and Impala.
server=server1->db=sales->table=customer->action=SelectEach object must be specified as a hierarchy of the containing objects, from server to table, followed by the privilege granted for that object. A role can contain multiple such rules, separated by commas. For example, a role might contain the Select privilege for the customer and items tables in the sales database, and the Insert privilege for the sales_insights table in the reports database. You would specify this as follows:
sales_reporting = \server=server1->db=sales->table=customer->action=Select, \server=server1->db=sales->table=items>action=Select, \server=server1->db=reports->table=sales_insights>action=Insert
User to Group Mapping
You can configure Sentry to use either Hadoop groups or groups defined in the policy file. By default, Sentry looks up groups locally, but it can be
configured to look up Hadoop groups using LDAP (for Active Directory). Local groups will be looked up on the host Sentry runs on. For Hive, this will be the host running HiveServer2. Group mappings
in Sentry can be summarized as in the figure below:
The sections that follow contain notes on creating and maintaining the policy file, and using URIs to load external data and JARs.
Storing the Policy File
Considerations for storing the policy file(s) in HDFS include:
- Replication count - Because the file is read for each query in Hive and read once every five minutes by all Impala daemons, you should increase this value; since it is a small file, setting the replication count equal to the number of client nodes in the cluster is reasonable.
- Updating the file - Updates to the file are reflected immediately, so you should write them to a temporary copy of the file first, and then replace the existing file with the temporary one after all the updates are complete. This avoids race conditions caused by reads on an incomplete file.
role1 = privilege1 role1 = privilege2Role names are scoped to a specific file. For example, if you give role1 the ALL privilege on db1 in the global policy file and give role1 ALL on db2 in the per-db db2 policy file, the user will be given both privileges.
Any command which references a URI such as CREATE TABLE EXTERNAL, LOAD, IMPORT, EXPORT, and more, in addition to CREATE TEMPORARY FUNCTION requires the URI privilege. This is an important security control because without this users could simply create an external table over an existing table they do not have access to and bypass Sentry.
URIs must start with either hdfs:// or file://. If a URI starts with anything else, it will cause an exception and the policy file will be invalid.
data_read = server=server1->uri=file:///path/to/dir,\ server=server1->uri=hdfs://namenode:port/path/to/dir
- Load data from a local/NFS directory:
- Load data from HDFS (MapReduce, Pig, and so on):
In addition to the privilege in Sentry, the hive or impala user will require the appropriate file permissions to access the data being loaded. Groups can be used for this purpose. For example, create a group hive-users, and add the hive and impala users along with the users who will be loading data, to this group.
$ groupadd hive-users $ usermod -G someuser,hive-users someuser $ usermod -G hive,hive-users hive
External tables require the ALL@database privilege in addition to the URI privilege. When data is being inserted through the EXTERNAL TABLE statement, or is referenced from an HDFS location outside the normal Hive database directories, the user needs appropriate permissions on the URIs corresponding to those HDFS locations. This means that the URI location must either be owned by the hive:hive user OR the hive/impala users must be members of the group that owns the directory.
[roles] someuser_home_dir_role = server=server1->uri=hdfs://ha-nn-uri/user/someuserYou should now be able to create an external table:
CREATE EXTERNAL TABLE ... LOCATION 'hdfs://ha-nn-uri/user/someuser/mytable';
Sample Sentry Configuration Files
This section provides a sample configuration.
The following is an example of a policy file with a per-DB policy file. In this example, the first policy file, sentry-provider.ini would exist in HDFS; hdfs://ha-nn-uri/etc/sentry/sentry-provider.ini might be an appropriate location. The per-DB policy file is for the customer's database. It is located at hdfs://ha-nn-uri/etc/sentry/customers.ini.
[databases] # Defines the location of the per DB policy file for the customers DB/schema customers = hdfs://ha-nn-uri/etc/sentry/customers.ini [groups] # Assigns each Hadoop group to its set of roles manager = analyst_role, junior_analyst_role analyst = analyst_role jranalyst = junior_analyst_role customers_admin = customers_admin_role admin = admin_role [roles] # The uris below define a define a landing skid which # the user can use to import or export data from the system. # Since the server runs as the user "hive" files in that directory # must either have the group hive and read/write set or # be world read/write. analyst_role = server=server1->db=analyst1, \ server=server1->db=jranalyst1->table=*->action=select server=server1->uri=hdfs://ha-nn-uri/landing/analyst1 junior_analyst_role = server=server1->db=jranalyst1, \ server=server1->uri=hdfs://ha-nn-uri/landing/jranalyst1 # Implies everything on server1 -> customers. Privileges for # customers can be defined in the global policy file even though # customers has its only policy file. Note that the Privileges from # both the global policy file and the per-DB policy file # are merged. There is no overriding. customers_admin_role = server=server1->db=customers # Implies everything on server1. admin_role = server=server1
[groups] manager = customers_insert_role, customers_select_role analyst = customers_select_role [roles] customers_insert_role = server=server1->db=customers->table=*->action=insert customers_select_role = server=server1->db=customers->table=*->action=select
Sentry Configuration File
The following is an example of a sentry-site.xml file.
<configuration> <property> <name>hive.sentry.provider</name> <value>org.apache.sentry.provider.file.HadoopGroupResourceAuthorizationProvider</value> </property> <property> <name>hive.sentry.provider.resource</name> <value>/path/to/authz-provider.ini</value> <!-- If the hdfs-site.xml points to HDFS, the path will be in HDFS; alternatively you could specify a full path, e.g.: hdfs://namenode:port/path/to/authz-provider.ini file:///path/to/authz-provider.ini --> </property> <property> <name>sentry.hive.server</name> <value>server1</value> </property> </configuration>
Accessing Sentry-Secured Data Outside Hive/Impala
However, there are scenarios where fully vetted and reviewed jobs will also need to access the data stored in the Hive warehouse. A typical scenario would be a secured MapReduce transformation job that is executed automatically as an application user. In such cases it's important to know that the user executing this job will also have full access to the data in the Hive warehouse.
Scenario One: Authorizing Jobs
A reviewed, vetted, and automated job requires access to the Hive warehouse and cannot use Hive/Impala to access the data.
Create a group which contains hive, impala, and the user executing the automated job. For example, if the etl user is executing the automated job, you can create a group called hive-users which contains the hive, impala, and etl users.
$ groupadd hive-users $ usermod -G hive,impala,hive-users hive $ usermod -G hive,impala,hive-users impala $ usermod -G etl,hive-users etlOnce you have added users to the hive-users group, change directory permissions in the HDFS:
$ hadoop fs -chgrp -R hive:hive-users /user/hive/warehouse $ hadoop fs -chmod -R 770 /user/hive/warehouse
Scenario Two: Authorizing Group Access to Databases
One group of users, grp1 should have full access to the database, db1, outside of Sentry. The database, db1 should not be accessible to any other groups, outside of Sentry. Sentry should be used for all other authorization needs.
$ usermod -G hive,impala,grp1 hive $ usermod -G hive,impala,grp1 impalaThen change group ownerships of all directories and files in db1 to grp1, and modify directory permissions in the HDFS. This example is only applicable to local groups on a single host.
$ hadoop fs -chgrp -R hive:grp1 /user/hive/warehouse/db1.db $ hadoop fs -chmod -R 770 /user/hive/warehouse/db1.db
Debugging Failed Sentry Authorization Requests
- In Cloudera Manager, add log4j.logger.org.apache.sentry=DEBUG to the logging settings for your service through the corresponding Logging Safety Valve field for the Impala, Hive Server 2, or Solr Server services.
- On systems not managed by Cloudera Manager, add log4j.logger.org.apache.sentry=DEBUG to the log4j.properties file on each host in the cluster, in the appropriate configuration directory for each service.
FilePermission server..., RequestPermission server...., result [true|false]which indicate each evaluation Sentry makes. The FilePermission is from the policy file, while RequestPermission is the privilege required for the query. A RequestPermission will iterate over all appropriate FilePermission settings until a match is found. If no matching privilege is found, Sentry returns false indicating "Access Denied" .
Authorization Privilege Model for Hive and Impala
Privileges can be granted on different objects in the Hive warehouse. Any privilege that can be granted is associated with a level in the object hierarchy. If a privilege is granted on a container object in the hierarchy, the base object automatically inherits it. For instance, if a user has ALL privileges on the database scope, then (s)he has ALL privileges on all of the base objects contained within that scope.
Object Hierarchy in Hive
Server URI Database Table Partition Columns View Index Function/Routine Lock
|ALL||SERVER, TABLE, DB, URI|
|Base Object||Granular privileges on object||Container object that contains the base object||Privileges on container object that implies privileges on the base object|
|CREATE VIEW||DATABASE; SELECT on TABLE||ALL||SELECT on TABLE|
|ALTER TABLE .. ADD COLUMNS||TABLE||ALL|
|ALTER TABLE .. REPLACE COLUMNS||TABLE||ALL|
|ALTER TABLE .. CHANGE column||TABLE||ALL|
|ALTER TABLE .. RENAME||TABLE||ALL|
|ALTER TABLE .. SET TBLPROPERTIES||TABLE||ALL|
|ALTER TABLE .. SET FILEFORMAT||TABLE||ALL|
|ALTER TABLE .. SET LOCATION||TABLE||ALL||URI|
|ALTER TABLE .. ADD PARTITION||TABLE||ALL|
|ALTER TABLE .. ADD PARTITION location||TABLE||ALL||URI|
|ALTER TABLE .. DROP PARTITION||TABLE||ALL|
|ALTER TABLE .. PARTITION SET FILEFORMAT||TABLE||ALL|
|SHOW CREATE TABLE||TABLE||SELECT/INSERT|
|INSERT OVERWRITE TABLE||TABLE||INSERT|
|CREATE TABLE .. AS SELECT||DATABASE; SELECT on TABLE||ALL||SELECT on TABLE|
|ALTER TABLE .. SET SERDEPROPERTIES||TABLE||ALL|
|ALTER TABLE .. PARTITION SET SERDEPROPERTIES||TABLE||ALL|
|INSERT OVERWRITE DIRECTORY||TABLE||INSERT||URI|
|Analyze TABLE||TABLE||SELECT + INSERT|
|ALTER TABLE TOUCH||TABLE||ALL|
|ALTER TABLE TOUCH PARTITION||TABLE||ALL|
|ALTER TABLE .. CLUSTERED BY SORTED BY||TABLE||ALL|
|ALTER TABLE .. ENABLE/DISABLE||TABLE||ALL|
|ALTER TABLE .. PARTITION ENABLE/DISABLE||TABLE||ALL|
|ALTER TABLE .. PARTITION.. RENAME TO PARTITION||TABLE||ALL|
|MSCK REPAIR TABLE||TABLE||ALL|
|GRANT PRIVILEGE||Allowed only for Sentry admin users|
|REVOKE PRIVILEGE||Allowed only for Sentry admin users|
|SHOW GRANT||Allowed only for Sentry admin users|
|DESCRIBE TABLE .. PARTITION||TABLE||SELECT/INSERT|
|ADD JAR||Not Allowed|
|ADD FILE||Not Allowed|
|INVALIDATE METADATA <table name>||TABLE||SELECT/INSERT|
|REFRESH <table name>||TABLE||SELECT/INSERT|