New Features and Changes in Cloudera Navigator Data Management

These two sections on this page describe what's new and what has changed in each Cloudera Navigator release:

New Features in Cloudera Navigator

The following sections describe what's new in each Cloudera Navigator release.

What's New in Cloudera Navigator 2.10.1

This release of Cloudera Navigator data management includes no new features. See Issues Fixed in Cloudera Navigator 2.10.1 for resolved issues and see Known Issues and Workarounds in Cloudera Navigator Data Management for known issues.

What's New in Cloudera Navigator 2.10.0

In addition to these new features and the changes listed below, several issues have been resolved in this release. See Issues Fixed in Cloudera Navigator 2.10.0.
  • Data Stewardship Dashboard. The Data Stewardship Dashboard is fully supported and enabled by default in this release. Originally introduced in Cloudera Navigator 2.9 but disabled by default, the Data Stewardship Dashboard displays summary activity about data entities—tables created, tables altered, databases created, SQL queries started, for example. The time frame for the summary is selectable using the Dashboard's trend widget (daily, weekly, monthly, quarterly) to instantly adjust the overall picture of your data. Displayed below the summary results are more detailed views of the data, ranked by count and listing details for the entities as captured by the Navigator Metadata Server. For example, top files by size lists the top 10 files by file name, path, file size, with graphical bar chart rendering for easy visual comparison.
  • Data Explorer. The Data Explorer is an interactive companion to the Data Stewardship Dashboard for viewing and comparing selected data sources over time in terms of averages or trends. Users can select any of the various data sources (databases created, tables created, tables altered, and so on), select chart tools (average, trendline), and select a time period over which to render the data.
  • Spark (Spark 1.6) Lineage Support. The collection mechanism for Spark job lineage has been enhanced to capture Spark application execution—input data, output data, and column-level when possible for each invocation of spark-submit. In addition, lineage for Spark jobs is now collected automatically. Support for this capability in the prior release required manual configuration using a safety valve (advanced configuration snippet), Currently, support is for Spark 1.6 (not Spark 2) and lineage is collected for data read and written using Spark Dataframes and SparkSQL only (RDDs not yet supported). See Known Issues and Workarounds in Cloudera Navigator Data Management for other limitations.
  • Integration with Navigator for Hue for metadata discovery. Search and tag partitions, databases, views, tables, columns. This capability must be enabled. See How to Enable and Use Navigator in Hue for details.
  • Improved logging for Amazon S3 data. Navigator Server logs now capture information (INFO) about Amazon S3 buckets from which the server is both collecting and not collecting metadata. This enhancement can facilitate better root cause analysis for issues that may arise.

What's New in Cloudera Navigator 2.9.2

There are no new features in Cloudera Navigator 2.9.2.

What's New in Cloudera Navigator 2.9.1

A number of issues have been fixed. See Issues Fixed in Cloudera Navigator 2.9.1.

What's New in Cloudera Navigator 2.9.0

  • Navigator can now extract technical metadata from data stored on Amazon S3. You can filter results based on the S3 source type and other related S3 properties. See Navigator and S3.
  • Solr indexing has been optimized to improve search speed and reduce memory requirements. When Cloudera Navigator is started after an upgrade, you will see a message indicating upgrade progress. See Upgrading the Cloudera Navigator Management Component.
  • Managed metadata properties editing has been enhanced. Properties can now be individually viewed, updated, or deleted. Deleted properties are still available from the Administration page and can be restored until they are purged and permanently removed.
  • Policy editor has been enhanced:
    • All types of managed metadata can be assigned (text, number, Boolean, date, and enumeration) instead of only text.
    • UI support for managed metadata assignment has been added.
    • Managed metadata assignments can now be multi-valued.
  • Auditing health check - You can configure an auditing pipeline health check to verify that auditing is working and is not silently down. See Verifying that Auditing Is Running.
  • You can now specify which columns to export when you export an audit report.
  • The overall look and feel of the user interface for Cloudera Navigator has been updated.
  • Navigator Data Stewardship dashboard - Cloudera Navigator 2.9 introduces the Data Stewardship dashboard, which provides "at-a-glance" information and metrics to help you understand the state of the data and data usage. The dashboard displays information about table, file, and database activity, operations and operation executions, and other information captured by the Navigator Metadata Server. The dashboard is currently unsupported. See Data Stewardship Dashboard.

What's New in Cloudera Navigator 2.8.1

A number of issues have been fixed. See Issues Fixed in Cloudera Navigator 2.8.1.

What's New in Cloudera Navigator 2.8.0

A number of issues have been fixed. See Issues Fixed in Cloudera Navigator 2.8.0.

What's New in Cloudera Navigator 2.7.5

There are no new features or fixed issues.

What's New in Cloudera Navigator 2.7.4

There are no new features or fixed issues.

What's New in Cloudera Navigator 2.7.3

An issue has been fixed. See Issues Fixed in Cloudera Navigator 2.7.3.

What's New in Cloudera Navigator 2.7.2

A number of issues have been fixed. See Issues Fixed in Cloudera Navigator 2.7.2.

What's New in Cloudera Navigator 2.7.1

An issue has been fixed. See Issues Fixed in Cloudera Navigator 2.7.1.

What's New in Cloudera Navigator 2.7.0

  • Platform enhancements
    • Added support for assigning managed metadata through policies. Only single-valued managed metadata of type Text is currently supported. See Metadata Policies.
    • Added support for purging physical operations corresponding to logical operations. See Performing Actions on Entities.
  • Cross Site Request Forgery (CSRF) protection is enabled by default. If you currently use cookies for authentication in API requests with a non-GET method (for example, the POST method), you must also obtain CSRF tokens for the request, or disable CSRF protection.

    You can disable CSRF protection by setting nav.disable_api_csrf_security=true.

A number of issues have been fixed. See Issues Fixed in Cloudera Navigator 2.7.0

What's New in Cloudera Navigator 2.6.6

There are no new features or fixed issues.

What's New in Cloudera Navigator 2.6.5

A number of issues have been fixed. See Issues Fixed in Cloudera Navigator 2.6.5

What's New in Cloudera Navigator 2.6.4

There are no new features or fixed issues.

What's New in Cloudera Navigator 2.6.3

There are no new features or fixed issues.

What's New in Cloudera Navigator 2.6.2

A number of issues have been fixed. See Issues Fixed in Cloudera Navigator 2.6.2.

What's New in Cloudera Navigator 2.6.1

A number of issues have been fixed. See Issues Fixed in Cloudera Navigator 2.6.1.

What's New in Cloudera Navigator 2.6.0

A number of issues have been fixed. See Issues Fixed in Cloudera Navigator 2.6.0.

What's New in Cloudera Navigator 2.5.1

What's New in Cloudera Navigator 2.5.0

An issue has been fixed. See Issues Fixed in Cloudera Navigator 2.5.0.

What's new in Cloudera Navigator 2.4.6

What's New in Cloudera Navigator 2.4.4

Several issues have been fixed. See Issues Fixed in Cloudera Navigator 2.4.4.

What's New in Cloudera Navigator 2.4.3

What's New in Cloudera Navigator 2.4.2

A number of issues have been fixed. See Issues Fixed in Cloudera Navigator 2.4.2.

What's New in Cloudera Navigator 2.4.1

An issue has been fixed. See Issues Fixed in Cloudera Navigator 2.4.1.

What's New in Cloudera Navigator 2.4.0

  • Platform enhancements
    • New entity details page.
    • HDFS metadata and audit analytics.
    • Command actions, integrated with policies and search, for archiving and purging files.
    • Redesigned policies page for scalability.
    • Publishing of audit events to Kafka topics. Kafka audit event publishing does not support authorization and Cloudera does not recommend its use in production.
    • Navigator SDK for metadata and lineage augmentation.
    • Significant scale improvements. Large lineage graphs render faster and consume less memory.
  • Expanded service coverage
    • Hive metadata: support for extended attributes.
    • Hive on Spark metadata and lineage.
    • Oozie metadata and lineage added support for the hive2 action, which Cloudera recommends that you use in preference to the hive action.
    • Hue auditing (CDH 5.5.0 and higher): added login and logout, user and group events.
    • Cloudera Manager auditing: login and logout events.
    • Navigator Metadata Server auditing: successful and failed login events.
    • Added filtering of audit events for Sentry, Solr, Impala, and Navigator Metadata Server.
Also see What's New in Cloudera Manager 5.5.0.

What's New in Cloudera Navigator 2.3.10

What's New in Cloudera Navigator 2.3.9

An issue has been fixed. See Issues Fixed in Cloudera Navigator 2.3.9.

What's New in Cloudera Navigator 2.3.8

Several issues have been fixed. See Issues Fixed in Cloudera Navigator 2.3.8.

What's New in Cloudera Navigator 2.3.3

Several issues have been fixed. See Issues Fixed in Cloudera Navigator 2.3.3.

What's New in Cloudera Navigator 2.3.1

  • Navigator self audit events have been enhanced with additional information such as the names of audit reports and policies
  • Performance and stability improvements
Also, a number of issues have been fixed. See Issues Fixed in Cloudera Navigator 2.3.1.

What's New in Cloudera Navigator 2.3.0

  • Platform enhancements
    • Redesigned metadata search provides autocomplete, enhanced filtering, and saving searches.
    • Added support for SAML for single sign-on.
  • Expanded service coverage
    • Added Impala (CDH 5.4 and higher) metadata and lineage
    • Added Cloudera Search (CDH 5.4 and higher) auditing
    • Added auditing for Navigator Metadata Server activity, such as audit views, metadata searches, and policy editing
    • Added support for inferring the schema of HDFS Avro and Parquet entities
    • Added Spark (CDH 5.4 and higher) metadata and lineage.

What's New in Cloudera Navigator 2.2.10

What's New in Cloudera Navigator 2.2.9

An issue has been fixed. See Issues Fixed in Cloudera Navigator 2.3.9.

What's New in Cloudera Navigator 2.2.4

An issue has been fixed. See Issues Fixed in Cloudera Navigator 2.2.4.

What's New in Cloudera Navigator 2.2.3

An issue has been fixed. See Issues Fixed in Cloudera Navigator 2.2.3.

What's New in Cloudera Navigator 2.2.2

An issue has been fixed. See Issues Fixed in Cloudera Navigator 2.2.2.

What's New in Cloudera Navigator 2.2.1

A number of issues have been fixed. See Issues Fixed in Cloudera Navigator 2.2.1.

What's New in Cloudera Navigator 2.2.0

  • Policies are generally available and are always enabled. Policy properties now support Java expressions.
  • Search - Search functionality now includes autocomplete.

What's New in Cloudera Navigator 2.1.6

An issue has been fixed. See Issues Fixed in Cloudera Navigator 2.1.6.

What's New in Cloudera Navigator 2.1.5

A number of issues have been fixed. See Issues Fixed in Cloudera Navigator 2.1.5.

What's New in Cloudera Navigator 2.1.4

An issue has been fixed. See Issues Fixed in Cloudera Navigator 2.1.4.

What's New in Cloudera Navigator 2.1.2

A number of issues have been fixed. See Issues Fixed in Cloudera Navigator 2.1.2.

What's New in Cloudera Navigator 2.1.1

A number of issues have been fixed. See Issues Fixed in Cloudera Navigator 2.1.1.

What's New in Cloudera Navigator 2.1.0

  • Auditing Component
    • New auditing UI featuring saved audit reports. The Navigator auditing UI is no longer available from Cloudera Manager. Instead, auditing is integrated with lineage, discovery, and the policy engine. The UI is provided by the Metadata Server, which is now required for the auditing component.
    • Sentry auditing now includes Sentry commands issued from Impala.
  • Metadata Component
    • Search results contain a type appropriate link to a Hue browser.
      • HDFS directories and files - File Browser
      • Hive database and tables - Metastore Manager
      • MapReduce, YARN, Pig - Job Browser
    • Policies - support rules for modifying metadata and sending notifications when entities are extracted.
  • Security
    • Role-Based Access Control - support assigning groups to roles that constrain access to Navigator features
    • Authentication - LDAP and Active Directory authentication of Navigator users
    • TLS/SSL - enable TLS/SSL for encrypted communication
  • API
    • Version changed to v3
    • Supports auditing and policies

What's New in Cloudera Navigator 2.0.5

An issue was fixed. See Issues Fixed in Cloudera Navigator 2.0.5.

What's New in Cloudera Navigator 2.0.3

A number of issues have been fixed. See Issues Fixed in Cloudera Navigator 2.0.3.

What's New in Cloudera Navigator 2.0.2

An issue was fixed. See Issues Fixed in Cloudera Navigator 2.0.2.

What's New in Cloudera Navigator 2.0.1

  • Masking of personally identifiable information (PII) in query strings that appear in audit events and lineage. Enabled by default.
  • REST API support for registering custom metadata for entities before they appear in Navigator.
A number of issues have been fixed. See Issues Fixed in Cloudera Navigator 2.0.1.

What's New in Cloudera Navigator 2.0.0

  • Auditing Component
    • Added support for auditing the Sentry service
    • Added support for publishing audit logs to syslog
  • Metadata Component
    • Newly designed Query Builder with faceted filtering
    • Simplified Pig lineage
    • Added support for Sqoop and Oozie lineage
    • Many performance and stability improvements
  • Security - includes Navigator Encrypt and Navigator Key Trustee, formerly known as Gazzang zNcrypt and Gazzang zTrustee. These features provide enterprise-grade encryption and key management. For information on these features, see the Cloudera Security Datasheet and contact your account team.

Changed Features in Cloudera Navigator

The sections below describe changes to functionality, if any, for each Cloudera Navigator release.

What's Changed in Cloudera Navigator 2.10.1

This release of Cloudera Navigator data management includes no change features. See Issues Fixed in Cloudera Navigator 2.10.1 for resolved issues and see Known Issues and Workarounds in Cloudera Navigator Data Management for known issues.

What's Changed in Cloudera Navigator 2.10.0

  • Cloudera Navigator Data Stewardship Analytics (Data Stewardship Dashboard) is now fully enabled. See Data Stewardship Dashboard (above) for details.
  • Configurable aggregation scheme for metrics. Every 120 minutes, an internal task runs to extract, transform, and load various counts collected by Navigator to derive metrics and populate the Data Stewardship Dashboard with values. By default, data is aggregated for a 1-day time period. However, the window can be changed by setting the Advanced Configuration Snippet (Safety Valve). The values for new partitions, new tables, and new files depend on the value for this window.
    Safety Valve Description
    nav.dashboard.etl.maxDataSetSize=n Cloudera Manager 5.11/Navigator 2.10 (and later): Default window size is 1 (1 day)
    Cloudera Manager 5.10/Navigator 2.9 (and prior): Default window size is 7 days
    nav.dashboard.etl.include.hdfs=true Enables the Access Denied field on the Dashboard to display values.
    By default, the aggregated Access Denied metric is not populated but you can enable its display by setting the safety valve. To add this safety valve to your system, or to change the default window for the aggregation calculation:
    1. Log in to Cloudera Manager Admin Console.
    2. Navigate to Cloudera Management Service and then select Navigator Metadata Server (under Scope filter).
    3. Click Configuration and then click Service Configuration.
    4. Scroll to the Navigator Metadata Server Advanced Configuration Snippet (Safety Valve) for cloudera-navigator.properties.
    5. Enter one or both safety valves into the field as needed. For example:
      nav.dashboard.etl.maxDataSetSize=3
      nav.dashboard.etl.include.hdfs=true

What's Changed in Cloudera Navigator 2.9.0

Name and description fields, which were previously in custom metadata, have been moved to managed metadata.

What's Changed in Cloudera Navigator 2.7.0

Previously, Cloudera Navigator captured the underlying MapReduce and YARN operations for Hive, Sqoop, and Pig operations, and stored them as physical operations. As of release 2.7.0, Cloudera Navigator removes previously captured physical operations, operation executions, and associated relations during purge. Removing these physical operations increases the overall scalability of Cloudera Navigator.

During the upgrade to Cloudera Navigator 2.7, Cloudera Navigator removes previously captured physical operations, operation executions, and associated relations.

What's Changed in Cloudera Navigator 2.6.0

  • Queries generated using filters now use the Lucene Query Parser ,+, - Boolean operator syntax. See Constructing Compound Search Strings.
  • Hive on Spark metadata extraction is now supported.
  • The Navigator Metadata Server requires 192 MiB of Java PermGen space instead of 128 MiB. This is increased automatically when upgrading to Cloudera Manager 5.7.

What's Changed in Cloudera Navigator 2.4.0

  • Cloudera Navigator no longer supports JDK 1.6.
  • In the Administration tab, after roles have been assigned to at least one group, Groups with Navigator Roles is the default selection.

What's Changed in Cloudera Navigator 2.3.3

  • In the Search UI, facet values with the count of 0 are not displayed.

What's Changed in Cloudera Navigator 2.3.0

  • The PII masking regular expression is superseded by log file redaction in CDH 5.4.

What's Changed in Cloudera Navigator 2.2.0

  • Metadata Component - Policies created with Cloudera Navigator 2.1 (containing the Beta version policy engine) are not retained when upgrading to Cloudera Navigator 2.2.

What's Changed in Cloudera Navigator 2.1.0

  • Auditing Component
    • HDFS audit events generated by the impala user are discarded by default.

What's Changed in Cloudera Navigator 2.0.1

  • Metadata Component
    • The REST API version changed to v2.

What's Changed in Cloudera Navigator 2.0.0

  • Auditing Component
    • HDFS audit events generated by the solr user are discarded by default.