Period Stacks Collection
Periodic stacks collection allows you to enable and configure the periodic collection of thread stack traces in Cloudera Manager. When stacks collection is enabled for a role, call stacks are output to a log file at regular intervals. The logs can help with diagnosis of performance issues such as deadlock, slow processing, or excessive numbers of threads.
Stacks collection may impact performance for the processes being collected as well as other processes on the host, and is turned off by default. For troubleshooting performance issues, you may be asked by Cloudera Support to enable stacks collection and send the resulting logs to Cloudera for analysis.
Stacks collection is available for the majority of roles in Cloudera Manager. For the HDFS service, for example, you can enable stacks collection for the DataNode, NameNode, Failover Controller, HttpFS, JournalNode, and NFS Gateway. If the Stacks Collection category does not appear in the role's configuration settings, the feature is not available for that role.
Configuring Period Stacks Collection
- From the service page in Cloudera Manager, click the Configuration tab, and expand the corresponding role category in the left-hand pane, and click the Stacks Collection category.
- From the service page in Cloudera Manager, click the Instances tab, click the role type, click the Configuration tab on the role page, expand the role name in the left-hand pane, and click the Stacks Collection category.
- Stacks Collection Enabled - Whether or not periodic stacks collection is enabled.
- Stacks Collection Directory - The directory in which stack logs will be placed. If not set, stacks will be logged into a stacks subdirectory of the role's log directory.
- Stacks Collection Frequency - The frequency with which stacks will be collected.
- Stacks Collection Data Retention - The amount of stacks data that will be retained. When the retention limit is reached, the oldest data will be deleted.
- Stacks Collection Method - The method that will be used to collect stacks. The jstack option involves periodically running the jstack command against the role's daemon process. The servlet method is available for those roles with an HTTP server endpoint that exposes the current stacks traces of all threads. When the servlet method is selected, that HTTP endpoint is periodically scraped.
- On the HDFS service page, click Configuration.
- Click the arrow to expand NameNode Default Group in the left-hand pane.
- Click Stacks Collection.
- Modify the configuration settings if desired.
- Click Save Changes.
Stacks collection configuration settings are stored in a per-role configuration file called cloudera-stacks-monitor.properties. Cloudera Manager reads the configuration file and coordinates stack collection. Changes to the configuration settings take effect after a short delay. It is not necessary to restart the role.
Viewing and Downloading Stacks Logs
Stacks are collected and logged to a compressed, rotated log file. A certain amount of the log data is in an uncompressed file. When that file reaches a limit, the file is rotated and bzip2 compressed. Once the total number of files exceeds the configured retention limit, the oldest files are deleted.
- On the service page, click the Instances tab.
- Click the role in the Role Type column.
- In the Summary section of the role page, click Stacks Logs.
- Click Stacks Log File to view the most recent stacks file. Click Download Stacks Logs to download a zipped bundle of the stacks logs.