Configuring HBase Snapshots
About HBase Snapshots
In previous HBase releases, the only way to a backup or to clone a table was to use CopyTable or ExportTable, or to copy all the hfiles in HDFS after disabling the table. The disadvantages of these methods are:
- CopyTable and ExportTable can degrade region server performance.
- Disabling the table means no reads or writes; this is usually unacceptable.
HBase Snapshots allow you to clone a table without making data copies, and with minimal impact on Region Servers. Exporting the table to another cluster should not have any impact on the the region servers.
- Recovery from user or application errors
- Useful because it may be some time before the database
administrator notices the error Note
The database administrator needs to schedule the intervals at which to take and delete snapshots. Use a script or your preferred management tool for this; it is not built into HBase.
- The database administrator may want to save a snapshot right
before a major application upgrade or change. Note
Snapshots are not primarily used for system upgrade protection because they would not roll back binaries, and would not necessarily be proof against bugs or errors in the system or the upgrade.
- Sub-cases for recovery:
- Rollback to previous snapshot and merge in reverted data
- View previous snapshots and selectively merge them into production
- Useful because it may be some time before the database administrator notices the error
- Capture a copy of the database and store it outside HBase for disaster recovery
- Capture previous versions of data for compliance, regulation, archiving
- Export from snapshot on live system provides a more consistent view of HBase than CopyTable and ExportTable
- Audit and/or report view of data at a specific time
- Capture monthly data for compliance
- Use for end-of-day/month/quarter reports
- Use for Application testing
- Test schema or application changes on like production data from snapshot and then throw away
- For example: take a snapshot; create a new table from the snapshot content (schema plus data); manipulate the new table by changing the schema, adding and removing rows, and so on (the original table, the snapshot, and the new table remain independent of each other)
- Offload work
- Capture, copy, and restore data to another site
- Export data to another cluster
Where Snapshots Are Stored
The snapshot metadata is stored in the .snapshot directory under the hbase root directory (/hbase/.snapshot). Each snapshot has its own directory that includes all the references to the hfiles, logs, and metadata needed to restore the table.
hfiles needed by the snapshot are in the traditional /hbase/<tableName>/<regionName>/<familyName>/ location if the table is still using them; otherwise they will be placed in /hbase/.archive/<tableName>/<regionName>/<familyName>/
Zero-copy Restore and Clone Table
From a snapshot you can create a new table (clone operation) or restore the original table. These two operations do not involve data copies; instead a link is created to point to the original hfiles.
Changes to a cloned or restored table do not affect the snapshot or (in case of a clone) the original table.
If you want to clone a table to another cluster, you need to export the snapshot to the other cluster and then execute the clone operation; see Exporting a Snapshot to Another Cluster.
Reverting to a Previous HBase Version
Snapshots don’t affect HBase backward compatibility if they are not used.
If you do use the snapshot capability, backward compatibility is affected as follows:
- If you only take snapshots, you can still go back to a previous HBase version
- If you have used restore or clone, you cannot go back to a previous version unless the cloned or restored tables have no links (there is no automated way to check; you would need to inspect the file system manually).
Since the hfiles are immutable, a snapshot consists of reference to the files that are in the table at the moment the snapshot is taken. No copies of the data are made during the snapshot operation, but copies may be made when a compaction or deletion is triggered. In this case, if a snapshot has a reference to the files to be removed, the files are moved to an archive folder, instead of being deleted. This allows the snapshot to be restored in full.
Because no copies are performed, multiple snapshots share the same hfiles, but in the worst case scenario, each snapshot could have different set of hfiles (tables with lots of updates, and compactions).
Configuring and Enabling Snapshots
Snapshots are off by default; to enable them, set the hbase.snapshot.enabled property in hbase-site.xml to true:
<property> <name>hbase.snapshot.enabled</name> <value> true </value> </property>
To disable snapshots after you have enabled them, set hbase.snapshot.enabled to false.
If you have taken snapshots and then decide to disable snapshots, you must delete the snapshots before restarting the HBase master; the HBase master will not start if snapshots are disabled and snapshots exist.
Snapshots don’t affect HBase performance if they are not used.
You can manage snapshots by using the HBase shell or the HBaseAdmin Java API.
The following table shows actions you can take from the shell:
Take a snapshot of tableX called snapshotX
snapshot 'tableX', 'snapshotX'
Snapshots can be taken while a table is disabled, or while a table is online and serving traffic.
Restore snapshot snapshotX (it will replace the source table content)
Restoring a snapshot attempts to replace the current version of a table with another version of the table. To run this command, you must disable the target table. The restore command takes a snapshot of the table (appending a timestamp code), and then essentially clones data into the original data and removes data not in the snapshot. If the operation succeeds, the target table will be enabled. Use this capability only in an emergency; see Restrictions.
List all available snapshots
List all available snapshots starting with ‘mysnapshot_’ (regular expression)
Remove a snapshot called snapshotX
Create a new table tableY from a snapshot snapshotX
clone_snapshot ‘snapshotX’, ‘tableY’
Cloning a snapshot creates a new read/write table that can serve the data kept at the time of the snapshot. The original table and the cloned table can be modified independently without interfering – new data written to one table will not show up on the other.
Exporting a Snapshot to Another Cluster
The ExportSnapshot tool executes a MapReduce Job, similar to distcp, to copy files to the other cluster. It works at file-system level, so the hbase cluster can be offline.
To copy a snapshot called MySnapshot to an HBase cluster srv2 (hdfs:///srv2:8082/hbase) using 16 mappers:
hbase class org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot MySnapshot -copy-to hdfs:///srv2:8082/hbase -mappers 16
To export the snapshot and change the ownership of the files during the copy:
hbase class org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot MySnapshot -copy-to hdfs:///srv2:8082/hbase -chuser MyUser -chgroup MyGroup -chmod 700 -mappers 16
hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -Dsnapshot.export.default.map.group=10 -snapshot MY_SNAPSHOT -copy-to hdfs://cluster2/hbase(The number of mappers is calculated as TotalNumberOfHFiles/10.)
Do not use merge in combination with snapshots. Merging two regions can cause data loss if snapshots or cloned tables exist for this table.
The merge is likely to corrupt the snapshot and any tables cloned from the snapshot. In addition, if the table has been restored from a snapshot, the merge may also corrupt the table. The snapshot may survive intact if the regions being merged are not in the snapshot, and clones may survive if they do not share files with the original table or snapshot. You can use the Snapinfo tool (see Information and Debugging) to check the status of the snapshot. If the status is BROKEN, the snapshot is unusable.
- All the Masters and Region Servers must be running at least CDH4.2.
- If you have enabled the AccessController Coprocessor for HBase, only a global administrator can take, clone, or restore a snapshot, and these actions do not capture the ACL rights. This means that restoring a table preserves the ACL rights of the existing table, while cloning a table creates a new table that has no ACL rights until the administrator adds them.
- Do not take, clone, or restore a snapshot during a rolling restart. Snapshots rely on
the Region Servers being up; otherwise the snapshot will fail.Note
:This restriction also applies to rolling upgrade, which can currently be done only via Cloudera Manager.
If you are using HBase Replication and you need to restore a snapshot: If you are using HBase Replication the replicas will be out of synch when you restore a snapshot. Do this only in an emergency.
Snapshot restore is an emergency tool; you need to disable the table and table replication to get to an earlier state, and you may lose data in the process.
If you need to restore a snapshot, proceed as follows:
- Disable the table that is the restore target, and stop the replication
- Remove the table from both the master and slave clusters
- Restore the snapshot on the master cluster
- Create the table on the slave cluster and use CopyTable to initialize it.
If this is not an emergency (for example, if you know that you have lost just a set of rows such as the rows starting with "xyz"), you can create a clone from the snapshot and create a MapReduce job to copy the data that you've lost.
In this case you don't need to stop replication or disable your main table.
Region moves, splits, and other metadata actions that happen while a snapshot is in progress will probably cause the snapshot to fail; the software detects and rejects corrupted snapshot attempts.
Information and Debugging
You can use the SnapshotInfo tool to get information about a snapshot, including status, files, disk usage, and debugging information.
$ hbase org.apache.hadoop.hbase.snapshot.SnapshotInfo -snapshot test-snapshot Snapshot Info ---------------------------------------- Name: test-snapshot Type: DISABLED Table: test-table Version: 0 Created: 2012-12-30T11:21:21 ************************************************************** BAD SNAPSHOT: 1 hfile(s) and 0 log(s) missing. ************************************************************** 6 HFiles (6 in archive), total size 589.7k (0.00% 0.0 shared with the source table) 0 Logs, total size 0.0
$ hbase org.apache.hadoop.hbase.snapshot.SnapshotInfo -snapshot test-snapshot -files Snapshot Info ---------------------------------------- Name: test-snapshot Type: DISABLED Table: test-table Version: 0 Created: 2012-12-30T11:21:21 Snapshot Files ---------------------------------------- 52.4k test-table/02ba3a0f8964669520cf96bb4e314c60/cf/bdf29c39da2a4f2b81889eb4f7b18107 (archive) 52.4k test-table/02ba3a0f8964669520cf96bb4e314c60/cf/1e06029d0a2a4a709051b417aec88291 (archive) 86.8k test-table/02ba3a0f8964669520cf96bb4e314c60/cf/506f601e14dc4c74a058be5843b99577 (archive) 52.4k test-table/02ba3a0f8964669520cf96bb4e314c60/cf/5c7f6916ab724eacbcea218a713941c4 (archive) 293.4k test-table/02ba3a0f8964669520cf96bb4e314c60/cf/aec5e33a6564441d9bd423e31fc93abb (archive) 52.4k test-table/02ba3a0f8964669520cf96bb4e314c60/cf/97782b2fbf0743edaacd8fef06ba51e4 (archive) 6 HFiles (6 in archive), total size 589.7k (0.00% 0.0 shared with the source table) 0 Logs, total size 0.0