diff --git a/gpdb-doc/dita/admin_guide/managing/backup-plugin-api.xml b/gpdb-doc/dita/admin_guide/managing/backup-plugin-api.xml new file mode 100644 index 0000000000000000000000000000000000000000..23ce62e5f6f75396188f32f9411d14727e84e039 --- /dev/null +++ b/gpdb-doc/dita/admin_guide/managing/backup-plugin-api.xml @@ -0,0 +1,231 @@ + + + + Backup/Restore Storage Plugin API + +

This topic describes how to develop a custom storage plugin with the Greenplum Database Backup/Restore Storage Plugin API.

+ The Greenplum Database Backup/Restore Storage Plugin API + is an experimental feature and is not intended for use in a production + environment. Experimental features are subject to change without notice + in future releases. +

The Backup/Restore Storage Plugin API provides a framework that you can use to develop and integrate a custom backup storage system with the Greenplum Database gpbackup and gprestore utilities.

+

The Backup/Restore Storage Plugin API defines a set of interfaces that a plugin must support. The API also specifies the format and content of a configuration file for a plugin.

+

When you use the Backup/Restore Storage Plugin API, you create a plugin that the Greenplum Database administrator deploys to the Greenplum Database cluster. Once deployed, the plugin is available for use in certain backup and restore operations.

+ Greenplum Database supports using a custom storage plugin on a backup operation only when gpbackup is run with the --single-data-file or --metadata-only options. + +

This topic includes the following subtopics:

+ + + + + + Plugin Configuration File + +

Specifying the --plugin-config option to the gpbackup and gprestore commands instructs the utilities to use the plugin specified in the configuration file for the operation.

+

The plugin configuration file provides information for both Greenplum Database and the plugin. The Backup/Restore Storage Plugin API defines the format of, and certain keywords used in, the plugin configuration file.

+

A plugin configuration file is a YAML file in the following format: executablepath: path_to_plugin_executable +options: + keyword1: value1 + keyword2: value2 + ... + keywordN: valueN

+

gpbackup and gprestore use the executablepath value to determine the file system location of the plugin executable program.

+

The plugin configuration file may also include keywords and values specific to a plugin instance. A backup/restore storage plugin can use the options block specified in the file to obtain information from the user that may be required to perform its tasks. This information may include location, connection, or authentication information, for example. The plugin should both specify and consume the content of this information in keyword:value syntax.

+ +

A sample plugin configuration file for the Greenplum Database S3 backup/restore storage plugin follows:executablepath: $GPHOME/bin/gpbackup_s3_plugin +options: + region: us-west-2 + aws_access_key_id: notarealID + aws_secret_access_key: notarealkey + bucket: gp_backup_bucket + backupdir: greenplum_backups

+ + + + + Plugin API + +

The plugin that you implement when you use the Backup/Restore Storage Plugin API is an executable program that supports specific commands invoked by gpbackup and gprestore at defined points in their respective life cycle operations:

+ +

The Backup/Restore Storage Plugin API defines the following call syntax for a backup/restore storage plugin executable program:plugin_executable command config_file arg

+ +

where:

+ + + + Plugin Commands + +

The Greenplum Database Backup/Restore Storage Plugin API defines the following commands: + + Backup/Restore Storage Plugin API Commands + + + + + + Command Name + Description + + + + + plugin_api_version + Return the version of the Backup/Restore Storage Plugin API supported by the plugin. The only supported version is 0.1.0. + + + setup_plugin_for_backup + Initialize the plugin for a backup operation. + + + backup_file + Move a backup file to the remote storage system. + + + backup_data + Move streaming data from stdin to a file on the remote storage system. + + + cleanup_plugin_for_backup + Clean up after a backup operation. + + + setup_plugin_for_restore + Initialize the plugin for a restore operation. + + + restore_file + Move a backup file from the remote storage system to a designated location on the local host. + + + restore_data + Move a backup file from the remote storage system, streaming the data to stdout. + + + cleanup_plugin_for_restore + Clean up after a restore operation. + + + +
+

+

A backup/restore storage plugin must support every command identified above, even if it is a no-op.

+ + + + + + Implementing a Backup/Restore Storage Plugin + +

You can implement a backup/restore storage plugin executable in any programming or scripting language.

+

The tasks performed by a backup/restore storage plugin will be very specific to the remote storage system. As you design the plugin implementation, you will want to:

+

A backup/restore storage plugin that you implement must:

+

Refer to the gpbackup-s3-plugin github repository for an example plugin implementation.

+ + + + + Verifying a Backup/Restore Storage Plugin + +

The Backup/Restore Storage Plugin API includes a test bench that you can run to ensure that a plugin is well integrated with gpbackup and gprestore.

+

The test bench is a bash script that you run in a Greenplum Database installation. The script generates a small (<1MB) data set in a Greenplum Database table, explicitly tests each command, and runs a backup and restore of the data (file and streaming). The test bench invokes gpbackup and gprestore, which in turn individually call/test each Backup/Restore Storage Plugin API command implemented in the plugin.

+

The test bench program calling syntax is: plugin_test_bench.sh plugin_executable plugin_config

+
+ Procedure +

To run the Backup/Restore Storage Plugin API test bench against a plugin:

    +
  1. Log in to the Greenplum Database master host and set up your environment. For example:$ ssh gpadmin@<gpmaster> +gpadmin@gpmaster$ . /usr/local/greenplum-db/greenplum_path.sh
    2. +
  2. Obtain a copy of the test bench from the gpbackup github repository. For example:$ git clone git@github.com:greenplum-db/gpbackup.gitThe clone operation creates a directory named gpbackup/ in the current working directory.
    3. +
  3. Locate the test bench program in the gpbackup/master/plugins directory. For example:$ ls gpbackup/master/plugins/plugin_test_bench.sh
    4. +
  4. Copy the plugin executable program and the plugin configuration YAML file from your development system to the Greenplum Database master host. Note the file system location to which you copied the files.
    5. +
  5. Copy the plugin executable program from the Greenplum Database master host to the same file system location on each segment host.
    6. +
  6. If required, edit the plugin configuration YAML file to specify the absolute path of the plugin executable program that you just copied to the Greenplum segments.
    7. +
  7. Run the test bench program against the plugin. For example:$ gpbackup/master/plugins/plugin_test_bench.sh /path/to/pluginexec /path/to/plugincfg.yaml
    8. +
  8. Examine the test bench output. Your plugin passed the test bench if all output messages specify RUNNING and PASSED. For example:# ---------------------------------------------- +# Starting gpbackup plugin tests +# ---------------------------------------------- +[RUNNING] plugin_api_version +[PASSED] plugin_api_version +[RUNNING] setup_plugin_for_backup +[RUNNING] backup_file +[RUNNING] setup_plugin_for_restore +[RUNNING] restore_file +[PASSED] setup_plugin_for_backup +[PASSED] backup_file +[PASSED] setup_plugin_for_restore +[PASSED] restore_file +[RUNNING] backup_data +[RUNNING] restore_data +[PASSED] backup_data +[PASSED] restore_data +[RUNNING] cleanup_plugin_for_backup +[PASSED] cleanup_plugin_for_backup +[RUNNING] cleanup_plugin_for_restore +[PASSED] cleanup_plugin_for_restore +[RUNNING] gpbackup with test database +[RUNNING] gprestore with test database +[PASSED] gpbackup and gprestore +# ---------------------------------------------- +# Finished gpbackup plugin tests +# ----------------------------------------------
+
+ + + + Packaging and Deploying a Backup/Restore Storage Plugin + +

Your backup/restore storage plugin is ready to be deployed to a Greenplum Database installation after the plugin passes your testing and the test bench verification. When you package the backup/restore storage plugin, consider the following:

+ + + diff --git a/gpdb-doc/dita/admin_guide/managing/backup.ditamap b/gpdb-doc/dita/admin_guide/managing/backup.ditamap index c460f613cb1ce2b7deacf6a55fcb1bb10ae2fb46..0b2366550d5c1ecde5ddb00e7157b7d1175c0d31 100644 --- a/gpdb-doc/dita/admin_guide/managing/backup.ditamap +++ b/gpdb-doc/dita/admin_guide/managing/backup.ditamap @@ -30,6 +30,17 @@ + + + + + + + + + + + diff --git a/gpdb-doc/dita/admin_guide/managing/plugin_api/backup_data.xml b/gpdb-doc/dita/admin_guide/managing/plugin_api/backup_data.xml new file mode 100644 index 0000000000000000000000000000000000000000..2dd1c6820fe5449d4f2353f89a016bc8aec369fd --- /dev/null +++ b/gpdb-doc/dita/admin_guide/managing/plugin_api/backup_data.xml @@ -0,0 +1,33 @@ + + + + backup_data + +

Plugin command to move streaming data from stdin to the remote storage system.

+
+ Synopsis + plugin_executable backup_data plugin_config_file data_filenamekey +
+
+ Description +

gpbackup invokes the backup_data plugin command on each segment host during a streaming backup.

+

The backup_data implementation should read a potentially large stream of data from stdin and write the data to a single file on the remote storage system. The data is sent to the command as a single continuous stream per Greenplum Database segment. If backup_data modifies the data in any manner (i.e. compresses), restore_data must perform the reverse operation.

+

Name or maintain a mapping from the destination file to data_filenamekey. This will be the file key used for the restore operation.

+
+
Arguments + + plugin_config_file + The absolute path to the plugin configuration YAML file. + + + data_filenamekey + The mapping key for a specially-named backup file for streamed data. + + +
+
Exit Code +

The backup_data command must exit with a value of 0 on success, non-zero if an error occurs. In the case of a non-zero exit code, gpbackup displays the contents of stderr to the user.

+
+ + diff --git a/gpdb-doc/dita/admin_guide/managing/plugin_api/backup_file.xml b/gpdb-doc/dita/admin_guide/managing/plugin_api/backup_file.xml new file mode 100644 index 0000000000000000000000000000000000000000..d70fec2f20863090416a0734cb09d885e7dfe657 --- /dev/null +++ b/gpdb-doc/dita/admin_guide/managing/plugin_api/backup_file.xml @@ -0,0 +1,32 @@ + + + + backup_file + +

Plugin command to move a backup file to the remote storage system.

+
+ Synopsis + plugin_executable backup_file plugin_config_file file_to_backup +
+
+ Description +

gpbackup invokes the backup_file plugin command on the master and each segment host for the file that gpbackup writes to a backup directory on local disk.

+

The backup_file implementation should process and move the file to the remote storage system. gpbackup does not remove the file_to_backup from the local system. Remove the file in the backup_file command, if appropriate.

+
+
Arguments + + plugin_config_file + The absolute path to the plugin configuration YAML file. + + + file_to_backup + The absolute path to a local backup file generated by gpbackup. + + +
+
Exit Code +

The backup_file command must exit with a value of 0 on success, non-zero if an error occurs. In the case of a non-zero exit code, gpbackup displays the contents of stderr to the user.

+
+ + diff --git a/gpdb-doc/dita/admin_guide/managing/plugin_api/cleanup_plugin_for_backup.xml b/gpdb-doc/dita/admin_guide/managing/plugin_api/cleanup_plugin_for_backup.xml new file mode 100644 index 0000000000000000000000000000000000000000..4736429b0f6a50d7cfc4b536083f636483ffd419 --- /dev/null +++ b/gpdb-doc/dita/admin_guide/managing/plugin_api/cleanup_plugin_for_backup.xml @@ -0,0 +1,32 @@ + + + + cleanup_plugin_for_backup + +

Plugin command to clean up a storage plugin after backup.

+
+ Synopsis + plugin_executable cleanup_plugin_for_backup plugin_config_file local_backup_dir +
+
+ Description +

gpbackup invokes the cleanup_plugin_for_backup plugin command on the master and all segment hosts when a gpbackup operation completes, both in success and failure cases.

+

The cleanup_plugin_for_backup command should perform the actions necessary to clean up the remote storage system after a backup. Clean up activities may include removing remote directories or temporary files created during the backup, disconnecting from the backup service, etc.

+
+
Arguments + + plugin_config_file + The absolute path to the plugin configuration YAML file. + + + local_backup_dir + The local directory on the Greenplum Database host (master and segments) to which gpbackup wrote backup files. + + +
+
Exit Code +

The cleanup_plugin_for_backup command must exit with a value of 0 on success, non-zero if an error occurs. In the case of a non-zero exit code, gpbackup displays the contents of stderr to the user.

+
+ + diff --git a/gpdb-doc/dita/admin_guide/managing/plugin_api/cleanup_plugin_for_restore.xml b/gpdb-doc/dita/admin_guide/managing/plugin_api/cleanup_plugin_for_restore.xml new file mode 100644 index 0000000000000000000000000000000000000000..9e080350dc22cffccc145d84d65a6cfa099d06fd --- /dev/null +++ b/gpdb-doc/dita/admin_guide/managing/plugin_api/cleanup_plugin_for_restore.xml @@ -0,0 +1,32 @@ + + + + cleanup_plugin_for_restore + +

Plugin command to clean up a storage plugin after restore.

+
+ Synopsis + plugin_executable cleanup_plugin_for_restore plugin_config_file local_backup_dir +
+
+ Description +

gprestore invokes the cleanup_plugin_for_restore plugin command on the master and all segment hosts when a gprestore operation completes, both in success and failure cases.

+

The cleanup_plugin_for_restore implementation should perform the actions necessary to clean up the remote storage system after a restore. Clean up activities may include removing remote directories or temporary files created during the restore, disconnecting from the backup service, etc.

+
+
Arguments + + plugin_config_file + The absolute path to the plugin configuration YAML file. + + + local_backup_dir + The local directory on the Greenplum Database host (master and segments) from which gprestore reads backup files. + + +
+
Exit Code +

The cleanup_plugin_for_restore command must exit with a value of 0 on success, non-zero if an error occurs. In the case of a non-zero exit code, gprestore displays the contents of stderr to the user.

+
+ + diff --git a/gpdb-doc/dita/admin_guide/managing/plugin_api/plugin_api_version.xml b/gpdb-doc/dita/admin_guide/managing/plugin_api/plugin_api_version.xml new file mode 100644 index 0000000000000000000000000000000000000000..a8b092116d585f1f7a85d8e8f8306c89aed65c71 --- /dev/null +++ b/gpdb-doc/dita/admin_guide/managing/plugin_api/plugin_api_version.xml @@ -0,0 +1,20 @@ + + + + plugin_api_version + +

Plugin command to display the supported Backup Storage Plugin API version.

+
+ Synopsis + plugin_executable plugin_api_version +
+
+ Description +

gpbackup and gprestore invoke the plugin_api_version plugin command before a backup or restore operation to determine Backup Storage Plugin API version compatibility.

+
+
Return Value +

The plugin_api_version command must return the Backup Storage Plugin API version number supported by the storage plugin, "0.1.0".

+
+ + diff --git a/gpdb-doc/dita/admin_guide/managing/plugin_api/restore_data.xml b/gpdb-doc/dita/admin_guide/managing/plugin_api/restore_data.xml new file mode 100644 index 0000000000000000000000000000000000000000..182afb14b1aa24b2e256fdf630cfbef7417d5d4b --- /dev/null +++ b/gpdb-doc/dita/admin_guide/managing/plugin_api/restore_data.xml @@ -0,0 +1,32 @@ + + + + restore_data + +

Plugin command to stream data from the remote storage system to stdout.

+
+ Synopsis + plugin_executable restore_data plugin_config_file data_filenamekey +
+
+ Description +

gprestore invokes the restore_data plugin command on each segment host when a restoring a streaming backup.

+

The restore_data implementation should read a potentially large data file named or mapped to data_filenamekey from the remote storage system and write the contents to stdout. If the backup_data command modified the data in any way (i.e. compressed), restore_data should perform the reverse operation.

+
+
Arguments + + plugin_config_file + The absolute path to the plugin configuration YAML file. + + + data_filenamekey + The mapping key to a backup file on the remote storage system. data_filenamekey is the same key provided to the backup_data command. + + +
+
Exit Code +

The restore_data command must exit with a value of 0 on success, non-zero if an error occurs. In the case of a non-zero exit code, gprestore displays the contents of stderr to the user.

+
+ + diff --git a/gpdb-doc/dita/admin_guide/managing/plugin_api/restore_file.xml b/gpdb-doc/dita/admin_guide/managing/plugin_api/restore_file.xml new file mode 100644 index 0000000000000000000000000000000000000000..ee5bb3fa65e2ea8405e86a23d5c588e20339cec0 --- /dev/null +++ b/gpdb-doc/dita/admin_guide/managing/plugin_api/restore_file.xml @@ -0,0 +1,32 @@ + + + + restore_file + +

Plugin command to move a backup file from the remote storage system.

+
+ Synopsis + plugin_executable restore_file plugin_config_file file_to_restore +
+
+ Description +

gprestore invokes the restore_file plugin command on the master and each segment host for the file that gprestore will read from a backup directory on local disk.

+

The restore_file command should process and move the file from the remote storage system to file_to_restore on the local host.

+
+
Arguments + + plugin_config_file + The absolute path to the plugin configuration YAML file. + + + file_to_restore + The absolute path to which to move a backup file from the remote storage system. + + +
+
Exit Code +

The restore_file command must exit with a value of 0 on success, non-zero if an error occurs. In the case of a non-zero exit code, gprestore displays the contents of stderr to the user.

+
+ + diff --git a/gpdb-doc/dita/admin_guide/managing/plugin_api/setup_plugin_for_backup.xml b/gpdb-doc/dita/admin_guide/managing/plugin_api/setup_plugin_for_backup.xml new file mode 100644 index 0000000000000000000000000000000000000000..a568f20a83a545f59fc68de843e7fb7e4ad5932b --- /dev/null +++ b/gpdb-doc/dita/admin_guide/managing/plugin_api/setup_plugin_for_backup.xml @@ -0,0 +1,32 @@ + + + + setup_plugin_for_backup + +

Plugin command to initialize a storage plugin for the backup operation.

+
+ Synopsis + plugin_executable setup_plugin_for_backup plugin_config_file local_backup_dir +
+
+ Description +

gpbackup invokes the setup_plugin_for_backup plugin command on the master and all segment hosts during gpbackup initialization phase.

+

The setup_plugin_for_backup command should perform the activities necessary to initialize the remote storage system before backup begins. Set up activities may include creating remote directories, validating connectivity to the remote storage system, checking disks, and so forth.

+
+
Arguments + + plugin_config_file + The absolute path to the plugin configuration YAML file. + + + local_backup_dir + The local directory on the Greenplum Database host (master and segments) to which gpbackup will write backup files. gpbackup creates this local directory. + + +
+
Exit Code +

The setup_plugin_for_backup command must exit with a value of 0 on success, non-zero if an error occurs. In the case of a non-zero exit code, gpbackup displays the contents of stderr to the user.

+
+ + diff --git a/gpdb-doc/dita/admin_guide/managing/plugin_api/setup_plugin_for_restore.xml b/gpdb-doc/dita/admin_guide/managing/plugin_api/setup_plugin_for_restore.xml new file mode 100644 index 0000000000000000000000000000000000000000..f93d91fbcc97cb8b8056dea0df871f2cbad734f8 --- /dev/null +++ b/gpdb-doc/dita/admin_guide/managing/plugin_api/setup_plugin_for_restore.xml @@ -0,0 +1,32 @@ + + + + setup_plugin_for_restore + +

Plugin command to initialize a storage plugin for the restore operation.

+
+ Synopsis + plugin_executable setup_plugin_for_restore plugin_config_file local_backup_dir +
+
+ Description +

gprestore invokes the setup_plugin_for_restore plugin command on the master and all segment hosts during gprestore initialization phase.

+

The setup_plugin_for_restore command should perform the activities necessary to initialize the remote storage system before a restore operation begins. Set up activities may include creating remote directories, validating connectivity to the remote storage system, etc.

+
+
Arguments + + plugin_config_file + The absolute path to the plugin configuration YAML file. + + + local_backup_dir + The local directory on the Greenplum Database host (master and segments) from which gprestore reads backup files. setup_plugin_for_restore must create this directory if it does not already exist. + + +
+
Exit Code +

The setup_plugin_for_restore command must exit with a value of 0 on success, non-zero if an error occurs. In the case of a non-zero exit code, gprestore displays the contents of stderr to the user.

+
+ +