Back up and restore meta service

This guide introduces how to back up meta service data and restore from a backup.

A meta snapshot is a backup of meta service’s data at a specific point in time. Meta snapshots are persisted in object storage. RisingWave supports multiple object storage backends including S3, MinIO, Google Cloud Storage (GCS), Alibaba Cloud OSS, Huawei Cloud OBS, Azure Blob Storage, HDFS, WebHDFS, and local filesystem.

Set backup parameters

Before you can create a meta snapshot, you need to set the backup_storage_url and backup_storage_directory system parameters prior to the first backup attempt.

Be careful not to set the backup_storage_url and backup_storage_directory when there are snapshots. However, it’s not strictly forbidden. If you insist on doing so, please note the snapshots taken before the setting will all be invalidated and cannot be used in restoration anymore.

If these parameters are not explicitly set, RisingWave will automatically initialize them from your state store configuration: backup_storage_url will default to your state_store URL (or “memory” if using in-memory state store), and backup_storage_directory will default to {data_directory}/backup.

To learn about how to configure system parameters, see How to configure system parameters.

Create a meta snapshot

Meta snapshot is created by meta service whenever requested by users. There is no automatic process in the RisingWave kernel that creates meta snapshot regularly. However, external tools like RisingWave Console can provide automated backup scheduling and management.

Using RisingWave Console

You can create and manage metadata snapshots through the RisingWave Console, which provides a web-based interface for snapshot operations including manual snapshot creation and automated backup configuration.

Using Command Line

Here’s an example of how to create a new meta snapshot with risectl:

risectl meta backup-meta

risectl is included in the pre-built RisingWave binary. Use the following command instead:

./risingwave ctl meta backup-meta

View existing meta snapshots

The following SQL command lists existing meta snapshots:

SELECT meta_snapshot_id, hummock_version_id, remarks, rw_version FROM rw_catalog.rw_meta_snapshot;

Example output:

 meta_snapshot_id | hummock_version_id | remarks | rw_version
------------------
                3 |                123 | manual  | v2.0.0
                4 |                456 | NULL    | v2.0.1

The rw_catalog.rw_meta_snapshot catalog also includes a state_table_info field containing detailed state table information in JSON format.

Delete a meta snapshot

Here’s an example of how to delete a meta snapshot with risectl:

risectl meta delete-meta-snapshots --snapshot-ids <snapshot_id> [--snapshot-ids <snapshot_id>...]

Restore from a meta snapshot

Below are two separate methods to restore from a meta snapshot using SQL database and etcd as the meta store backend.

SQL database as meta store backend

If the cluster has been using a SQL database as meta store backend, follow these steps to restore from a meta snapshot.

1. Shut down RisingWave nodes, including meta, compute, compactor and frontend.

   This step is especially important because the meta backup and recovery process does not replicate SST files. It is not permitted for multiple clusters to run with the same SSTs set at any time, as this can corrupt the SST files.

2. If the Elastic Disk Cache feature is enabled, clear the disk cache on every node where disk cache is enabled before starting the restored cluster. Specifically, delete the contents of the directories configured by storage.meta_file_cache.dir and storage.data_file_cache.dir, or delete and then recreate those directories with the correct permissions before starting the restored cluster.

3. Create a new meta store, i.e., a new SQL database. This database must have the same table structure as the original, but all tables must be empty. Use one of the following methods to achieve this:

   • (Preferred) Create the meta store schema using risectl meta create-meta-store-schema.
     • For RisingWave v3.1 and above, use the RisingWave image where the embedded risectl supports create-meta-store-schema. Ensure the RisingWave image version matches your meta snapshot.
     /risingwave/bin/risingwave ctl meta create-meta-store-schema \
      --meta-store-type postgres \
      --sql-endpoint 127.0.0.1:8432 \
      --sql-username placeholder \
      --sql-password placeholder \
      --sql-database placeholder
     
     • For RisingWave prior to v3.1, use the dedicated risectl image that supports create-meta-store-schema. Ensure risectl image version matches your meta snapshot. For example, for a RisingWave v2.8.3 meta snapshot, use risectl v2.8.
     /risingwave/bin/risectl meta create-meta-store-schema \
     --meta-store-type postgres \
     --sql-endpoint 127.0.0.1:8432 \
     --sql-username placeholder \
     --sql-password placeholder \
     --sql-database placeholder
     
     • After running the command, truncate tables populated during create-meta-store-schema, such as seaql_migrations.
   • Use database-specific tools (e.g., pgdump for Postgres) to duplicate table schemas.
   • Use the schema migration tool to create tables, then truncate those non-empty tables populated by the tool. The version of the migration tool must match the version of the meta snapshot. For example, if the meta snapshot was created with RisingWave v2.4.3, the migration tool should also be from RisingWave v2.4.3.

4. Restore the meta snapshot to the new meta store.

   risectl \
   meta \
   restore-meta \
   --meta-store-type sql \
   --meta-snapshot-id [snapshot_id] \
   --sql-endpoint [sql_endpoint] \
   --backup-storage-url [backup_storage_url, e.g. s3://bucket_read_from] \
   --backup-storage-directory [backup_storage_directory, e.g. dir_read_from] \
   --hummock-storage-url [hummock_storage_url, e.g. s3://bucket_write_to] \
   --hummock-storage-directory [hummock_storage_directory, e.g. dir_write_to]
   

   restore-meta reads snapshot data from backup storage and writes them to meta store and hummock storage. For example, given the cluster settings below:

   psql=> show parameters;
               Name                     |                Value                 | Mutable
   ---------------------------------------+--------------------------------------+---------
   state_store                            | hummock+s3://state_bucket            | f
   data_directory                         | state_data                           | f
   backup_storage_url                     | s3://backup_bucket                   | t
   backup_storage_directory               | backup_data                          | t
   

   Parameters to risectl meta restore-meta should be:

   • --backup-storage-url s3://backup_bucket.
   • --backup-storage-directory backup_data.
   • --hummock-storage-url s3://state_bucket. Note that the hummock+ prefix is stripped.
   • --hummock-storage-directory state_data.

   Additional optional parameters:

   • --dry-run: Print the target snapshot without restoring (default: false)
   • --read-attempt-timeout-ms: Read timeout for object store operations (default: 600000)
   • --read-retry-attempts: Maximum retry attempts for object store reads (default: 3)
   • --validate-integrity: Verify all referenced objects exist in object store (default: false)
   • --overwrite-hummock-storage-endpoint: Overwrite storage endpoint system parameters (default: false)
   • --overwrite-backup-storage-url: Override backup storage URL in restored meta store
   • --overwrite-backup-storage-directory: Override backup storage directory in restored meta store
   • --sql-url-params: Additional SQL connection parameters (e.g., sslmode=disable)

5. Configure meta service to use the new meta store.

etcd as meta store backends

If the cluster has been using etcd as meta store backend, follow these steps to restore from a meta snapshot.

1. Shut down RisingWave nodes, including meta, compute, compactor and frontend.

   This step is especially important because the meta backup and recovery process does not replicate SST files. It is not permitted for multiple clusters to run with the same SSTs set at any time, as this can corrupt the SST files.

2. Create a new meta store, i.e. a new and empty etcd instance.

3. Restore the meta snapshot to the new meta store.

   risectl \
   meta \
   restore-meta \
   --meta-store-type etcd \
   --meta-snapshot-id [snapshot_id] \
   --etcd-endpoints [etcd_endpoints, e.g. 127.0.0.1:2388] \
   --backup-storage-url [backup_storage_url, e.g. s3://bucket_read_from] \
   --backup-storage-directory [backup_storage_directory, e.g. dir_read_from] \
   --hummock-storage-url [hummock_storage_url, e.g. s3://bucket_write_to] \
   --hummock-storage-directory [hummock_storage_directory, e.g. dir_write_to]
   

   If etcd enables authentication, also specify the following:

   --etcd-auth \
   --etcd-username [etcd_username] \
   --etcd-password [etcd_password] \
   

   restore-meta reads snapshot data from backup storage and writes them to meta store and hummock storage. For example, given the cluster settings below:

   psql=> show parameters;
               Name                     |                Value                 | Mutable
   ---------------------------------------+--------------------------------------+---------
   state_store                            | hummock+s3://state_bucket            | f
   data_directory                         | state_data                           | f
   backup_storage_url                     | s3://backup_bucket                   | t
   backup_storage_directory               | backup_data                          | t
   

   Parameters to risectl meta restore-meta should be:

   • --backup-storage-url s3://backup_bucket.
   • --backup-storage-directory backup_data.
   • --hummock-storage-url s3://state_bucket. Note that the hummock+ prefix is stripped.
   • --hummock-storage-directory state_data.

   Additional optional parameters:

   • --dry-run: Print the target snapshot without restoring (default: false)
   • --read-attempt-timeout-ms: Read timeout for object store operations (default: 600000)
   • --read-retry-attempts: Maximum retry attempts for object store reads (default: 3)
   • --validate-integrity: Verify all referenced objects exist in object store (default: false)
   • --overwrite-hummock-storage-endpoint: Overwrite storage endpoint system parameters (default: false)
   • --overwrite-backup-storage-url: Override backup storage URL in restored meta store
   • --overwrite-backup-storage-directory: Override backup storage directory in restored meta store
   • --sql-url-params: Additional SQL connection parameters (e.g., sslmode=disable)

4. Configure meta service to use the new meta store.