A meta snapshot is a backup of meta service’s data at a specific point in time. Meta snapshots are persisted in object storage. RisingWave supports multiple object storage backends including S3, MinIO, Google Cloud Storage (GCS), Alibaba Cloud OSS, Huawei Cloud OBS, Azure Blob Storage, HDFS, WebHDFS, and local filesystem.
Set backup parameters
Before you can create a meta snapshot, you need to set the backup_storage_url and backup_storage_directory system parameters prior to the first backup attempt.
Be careful not to set the backup_storage_url and backup_storage_directory when there are snapshots. However, it’s not strictly forbidden. If you insist on doing so, please note the snapshots taken before the setting will all be invalidated and cannot be used in restoration anymore.
backup_storage_url will default to your state_store URL (or “memory” if using in-memory state store), and backup_storage_directory will default to {data_directory}/backup.
To learn about how to configure system parameters, see How to configure system parameters.
Meta snapshot is created by meta service whenever requested by users. There is no automatic process in the RisingWave kernel that creates meta snapshot regularly. However, external tools like RisingWave Console can provide automated backup scheduling and management.
Using RisingWave Console
You can create and manage metadata snapshots through the RisingWave Console, which provides a web-based interface for snapshot operations including manual snapshot creation and automated backup configuration.
Using Command Line
Here’s an example of how to create a new meta snapshot with risectl:
risectl is included in the pre-built RisingWave binary. Use the following command instead:
./risingwave ctl meta backup-meta
SELECT meta_snapshot_id, hummock_version_id, remarks, rw_version FROM rw_catalog.rw_meta_snapshot;
meta_snapshot_id | hummock_version_id | remarks | rw_version
------------------
3 | 123 | manual | v2.0.0
4 | 456 | NULL | v2.0.1
rw_catalog.rw_meta_snapshot catalog also includes a state_table_info field containing detailed state table information in JSON format.
Here’s an example of how to delete a meta snapshot with risectl:
risectl meta delete-meta-snapshots [snapshot_ids]
-
Shut down the meta service.
This step is especially important because the meta backup and recovery process does not replicate SST files. It is not permitted for multiple clusters to run with the same SSTs set at any time, as this can corrupt the SST files.
-
Create a new meta store, i.e., a new SQL database. This database must have the same table structure as the original, but all tables must be empty. To achieve this, you can either:
- Use database-specific tools (e.g.,
pgdump for Postgres) to duplicate table schemas, or
- Use the schema migration tool to create tables, then truncate those non-empty tables populated by the tool. The version of the migration tool must match the version of the meta snapshot. For example, if the meta snapshot was created with RisingWave v2.4.3, the migration tool should also be from RisingWave v2.4.3.
-
Restore the meta snapshot to the new meta store.
risectl \
meta \
restore-meta \
--meta-store-type sql \
--meta-snapshot-id [snapshot_id] \
--sql-endpoint [sql_endpoint] \
--backup-storage-url [backup_storage_url, e.g. s3://bucket_read_from] \
--backup-storage-directory [backup_storage_directory, e.g. dir_read_from] \
--hummock-storage-url [hummock_storage_url, e.g. s3://bucket_write_to] \
--hummock-storage-directory [hummock_storage_directory, e.g. dir_write_to]
restore-meta reads snapshot data from backup storage and writes them to meta store and hummock storage.
For example, given the cluster settings below:
psql=> show parameters;
Name | Value | Mutable
---------------------------------------+--------------------------------------+---------
state_store | hummock+s3://state_bucket | f
data_directory | state_data | f
backup_storage_url | s3://backup_bucket | t
backup_storage_directory | backup_data | t
risectl meta restore-meta should be:
--backup-storage-url s3://backup_bucket.--backup-storage-directory backup_data.--hummock-storage-url s3://state_bucket. Note that the hummock+ prefix is stripped.--hummock-storage-directory state_data.
Additional optional parameters:
--dry-run: Print the target snapshot without restoring (default: false)--read-attempt-timeout-ms: Read timeout for object store operations (default: 600000)--read-retry-attempts: Maximum retry attempts for object store reads (default: 3)--validate-integrity: Verify all referenced objects exist in object store (default: false)--overwrite-hummock-storage-endpoint: Overwrite storage endpoint system parameters (default: false)--overwrite-backup-storage-url: Override backup storage URL in restored meta store--overwrite-backup-storage-directory: Override backup storage directory in restored meta store--sql-url-params: Additional SQL connection parameters (e.g., sslmode=disable)
-
Configure meta service to use the new meta store.
If the cluster has been using etcd as meta store backend, follow these steps to restore from a meta snapshot.
-
Shut down the meta service.
This step is especially important because the meta backup and recovery process does not replicate SST files. It is not permitted for multiple clusters to run with the same SSTs set at any time, as this can corrupt the SST files.
-
Create a new meta store, i.e. a new and empty etcd instance.
-
Restore the meta snapshot to the new meta store.
risectl \
meta \
restore-meta \
--meta-store-type etcd \
--meta-snapshot-id [snapshot_id] \
--etcd-endpoints [etcd_endpoints, e.g. 127.0.0.1:2388] \
--backup-storage-url [backup_storage_url, e.g. s3://bucket_read_from] \
--backup-storage-directory [backup_storage_directory, e.g. dir_read_from] \
--hummock-storage-url [hummock_storage_url, e.g. s3://bucket_write_to] \
--hummock-storage-directory [hummock_storage_directory, e.g. dir_write_to]
--etcd-auth \
--etcd-username [etcd_username] \
--etcd-password [etcd_password] \
restore-meta reads snapshot data from backup storage and writes them to meta store and hummock storage.
For example, given the cluster settings below:
psql=> show parameters;
Name | Value | Mutable
---------------------------------------+--------------------------------------+---------
state_store | hummock+s3://state_bucket | f
data_directory | state_data | f
backup_storage_url | s3://backup_bucket | t
backup_storage_directory | backup_data | t
risectl meta restore-meta should be:
--backup-storage-url s3://backup_bucket.--backup-storage-directory backup_data.--hummock-storage-url s3://state_bucket. Note that the hummock+ prefix is stripped.--hummock-storage-directory state_data.
Additional optional parameters:
--dry-run: Print the target snapshot without restoring (default: false)--read-attempt-timeout-ms: Read timeout for object store operations (default: 600000)--read-retry-attempts: Maximum retry attempts for object store reads (default: 3)--validate-integrity: Verify all referenced objects exist in object store (default: false)--overwrite-hummock-storage-endpoint: Overwrite storage endpoint system parameters (default: false)--overwrite-backup-storage-url: Override backup storage URL in restored meta store--overwrite-backup-storage-directory: Override backup storage directory in restored meta store--sql-url-params: Additional SQL connection parameters (e.g., sslmode=disable)
-
Configure meta service to use the new meta store.
