REST catalog

Connect RisingWave to a REST-compliant catalog for Apache Iceberg.

A REST catalog uses a RESTful API to manage Iceberg table metadata. You can use a REST catalog with RisingWave when creating an Iceberg SOURCE, SINK, or CONNECTION (for internal Iceberg tables).

For a complete list of all catalog-related parameters, see the main Catalog configuration page.

Example

Examples for using a REST catalog with internal Iceberg tables, sinks, and sources.

Internal tables (recommended)

CREATE CONNECTION rest_conn WITH (
    type = 'iceberg',
    warehouse.path = 's3://my-bucket/warehouse/',
    s3.region = 'us-west-2',
    s3.access.key = '...',
    s3.secret.key = '...',
    enable_config_load = false,

    catalog.type = 'rest',
    catalog.uri = 'http://rest-catalog:8181'
);

SET iceberg_engine_connection = 'public.rest_conn';

-- `commit_checkpoint_interval` controls Iceberg commit frequency. Default: about every 60 seconds; set to 1 for faster commits and visibility.
CREATE TABLE my_internal_iceberg_table (
  id INT PRIMARY KEY,
  name VARCHAR
)
WITH (commit_checkpoint_interval = 1)
ENGINE = iceberg;

Write (CREATE SINK)

CREATE SINK my_rest_sink FROM my_mv WITH (
    connector = 'iceberg',
    type = 'upsert',
    primary_key = 'id',
    catalog.type = 'rest',
    catalog.uri = 'http://rest-catalog:8181',
    warehouse.path = 's3://my-bucket/warehouse/',
    s3.region = 'us-west-2',
    s3.access.key = '...',
    s3.secret.key = '...',
    database.name = 'my_db',
    create_table_if_not_exists = 'true',
    table.name = 'my_table'
);

Read (CREATE SOURCE)

CREATE SOURCE my_rest_source WITH (
    connector = 'iceberg',
    warehouse.path = 's3://my-bucket/warehouse/',
    database.name = 'my_db',
    table.name = 'my_table',

    catalog.type = 'rest',
    catalog.uri = 'http://rest-catalog:8181',

    s3.region = 'us-west-2',
    s3.access.key = '...',
    s3.secret.key = '...'
);

Vended credentials

Added in v2.7.0.

Vended credentials allow the REST catalog server to provide temporary credentials for accessing object storage, rather than requiring you to manage credentials directly in RisingWave.

To enable vended credentials, set vended_credentials = true along with catalog.type = 'rest':

Example: Source

CREATE SOURCE iceberg_t WITH (
    connector = 'iceberg',
    s3.endpoint = 'https://s3.ap-southeast-2.amazonaws.com',
    s3.region = 'ap-southeast-2',
    vended_credentials = true,
    s3.path.style.access = 'true',
    catalog.type = 'rest',
    catalog.uri = 'http://127.0.0.1:8181/catalog',
    warehouse.path = '<warehouse_name>',
    database.name = 'ns',
    table.name = 't'
);

Example: Sink

CREATE SINK sink_t FROM t WITH (
    connector = 'iceberg',
    type = 'append-only',
    force_append_only = 'true',
    s3.endpoint = 'https://s3.ap-southeast-2.amazonaws.com',
    s3.region = 'ap-southeast-2',
    vended_credentials = true,
    s3.path.style.access = 'true',
    catalog.type = 'rest',
    catalog.uri = 'http://127.0.0.1:8181/catalog',
    warehouse.path = '<warehouse_name>',
    database.name = 'ns',
    table.name = 't',
    create_table_if_not_exists = 'true',
    -- `commit_checkpoint_interval` controls Iceberg commit frequency. Default: about every 60 seconds; set to 1 for faster commits and visibility.
    commit_checkpoint_interval = 1
);

GCS with Google authentication

For a Google-authenticated REST catalog backed by GCS, set catalog.security = 'google' and enable vended credentials. RisingWave uses the native REST catalog implementation and authenticates catalog requests with gcs.credential, or with Google Application Default Credentials (ADC) when gcs.credential is omitted. Use gcp.auth.scopes to override the default https://www.googleapis.com/auth/cloud-platform scope.

Example: Authenticate with ADC and use vended GCS credentials

CREATE SOURCE gcs_iceberg_source WITH (
    connector = 'iceberg',
    catalog.type = 'rest',
    catalog.uri = '<rest_catalog_url>',
    catalog.security = 'google',
    gcp.auth.scopes = 'https://www.googleapis.com/auth/cloud-platform',
    vended_credentials = true,
    warehouse.path = 'gs://my-bucket/warehouse',
    database.name = 'ns',
    table.name = 't'
);

Catalog authentication and storage authentication are separate. When delegation is enabled, the catalog must vend valid GCS tokens for storage access. RisingWave does not fall back to gcs.credential or ambient storage credentials when a vended token is missing, expired, or revoked. Disable vended_credentials if the catalog does not vend GCS tokens and you need to use static or ambient storage credentials.

If DISABLE_DEFAULT_CREDENTIAL=true is set in your deployment, ADC is unavailable and you must provide gcs.credential when using catalog.security = 'google'.

Deploying a Lakekeeper REST catalog

For a complete, step-by-step guide to deploying a self-hosted REST catalog with Lakekeeper, see this topic.