REST catalog

Connect RisingWave to a REST-compliant catalog for Apache Iceberg.

A REST catalog uses a RESTful API to manage Iceberg table metadata. You can use a REST catalog with RisingWave when creating an Iceberg SOURCE, SINK, or CONNECTION (for internal Iceberg tables).

For a complete list of all catalog-related parameters, see the main Catalog configuration page.

Example

Examples for using a REST catalog with internal Iceberg tables, sinks, and sources.

Internal tables (recommended)

CREATE CONNECTION rest_conn WITH (
    type = 'iceberg',
    warehouse.path = 's3://my-bucket/warehouse/',
    s3.region = 'us-west-2',
    s3.access.key = '...',
    s3.secret.key = '...',
    enable_config_load = false,

    catalog.type = 'rest',
    catalog.uri = 'http://rest-catalog:8181'
);

SET iceberg_engine_connection = 'public.rest_conn';

-- `commit_checkpoint_interval` controls Iceberg commit frequency. Default: about every 60 seconds; set to 1 for faster commits and visibility.
CREATE TABLE my_internal_iceberg_table (
  id INT PRIMARY KEY,
  name VARCHAR
)
WITH (commit_checkpoint_interval = 1)
ENGINE = iceberg;

Write (CREATE SINK)

CREATE SINK my_rest_sink FROM my_mv WITH (
    connector = 'iceberg',
    type = 'upsert',
    primary_key = 'id',
    catalog.type = 'rest',
    catalog.uri = 'http://rest-catalog:8181',
    warehouse.path = 's3://my-bucket/warehouse/',
    s3.region = 'us-west-2',
    s3.access.key = '...',
    s3.secret.key = '...',
    database.name = 'my_db',
    create_table_if_not_exists = 'true',
    table.name = 'my_table'
);

Read (CREATE SOURCE)

CREATE SOURCE my_rest_source WITH (
    connector = 'iceberg',
    warehouse.path = 's3://my-bucket/warehouse/',
    database.name = 'my_db',
    table.name = 'my_table',

    catalog.type = 'rest',
    catalog.uri = 'http://rest-catalog:8181',

    s3.region = 'us-west-2',
    s3.access.key = '...',
    s3.secret.key = '...'
);

Vended credentials

Added in v2.7.0.

Vended credentials allow the REST catalog server to provide temporary credentials for accessing object storage, rather than requiring you to manage credentials directly in RisingWave.

To enable vended credentials, set vended_credentials = true along with catalog.type = 'rest':

Example: Source

CREATE SOURCE iceberg_t WITH (
    connector = 'iceberg',
    s3.endpoint = 'https://s3.ap-southeast-2.amazonaws.com',
    s3.region = 'ap-southeast-2',
    vended_credentials = true,
    s3.path.style.access = 'true',
    catalog.type = 'rest',
    catalog.uri = 'http://127.0.0.1:8181/catalog',
    warehouse.path = '<warehouse_name>',
    database.name = 'ns',
    table.name = 't'
);

Example: Sink

CREATE SINK sink_t FROM t WITH (
    connector = 'iceberg',
    type = 'append-only',
    force_append_only = 'true',
    s3.endpoint = 'https://s3.ap-southeast-2.amazonaws.com',
    s3.region = 'ap-southeast-2',
    vended_credentials = true,
    s3.path.style.access = 'true',
    catalog.type = 'rest',
    catalog.uri = 'http://127.0.0.1:8181/catalog',
    warehouse.path = '<warehouse_name>',
    database.name = 'ns',
    table.name = 't',
    create_table_if_not_exists = 'true',
    -- `commit_checkpoint_interval` controls Iceberg commit frequency. Default: about every 60 seconds; set to 1 for faster commits and visibility.
    commit_checkpoint_interval = 1
);

Deploying a Lakekeeper REST catalog

For a complete, step-by-step guide to deploying a self-hosted REST catalog with Lakekeeper, see this topic.