Hosted REST Catalog

Deploy and use the REST hosted catalog, powered by Lakekeeper, to manage internal Iceberg tables in RisingWave.

This guide shows how to deploy RisingWave with Lakekeeper, a hosted REST catalog service. This setup enables you to create and manage internal Iceberg tables that use a REST-compliant catalog.

You can deploy this configuration using one of two methods:

• Docker: Use this method for local testing and development.
• Kubernetes (Helm): Use this method for production or clustered environments.

Deploy with Docker

Prerequisites

1. Ensure Docker Desktop is installed and running in your environment.
2. Clone the RisingWave repository.
   git clone https://github.com/risingwavelabs/risingwave.git
   

3. Navigate to the docker directory within the repository.
   cd risingwave/docker
   

4. Run the docker compose command to start the services. This command deploys a RisingWave cluster and the Lakekeeper catalog. A default warehouse named risingwave-warehouse is automatically created. You can view its configuration at http://localhost:8181/ui/warehouse after the services are running.
   docker compose -f docker-compose-with-lakekeeper.yml up
   

5. Connect to RisingWave to begin creating tables:
   psql -h localhost -p 4566 -d dev -U root
   

Steps

1. Create an Iceberg connection.

   CREATE CONNECTION lakekeeper_catalog_conn
   WITH (
       type = 'iceberg',
       catalog.type = 'rest',
       catalog.uri = 'http://lakekeeper:8181/catalog/',
       warehouse.path = 'risingwave-warehouse',
       s3.access.key = 'hummockadmin',
       s3.secret.key = 'hummockadmin',
       s3.path.style.access = 'true',
       s3.endpoint = 'http://minio-0:9301',
       s3.region = 'us-east-1'
   );
   

2. Set the connection as the default for your session.

   SET iceberg_engine_connection = 'public.lakekeeper_catalog_conn';
   ALTER SYSTEM SET iceberg_engine_connection = 'public.lakekeeper_catalog_conn';
   

3. Create an internal Iceberg table.

   CREATE TABLE t_lakekeeper_basic(
       id INT PRIMARY KEY,
       name VARCHAR,
       v BIGINT
   )
   WITH (commit_checkpoint_interval = 1)
   ENGINE = iceberg;
   

4. Insert data into the table.

   INSERT INTO t_lakekeeper_basic VALUES
       (1, 'alice', 100),
       (2, 'bob', 200),
       (3, 'charlie', 300);
   FLUSH;
   

5. Query the table to verify the data.

   SELECT * FROM t_lakekeeper_basic;
   

Deploy with Kubernetes (Helm)

Prerequisites

Ensure your Kubernetes environment is configured with Helm by following the installation instructions.

Steps

1. Add the RisingWave and Lakekeeper Helm repositories.

   helm repo add risingwavelabs https://risingwavelabs.github.io/helm-charts/ --force-update
   helm repo add lakekeeper https://lakekeeper.github.io/lakekeeper-charts/
   helm repo update
   

2. Create a Kubernetes namespace for the deployment.

   kubectl create namespace risingwave
   

3. Use Helm to install RisingWave with its bundled components.

   helm install -n risingwave --set tags.bundle=true risingwavelabs/risingwave
   

4. Use Helm to install Lakekeeper.

   helm install -n risingwave my-lakekeeper lakekeeper/lakekeeper
   

5. Verify that all pods are running successfully.

   kubectl get pods -n risingwave
   

6. Retrieve the default root password for the MinIO service.

   k get secret risingwave-minio -o jsonpath="{.data['root-password']}" | base64 -d
   

7. Forward the necessary ports. Run each command in a separate terminal.

   kubectl -n risingwave port-forward svc/risingwave 4567
   kubectl -n risingwave port-forward svc/risingwave-minio 9001
   kubectl -n risingwave port-forward svc/my-lakekeeper 8181
   

8. Configure the storage warehouse in the Lakekeeper UI.

   This guide uses MinIO as an example. If you use a different object store, such as Amazon S3, configure it accordingly in the Lakekeeper UI.

   Open http://localhost:9001 in a browser and create a bucket (e.g., “abc”). Then, open http://localhost:8181 and create a warehouse (e.g., “abc”), providing the following details:

   • Access key: root
   • Password: Use the password you obtained in the previous step.
   • Path style: Enable this toggle.
   • Region: us-east-1

9. Connect to RisingWave and run the test queries.

   Connect to RisingWave

   psql -h localhost -p 4567 -d dev -U root
   

   Run SQL

   create connection lakekeeper_catalog_conn
   with (
       type = 'iceberg',
       catalog.type = 'rest',
       catalog.uri = 'http://my-lakekeeper:8181/catalog/',
       warehouse.path = 'abc',       -- replace with your warehouse name
       s3.access.key = 'root',
       s3.secret.key = 'K9xXOGgEHx', -- replace with your MinIO secret key
       s3.path.style.access = 'true',
       s3.endpoint = 'http://risingwave-minio:9000',
       s3.region = 'us-east-1'
   );
   
   set iceberg_engine_connection = 'public.lakekeeper_catalog_conn';
   
   create table t_lakekeeper_basic(id int primary key, name varchar, v bigint)
   with(commit_checkpoint_interval = 1) engine = iceberg;
   
   insert into t_lakekeeper_basic values(1, 'alice', 100), (2, 'bob', 200), (3, 'charlie', 300);
   flush;
   
   select * from t_lakekeeper_basic;