Serverless backfilling

Use serverless backfilling to run the backfill phase of materialized view, sink, and index creation on dedicated backfiller resource groups, reducing resource contention on your main streaming cluster.

Serverless backfilling is available starting from v2.8.0 and is disabled by default.

Overview

When you create a materialized view (MV), sink, or index on existing data, RisingWave must first process all historical records, a phase called backfill. By default, this runs on the same compute nodes as your regular streaming workloads, which can cause resource contention and increased latency for existing pipelines.

Serverless backfilling runs the backfill phase on dedicated backfiller resources — a temporary resource group that RisingWave Cloud provisions separately from your main streaming compute nodes. It is serverless because you don’t manually allocate or manage resources for each new job: after the CREATE MATERIALIZED VIEW, CREATE SINK, or CREATE INDEX request passes validation and the streaming job is ready to be created, RisingWave Cloud provisions the backfiller resource group and destroys it automatically once the job completes. Failed validation paths therefore do not provision backfiller resources. This decoupling helps large backfills complete while reducing their impact on the performance of existing streaming jobs.

Serverless backfilling is supported only in RisingWave Cloud. Enable it in the Backfilling section of the Resources page in the Cloud console first. See Configure serverless backfilling for portal setup, SKU, and replicas.

Scale your compaction resources before running heavy backfill jobs. Serverless backfilling can use a much larger backfiller SKU than your steady-state cluster — for example, a 32 RWU backfiller for a cluster that normally runs on 4 RWU. The extra write throughput increases compaction load proportionally, so a compactor sized only for steady-state traffic can fall behind and trigger compaction back pressure or write stalls (a critical alert).

We strongly recommend enabling Serverless Compaction so compactors scale automatically with the backfill load. Otherwise, scale your compactor manually before running heavy backfill jobs. See Compute-Compactor sizing and Compaction bottleneck.

How it works

When serverless backfilling is enabled for a CREATE MATERIALIZED VIEW, CREATE SINK, or CREATE INDEX statement:

1. After RisingWave finishes validation and is about to issue the create streaming job command, the serverless backfilling controller provisions a temporary resource group with dedicated backfiller nodes, sized by the Backfiller SKU and replica count you configure in the Cloud console.
2. The backfill job runs on those dedicated backfiller nodes, separately from your main streaming compute nodes, so it doesn’t compete with existing streaming jobs for resources.
3. Once all backfill fragments report completion, the job transitions to its standard steady-state resource group (the resource group of its parent database) for normal streaming execution.
4. If the cluster restarts during backfill, the job resumes from the last completed checkpoint rather than starting over.

Configuration

You must enable serverless backfilling in the RisingWave Cloud console (in the Backfilling section of the Resources page) before using it. If it isn’t enabled, CREATE MATERIALIZED VIEW, CREATE SINK, and CREATE INDEX statements that request serverless backfilling return an error.

Session variable

Enable serverless backfilling for subsequent CREATE MATERIALIZED VIEW, CREATE SINK, and CREATE INDEX statements in the current session:

SET enable_serverless_backfill = true;

The default is false. This setting only affects new DDL operations issued after the SET command; it does not change the behavior of already-running jobs.

If you set enable_serverless_backfill through dbt, use dbt-risingwave 1.11.4 or later. Earlier versions of the plugin do not apply this session variable.

Per-statement WITH clause

You can also enable serverless backfilling for a single CREATE MATERIALIZED VIEW, CREATE SINK, or CREATE INDEX statement using the cloud.serverless_backfill_enabled option in the WITH clause:

CREATE MATERIALIZED VIEW my_mv
WITH (cloud.serverless_backfill_enabled = true)
AS SELECT ...;

For sinks and indexes, use the same option in the statement-level WITH clause:

CREATE SINK my_sink AS
SELECT * FROM my_mv
WITH (
  connector = 'blackhole',
  cloud.serverless_backfill_enabled = true
);

CREATE INDEX my_index
ON my_table(id)
WITH (
  cloud.serverless_backfill_enabled = true
);

Example

The following example creates a large materialized view with serverless backfilling enabled to reduce the impact of the backfill on existing streaming workloads:

Example: serverless backfilling

-- Enable serverless backfilling for this session
SET enable_serverless_backfill = true;

-- Optionally run in the background to avoid blocking the client
SET BACKGROUND_DDL = true;

CREATE MATERIALIZED VIEW orders_summary AS
SELECT customer_id, COUNT(*) AS order_count, SUM(amount) AS total_amount
FROM orders
GROUP BY customer_id;

If you want to enable it only for one statement, use:

CREATE MATERIALIZED VIEW orders_summary
WITH (cloud.serverless_backfill_enabled = true)
AS
SELECT customer_id, COUNT(*) AS order_count, SUM(amount) AS total_amount
FROM orders
GROUP BY customer_id;

After issuing the statement, you can monitor progress with:

Monitor progress

SELECT * FROM rw_catalog.rw_ddl_progress;

Monitoring

Use the following system catalog and metrics to track serverless backfilling jobs:

Method	Description
SHOW JOBS	Lists all background DDL jobs and their current status.
SELECT * FROM rw_catalog.rw_ddl_progress	Shows per-fragment backfill progress as a percentage.
Cloud portal Metrics page	In the cluster-node panels such as CPU Utilization, look for the risingwave-backfill-* node series to confirm that the dedicated backfiller nodes are running and to observe their CPU and memory usage.

For example, the CPU Utilization panel below (on the Cloud portal Metrics page) lists each cluster node. The highlighted risingwave-backfill-0-* series is a dedicated backfiller node that RisingWave Cloud provisions for the serverless backfilling job, separate from your main compute nodes (risingwave-compute-*).

Limitations

• Serverless backfilling is supported only in RisingWave Cloud, and you must enable it in the Backfilling section of the Resources page before requesting it in SQL.

Related pages

• Configure serverless backfilling — Set the backfiller SKU, replicas, and other Cloud setup.
• Backfill — Overview of all backfill strategies in RisingWave.
• SET BACKGROUND_DDL — Run DDL operations in the background without blocking the client.
• CREATE MATERIALIZED VIEW — Full syntax and options for creating materialized views.
• CREATE SINK — Full syntax and options for creating sinks.
• CREATE INDEX — Full syntax and options for creating indexes.
• View and configure runtime parameters — All session variables, including enable_serverless_backfill.