Iceberg table maintenance

Configure compaction and snapshot expiration for Iceberg sinks and internal tables in RisingWave.

When RisingWave streams data into Iceberg, it generates many small data files and creates frequent snapshots. Over time, this can degrade query performance and increase storage costs. To maintain healthy tables, RisingWave provides both automatic and manual maintenance features for compaction and snapshot expiration.

• Compaction: Merges small data files and delete files into larger, optimized files to improve read performance.
• Snapshot Expiration: Removes old, unneeded snapshots and their associated data files to reclaim storage space.

Automatic maintenance

Version notes

• RisingWave introduced Iceberg automatic maintenance (default) in v2.5.0 and added compaction.type with the small-files and files-with-delete compaction types in v2.7.0.
• compaction.write_parquet_compression and compaction.write_parquet_max_row_group_rows were added in v2.9.0. The compaction.target_file_size_mb parameter now also controls the output file size for sink writes (previously only applied to compaction).
• Parameters prefixed with compaction are currently in technical preview stage and may change in future releases.

You can enable automatic maintenance to run periodically in the background for your Iceberg sinks and internal tables.

Dedicated compactor required

Automatic Iceberg maintenance requires a dedicated compactor node. Before enabling enable_compaction = true, ensure your cluster has at least one compactor node deployed. For deployment instructions and sizing guidelines, see Deploy a dedicated Iceberg compactor.

Compaction types

RisingWave supports three compaction types for Iceberg tables. You can specify the type using the compaction.type parameter.

Compaction type	Description
full	Rewrites all data files. This is the default type.
small-files	Only compacts files smaller than a specified threshold. Use the compaction.small_files_threshold_mb parameter to set the threshold.
files-with-delete	Only compacts data files that have associated delete files. Use the compaction.delete_files_count_threshold parameter to set the minimum number of delete files to trigger compaction.

The small-files and files-with-delete compaction types are only supported in Merge-on-Read mode. Copy-on-Write mode only supports the full compaction type.

Parameters

Configure automatic maintenance by specifying the following parameters in the WITH clause of a CREATE SINK or CREATE TABLE ... ENGINE = iceberg statement.

General parameters

Parameter	Description
enable_compaction	Required. Set to true to enable automatic compaction and snapshot expiration.
compaction_interval_sec	Optional. The interval in seconds between maintenance runs. Default: 3600.
enable_snapshot_expiration	Optional. Set to true to enable snapshot expiration. By default, it removes snapshots older than 5 days.
snapshot_expiration_max_age_millis	Optional. The maximum age (in milliseconds) for a snapshot to be retained. To keep only the latest snapshot, set this to 0.
snapshot_expiration_retain_last	Optional. The minimum number of snapshots to retain, regardless of their age.

Compaction parameters

Parameter	Description
compaction.type	Optional. The compaction strategy: full, small-files, or files-with-delete. Default: full.
compaction.max_snapshots_num	Optional. The maximum number of snapshots allowed since the last rewrite operation. If set, the sink will pause if this number is exceeded until compaction completes.
compaction.trigger_snapshot_count	Optional. The minimum number of snapshots since the last compaction required to trigger a new compaction. Both this threshold and the time interval must be met.
compaction.target_file_size_mb	Optional. The target output file size in MB. Applies to both sink writes and compaction. Default: 1024.
compaction.write_parquet_compression	Optional. The Parquet compression codec for output files. Accepted values: uncompressed, snappy, gzip, lzo, brotli, lz4, zstd. Default: snappy. Supports dynamic updates via ALTER SINK.
compaction.write_parquet_max_row_group_rows	Optional. The maximum number of rows per Parquet row group. Must be greater than 0. Default: 122880. Supports dynamic updates via ALTER SINK.
compaction.small_files_threshold_mb	Optional. For small-files compaction type, the threshold size in MB below which files will be compacted.
compaction.delete_files_count_threshold	Optional. For files-with-delete compaction type, the minimum number of delete files associated with a data file required to trigger compaction.

Deprecated storage config: The node-level storage config iceberg_compaction_write_parquet_max_row_group_rows is deprecated. Use the sink-level parameter compaction.write_parquet_max_row_group_rows instead.

Examples

Full compaction (default)

The following example enables automatic compaction with the default full compaction type:

CREATE TABLE my_iceberg_table (
    id INT PRIMARY KEY, 
    name VARCHAR
) WITH (
    enable_compaction = true, 
    compaction_interval_sec = 1800, 
    enable_snapshot_expiration = true,
    snapshot_expiration_retain_last = 10
) ENGINE = iceberg;

Small files compaction

For Merge-on-Read tables with many small files, use the small-files compaction type to only compact files smaller than a threshold:

CREATE SINK my_iceberg_sink FROM my_mv
WITH (
    connector = 'iceberg',
    type = 'upsert',
    primary_key = 'id',
    warehouse.path = 's3://my-bucket/warehouse',
    database.name = 'my_db',
    table.name = 'my_table',
    catalog.type = 'rest',
    catalog.uri = 'http://rest-catalog:8181',
    s3.access.key = 'your-key',
    s3.secret.key = 'your-secret',
    write_mode = 'merge-on-read',
    -- Maintenance settings
    enable_compaction = true, 
    compaction_interval_sec = 1800,
    compaction.type = 'small-files',
    compaction.small_files_threshold_mb = 128,
    compaction.target_file_size_mb = 512,
    enable_snapshot_expiration = true,
    snapshot_expiration_retain_last = 10
);

Files with delete compaction

For Merge-on-Read tables with accumulated delete files, use the files-with-delete compaction type to only compact data files that have associated delete files:

CREATE SINK my_iceberg_sink FROM my_mv
WITH (
    connector = 'iceberg',
    type = 'upsert',
    primary_key = 'id',
    warehouse.path = 's3://my-bucket/warehouse',
    database.name = 'my_db',
    table.name = 'my_table',
    catalog.type = 'rest',
    catalog.uri = 'http://rest-catalog:8181',
    s3.access.key = 'your-key',
    s3.secret.key = 'your-secret',
    write_mode = 'merge-on-read',
    -- Maintenance settings
    enable_compaction = true, 
    compaction_interval_sec = 1800,
    compaction.type = 'files-with-delete',
    compaction.delete_files_count_threshold = 100,
    compaction.trigger_snapshot_count = 10,
    enable_snapshot_expiration = true,
    snapshot_expiration_retain_last = 10
);

Manual maintenance

In addition to automatic background maintenance, you can trigger compaction and snapshot expiration manually at any time using the VACUUM command.

This gives you on-demand control over table optimization and storage cleanup.

-- Compact small files for a table
VACUUM my_iceberg_table;

-- Compact files and expire snapshots older than 2 days
VACUUM my_iceberg_table RETAIN 2 DAYS;