Bring your own cloud

This guide outlines the services that RisingWave deploys in a BYOC environment and walks you through the process of enabling BYOC in a step-by-step manner.

The Bring Your Own Cloud (BYOC) plan offers you the flexibility to tailor your cloud infrastructure instead of depending on a hosted service. It allows you to utilize the advantages of your chosen cloud provider, maintain full control over your environment, and adjust configurations to suit your specific needs.

Architecture overview

Before creating a BYOC deployment, familiarize yourself with the following architecture. In the BYOC environment, the entire data plane is deployed in the user’s space. To manage the RisingWave clusters within this environment, we deploy two key services for operation delegation:

• Agent Service: This service manages Kubernetes (K8s) and cloud resources. It handles tasks such as managing RisingWave Pods, storage services (including AWS S3, GCS, and Azure Blob Storage), IAM roles/accounts associated with the RisingWave cluster, network endpoints, etc.
• RWProxy: This is a TCP proxy that routes SQL statements from the control plane to the appropriate RisingWave instances. Control plane will issue SQL queries to retrieve cluster metadata and health status.

Prerequisites

BYOC uses the rwc (RisingWave Cloud CLI) to provision and manage environments. Before continuing, install and authenticate the CLI by following Install the RisingWave Cloud CLI.

Once rwc cluster list runs successfully in your chosen region, you are ready to create a BYOC environment.

Choose an IPv4 CIDR block of /19 or larger for a new BYOC environment, with a prefix length from /15 through /19. For example, 10.0.0.0/19 meets the minimum size. Private AWS environments with Internet access disabled can use /21 or larger, with a prefix length from /15 through /21. Smaller address ranges do not provide enough IP capacity for the supported infrastructure.

Create a BYOC environment

Follow the steps below to create your own cloud environment.

1. Navigate to the Projects page and click Create project.

2. Select a deployment mode. Choose BYOC.

3. Select a service provider. Choose your cloud platform (AWS, GCP, or Azure) and your preferred region. See Resource and permission for more details.

   If you’re using AWS, ensure you’re not using the AWS account root user for any deployment or operational tasks. Always use IAM users or roles with appropriate permissions instead.

   If you’re using AWS with AWS Firewall Manager (FMS) security group policies enabled, please contact our support team before creating a BYOC environment. FMS automatically attaches managed security groups to EC2 instances, which requires additional configuration to prevent node stability issues.

4. Prepare the BYOC environment section. Click Setup now and follow the steps to set up your BYOC environment. Please note that the final command rwc byoc apply --name xxx may take 30 to 40 minutes to complete, and a progress bar will keep you informed of its progress. During this time, it’s crucial to ensure a stable internet connection. If the command is interrupted or fails due to network instability, you can safely retry it.

   When you run the command rwc byoc apply --name xxx, it will deploy some resources in your AWS/GCP/Azure environment, such as AWS S3/Google Cloud Storage/Azure Blob Storage and EKS/GKE/AKS clusters. Please do not modify the configuration of these resources. If you encounter any issues during this process, please contact our support team.

5. Select compute size in RWU. For most users, standard sizes offer the best performance and cost balance. Choose Custom if the standard sizes don’t meet your workload requirements. To learn more about the nodes, see the architecture of RisingWave.

6. Click Summary, then Confirm, and run the command that appears to set up a BYOC cluster in your environment.

Once the cluster is successfully created, you can manage it through the portal just like hosted clusters.

Configure custom settings

RisingWave provides several custom settings for BYOC deployments. To enable this feature, you need to create a configuration file containing the custom settings. These settings can be applied when creating a new BYOC environment or updating an existing one.

Below are all supported custom settings:

General settings

Setting	Type	Description
region	String	Cloud region where the BYOC environment is deployed. If omitted, the environment uses the region selected for the RisingWave Cloud control plane. It cannot be changed after creation; include the unchanged value in later custom-settings updates.
container_security_context	Object	Kubernetes SecurityContext applied to all containers in RisingWave-managed namespaces: cloudagent, rwproxy, risingwave-operator-system, and rwc-*.
pod_security_context	Object	Kubernetes PodSecurityContext applied to all RisingWave-managed pods in cloudagent, rwproxy, risingwave-operator-system, and rwc-*.
pod_security_admission_labels	Map	Pod Security Admission labels applied to all RisingWave-managed namespaces.
extra_tags	Map	Custom tags applied to cloud vendor resources managed by RisingWave.
disable_external_cluster_access	Boolean	When set to true, disables external (Internet-facing) access to RisingWave clusters.
enable_heap_storage	Boolean	Enables object storage FUSE mount for heap dump persistence in BYOC clusters. If omitted, this is enabled by default on AWS and GCP.
disable_public_kubernetes_endpoint	Boolean	When set to true, restricts the Kubernetes control-plane API to in-VPC access only. It must be false when creating or terminating a BYOC environment, and can be enabled later through an update.

Deploy in a different region

By default, RisingWave deploys your BYOC environment in the same region as its RisingWave Cloud control plane. To deploy the environment in a different region, set the top-level region field when you create it.

The control plane and BYOC environment must use the same cloud provider. Cross-region BYOC is currently supported on AWS and GCP. Azure environments and private environments without internet access must use the control-plane region.

For a cross-region environment, you must also select exactly three unique zones in the target region:

• On AWS, set aws_settings.zone_ids to AWS Availability Zone IDs, such as usw2-az1.
• On GCP, set gcp_settings.zones to zone names, such as us-west1-a.

AWS

region: us-west-2
aws_settings:
  zone_ids:
    - usw2-az1
    - usw2-az2
    - usw2-az3

GCP

region: us-west1
gcp_settings:
  zones:
    - us-west1-a
    - us-west1-b
    - us-west1-c

RisingWave validates that the zones belong to the selected environment region. The environment region and zones cannot be changed after creation. Include their existing values in any later custom-settings update; use rwc byoc custom-settings describe --name <environment-name> to retrieve the current settings. To use another region or set of zones, create a new BYOC environment.

Cross-region GCP environments can be created only in regions enabled by RisingWave. If validation reports that the selected region is unavailable, contact our support team.

AWS settings (aws_settings)

Setting	Type	Description
eks_node_ami_release_version	String	Custom AMI release version for EKS nodes. If omitted, the default AMI is used.
zone_ids	List	AWS Availability Zone IDs, such as use1-az2, where the BYOC environment is provisioned. Provide exactly 3 unique IDs from the environment region. It is required for a cross-region environment; if omitted for a same-region environment, RisingWave uses the control-plane default. It cannot be changed after creation; include the unchanged value in later custom-settings updates.
additional_node_security_group_ids	List	Extra security group IDs to include in the Karpenter EC2NodeClass. Use this when external systems such as AWS Firewall Manager attach additional security groups to instances, to avoid SecurityGroupDrift on repeated Terraform applies.
enable_hosted_telemetry_lb	Boolean	Exposes hosted telemetry through internal load balancer endpoints. To connect these endpoints to Grafana, see Connect BYOC telemetry to Grafana.
iam_role_permissions_boundary	String	ARN of an IAM permissions boundary policy to attach to all IAM roles created by BYOC Terraform. Use this when your AWS account attaches a permissions boundary out of band, so Terraform does not report iam:DeleteRolePermissionsBoundary drift on every apply.

GCP settings (gcp_settings)

Setting	Type	Description
zones	List	GCP zones, such as us-west1-a, where the BYOC environment is provisioned. Provide exactly 3 unique zones from the environment region. It is required for a cross-region environment; if omitted for a same-region environment, RisingWave uses the control-plane default. It cannot be changed after creation; include the unchanged value in later custom-settings updates.
enable_hosted_telemetry_lb	Boolean	Exposes hosted telemetry through internal load balancer endpoints. To connect these endpoints to Grafana, see Connect BYOC telemetry to Grafana.

Metrics settings (metrics_settings)

Setting	Type	Description
discarded_risingwave_metrics_regex	List	A list of regular expressions matching metric names to discard when scraping RisingWave.
risingwave_scraping_interval_secs	Integer	Scraping interval in seconds for RisingWave metrics.

Create a configuration file

1. Create a file at path BYOC_CONFIG with desired custom settings. You can include only the settings you need and omit others:

   # General settings
   container_security_context:
     allowPrivilegeEscalation: false
     capabilities:
       drop:
         - ALL
     readOnlyRootFilesystem: true
     runAsNonRoot: true
     runAsUser: 65521
     seccompProfile:
       type: RuntimeDefault
   pod_security_context:
     fsGroup: 65521
     runAsNonRoot: true
     seccompProfile:
       type: RuntimeDefault
   pod_security_admission_labels:
     pod-security.kubernetes.io/enforce: restricted
   extra_tags:
     foo: bar
   disable_external_cluster_access: true
   enable_heap_storage: true
   disable_public_kubernetes_endpoint: false
   
   # AWS-specific settings
   aws_settings:
     eks_node_ami_release_version: 1.32.0-20241225
     additional_node_security_group_ids:
       - sg-0123456789abcdef0
     enable_hosted_telemetry_lb: true
     iam_role_permissions_boundary: arn:aws:iam::123456789012:policy/example-permissions-boundary
   
   # GCP-specific settings
   gcp_settings:
     enable_hosted_telemetry_lb: true
   
   # Metrics settings
   metrics_settings:
     discarded_risingwave_metrics_regex:
       - "^risingwave_.*_total$"
     risingwave_scraping_interval_secs: 30
   

2. Save the file path $BYOC_CONFIG, as you will use it in the later steps.

Apply to a BYOC environment

You may either create a new BYOC environment or update your existing one to apply the custom settings.

Create a new BYOC environment

1. Run the following command to create a new BYOC environment with custom settings. $BYOC_CONFIG is the file path of the config file created in the previous step. Set $BYOC_CIDR to an IPv4 CIDR block that meets the CIDR size requirements, such as 10.0.0.0/19.

$ rwc byoc create \
	--cidr $BYOC_CIDR \
	--cloud-account-id $ACCOUNT_ID \
	--name $BYOC_NAME \
	--custom-settings-path $BYOC_CONFIG

2. Apply custom settings to the new BYOC environment. This may take up to 30 minutes.

$ rwc byoc apply --name $BYOC_NAME

Update an existing BYOC environment

When updating the existing BYOC environment, the existing cluster namespaces rwc-* will NOT adopt the security feature update.

1. Run the following command to trigger a BYOC environment update. $BYOC_NAME should be the name of your existing BYOC environment.

$ rwc byoc update \
  --name $BYOC_NAME \
  --version $VERSION \
  --custom-settings-path $BYOC_CONFIG

2. Run the following command to retrieve the cluster status, until the status becomes Ready again. This may take up to 30 minutes.

$ rwc byoc describe --name $BYOC_NAME

Resource and permission

BYOC uses two types of cloud identities with different permission lifecycles:

Identity	Where it lives	Permission lifecycle	Can permissions be removed after setup?
Operator identity (runs rwc byoc apply and rwc byoc delete)	A customer-managed user, role, or service principal	Needed while the CLI creates or deletes the BYOC environment	Yes. Re-grant the permissions before a future apply or delete operation.
Cloud Agent service account	Inside the BYOC environment	Needed for the entire lifetime of the environment to create and update RisingWave clusters and manage environment components	No. Narrowing its permissions breaks environment updates and cluster creation.

The predefined roles and policies below are required by the current BYOC implementation. Some of them, such as AWS *FullAccess policies, GCP *Admin roles, and Azure Contributor roles, are broader than the minimum set of API actions that the services invoke. RisingWave may narrow these permissions in a future release.

There is currently no automatic permission downgrade for the Cloud Agent service account after setup. Do not manually remove or narrow its roles, or the roles of its supporting service identities. Deleting an individual RisingWave cluster does not remove these permissions while the BYOC environment is retained. They are cleaned up only when you delete the BYOC environment with rwc byoc delete.

AWS

• Required service-linked role

  The role AWSServiceRoleForAutoScaling needs to be in place. If it is not ready yet, you need to create it manually. See Create a service-linked role for detailed steps.

• Required quota increase

  For optimal performance, the quota for managed node groups per cluster should be increased to 36 or more. See Service quotas for more details.

• Operator identity — permissions to run rwc byoc apply and rwc byoc delete

  The customer-managed IAM role or user that runs the CLI requires the following managed policies while creating or deleting a BYOC environment:

  • AmazonDynamoDBFullAccess
  • AmazonEC2FullAccess
  • AmazonElasticContainerRegistryPublicReadOnly
  • AmazonPrometheusFullAccess
  • AmazonS3FullAccess
  • AmazonVPCFullAccess
  • AWSKeyManagementServicePowerUser
  • CloudWatchLogsFullAccess
  • IAMFullAccess

  Additionally, attach a policy that includes the following permissions:

  {
     "Version": "2012-10-17",
     "Statement": [
        {
              "Sid": "BYOCAdditionalAccess",
              "Effect": "Allow",
              "Action": [
                 "eks:*",
                 "kms:*",
                 "rds:DescribeDBInstances",
                 "rds:StopDBInstance",
                 "rds:StartDBInstance",
                 "rds:CreateDBSubnetGroup",
                 "rds:DeleteDBSubnetGroup",
                 "rds:ModifyDBSubnetGroup",
                 "rds:DescribeDBSubnetGroups",
                 "rds:AddTagsToResource",
                 "rds:RemoveTagsFromResource",
                 "rds:ListTagsForResource"
              ],
              "Resource": "*"
        }
     ]
  }
  

  For a cross-region environment, the IAM role/user must also allow the vpce:AllowMultiRegion permission-only action. Ensure that your AWS Organizations service control policies do not deny this action.

  You can remove these permissions from the operator identity after the command completes. Re-grant them before using that identity for a future rwc byoc apply or rwc byoc delete operation.

• Resources provisioned in BYOC environment

  We will set up the following resources in a BYOC environment:

  • 1 VPC: including VPC, its subnets, security, and IPs to host all BYOC resources.
  • 1 EKS cluster: to host all service and RisingWave clusters workloads.
  • 2 S3 buckets: for RisingWave cluster data and infra state data respectively.
  • 2 Internal network load balancer: to expose Agent Service and RWProxy.
  • (Optional) 1 External network load balancer: to expose RWProxy to the Internet.
  • IAM roles for the Cloud Agent, EKS, and Kubernetes workloads.

• Cloud Agent service account — permissions used inside the BYOC environment

  rwc byoc apply creates an IAM role for the Cloud Agent’s Kubernetes service account. The Cloud Agent uses this role for ongoing environment updates and cluster management. Terraform attaches:

  • The AWS managed policy AmazonElasticContainerRegistryPublicReadOnly.

  • The AWS managed policy AmazonEC2ContainerRegistryPullOnly when private Helm or container registries are enabled.

  • A custom IAM policy with the following actions:

    • dynamodb:*, ec2:*, elasticloadbalancing:*, autoscaling:*, s3:*, logs:*, iam:*, sqs:*, events:*, eks:*, and kms:*
    • rds:DescribeDBInstances, rds:StopDBInstance, rds:StartDBInstance, rds:CreateDBSubnetGroup, rds:DeleteDBSubnetGroup, rds:ModifyDBSubnetGroup, rds:DescribeDBSubnetGroups, rds:AddTagsToResource, rds:RemoveTagsFromResource, and rds:ListTagsForResource
    • route53:AssociateVPCWithHostedZone when the private Terraform registry is enabled
    • cloudformation:* and route53resolver:* when private AWS API access is enabled

  Terraform also creates supporting service identities. The ACK EC2 and RDS controllers use AmazonEC2FullAccess and AmazonRDSFullAccess, respectively. The ACK IAM controller uses a custom policy for managing the IAM resources required by RisingWave clusters.

GCP

• Required APIs for the operator identity

  You need to enable the following APIs to create or delete a BYOC environment:

  • Compute Engine API for VPC resources provision.
  • Cloud DNS API for VPC private service connect setup.
  • Kubernetes Engine API for provisioning the GKE cluster the data plane is hosted.
  • Cloud Resource Manager API for IAM provisioning.
  • Service Networking API for Cloud SQL (as meta store) connection.
  • Cloud SQL Admin API for Cloud SQL (as meta store) provisioning.

• Operator identity — permissions to run rwc byoc apply and rwc byoc delete

  The customer-managed Google IAM user or service account that runs the CLI requires the following roles while creating or deleting a BYOC environment:

  • Kubernetes Engine Admin
  • Compute Network Admin
  • Compute Security Admin
  • Storage Admin
  • Security Admin
  • Service Account Admin

  You can remove these roles from the operator identity after the command completes. Re-grant them before using that identity for a future rwc byoc apply or rwc byoc delete operation.

• Resources provisioned in BYOC environment

  We will set up the following resources in a BYOC environment:

  • 1 VPC: including VPC, its subnets, firewalls, IPs to host all BYOC resources.
  • 1 GKE cluster: to host all service and RisingWave clusters workloads.
  • 2 GCS buckets: for RisingWave cluster data and infra state data respectively.
  • 2 Internal network load balancer: to expose Agent Service and RWProxy.
  • (Optional) 1 External network load balancer: to expose RWProxy to the Internet.
  • Service accounts for the Cloud Agent, GKE, and Kubernetes workloads.

• Cloud Agent service account — permissions used inside the BYOC environment

  rwc byoc apply creates a Google service account bound to the Cloud Agent’s Kubernetes service account. The Cloud Agent uses it for ongoing environment updates and cluster management. Terraform grants the following roles:

  • roles/storage.objectAdmin, scoped by a condition to the RisingWave data bucket
  • roles/cloudsql.admin
  • roles/compute.loadBalancerAdmin
  • roles/compute.networkAdmin
  • roles/compute.securityAdmin
  • roles/dns.admin
  • roles/container.admin
  • roles/iam.securityAdmin
  • roles/iam.serviceAccountAdmin
  • roles/iam.serviceAccountUser
  • roles/storage.admin

  The supporting Config Connector service account also receives roles/iam.serviceAccountAdmin, roles/storage.admin, roles/compute.networkAdmin, and roles/cloudsql.admin.

Azure

• Required feature flags

  Feature flag EnableAPIServerVnetIntegrationPreview must be enabled for the subscription to deploy a BYOC environment, see Instruction for more details.

  The following providers must be registered in the subscription, see Prerequisites for more details.

  • Microsoft.ContainerService
  • Microsoft.Insights
  • Microsoft.AlertsManagement
  • Microsoft.Monitor

• Operator identity — permissions to run rwc byoc apply and rwc byoc delete

  The customer-managed service principal or user that runs the CLI requires the Owner role on the Azure subscription while creating or deleting a BYOC environment.

  Additionally, the service principal or user should be granted the following Azure AD API permissions.

  For service principal authentication (one of the following combinations):

  • AppRoleAssignment.ReadWrite.All and Application.Read.All
  • AppRoleAssignment.ReadWrite.All and Directory.Read.All
  • Application.ReadWrite.All
  • Directory.ReadWrite.All

  For user principal authentication:

  • Global Administrator

  You can remove these permissions from the operator identity after the command completes. Re-grant them before using that identity for a future rwc byoc apply or rwc byoc delete operation.

• Resources provisioned in BYOC environment

  We will set up the following resources in a BYOC environment:

  • 1 VPC: including VPC, its subnets, firewalls, IPs to host all BYOC resources.
  • 1 AKS cluster: to host all service and RisingWave clusters workloads.
  • 2 Azure storage accounts each with one blob container in it: for RisingWave cluster data and infra state data respectively.
  • 2 Internal network load balancer: to expose Agent Service and RWProxy.
  • (Optional) 1 External network load balancer: to expose RWProxy to the Internet.
  • User-assigned identities for the Cloud Agent, AKS, and Kubernetes workloads.

• Cloud Agent service account — permissions used inside the BYOC environment

  rwc byoc apply creates a user-assigned managed identity for the Cloud Agent’s Kubernetes service account. The Cloud Agent uses it for ongoing environment updates and cluster management. Terraform grants the following Azure roles:

  • Storage Blob Data Contributor, scoped to the RisingWave data container
  • Storage Account Contributor
  • Monitoring Contributor
  • Azure Kubernetes Service Contributor Role
  • Network Contributor
  • User Access Administrator
  • Managed Identity Contributor
  • A custom resource-group role with the following permissions:
  Resource-group role permissions

  • Microsoft.Resources/subscriptions/resourceGroups/read
  • Microsoft.Resources/subscriptions/resourceGroups/write
  • Microsoft.Resources/subscriptions/resourceGroups/delete
  • Microsoft.Resources/subscriptions/resourcegroups/deployments/read
  • Microsoft.Resources/subscriptions/resourcegroups/deployments/write
  • Microsoft.Resources/subscriptions/resourcegroups/resources/read
  • Microsoft.Resources/subscriptions/resourcegroups/deployments/operationstatuses/read
  • Microsoft.Resources/subscriptions/resourcegroups/deployments/operations/read
  • A custom BYOC administration role with the following permissions:
  BYOC administration role permissions

  • Microsoft.ManagedServices/*
  • microsoft.monitor/accounts/read
  • microsoft.monitor/accounts/write
  • microsoft.monitor/accounts/delete
  • microsoft.monitor/accounts/privateEndpointConnectionsApproval/action
  • Microsoft.Authorization/roleAssignments/read
  • Microsoft.Authorization/roleAssignments/write
  • Microsoft.Authorization/roleAssignments/delete
  • Microsoft.OperationalInsights/workspaces/read
  • Microsoft.OperationalInsights/workspaces/delete
  • Microsoft.OperationalInsights/workspaces/sharedkeys/read
  • Microsoft.OperationsManagement/solutions/read
  • Microsoft.OperationsManagement/solutions/write
  • Microsoft.OperationsManagement/solutions/delete
  • Microsoft.ManagedIdentity/userAssignedIdentities/assign/action
  • Microsoft.DBforPostgreSQL/flexibleServers/start/action
  • Microsoft.DBforPostgreSQL/flexibleServers/stop/action
  • Microsoft.DBforPostgreSQL/flexibleServers/read

  The Cloud Agent identity also receives the Microsoft Graph application roles Application.ReadWrite.OwnedBy, AppRoleAssignment.ReadWrite.All, and Application.Read.All.

  Terraform creates two additional supporting identities:

  • The Azure Service Operator identity receives Network Contributor, Managed Identity Contributor, Role Based Access Control Administrator, and a custom role with Microsoft.ManagedIdentity/userAssignedIdentities/federatedIdentityCredentials/* and Microsoft.DBForPostgreSQL/*.
  • The monitoring service principal receives Monitoring Reader on the environment’s network and tenant resource groups.

Data encryption

RisingWave Cloud implements comprehensive data encryption to protect your information:

Data at rest

• S3 storage: All data stored in AWS S3 is automatically encrypted at rest using AWS default encryption keys
• EBS volumes: All EBS volumes are encrypted at rest using AWS KMS managed keys
  • KMS keys are automatically rotated according to AWS security best practices
  • This includes all data stored on EBS volumes attached to RisingWave nodes

Data in transit

• Client connections to RisingWave are secured using SSL/TLS

Delete a BYOC environment

Follow the steps below to delete a BYOC environment deployed in your cloud.

1. Delete all BYOC projects running in the environment. Navigate to the Projects page, click the delete icon to delete all of your BYOC projects.

2. Delete resources you created that are not managed by RisingWave, such as VPC Peerings, GCP Firewalls, and other common resources you might have used.

3. Open the terminal and execute the following commands:

   $ rwc byoc terminate --name default-byoc-environment # This may take 2-3 minutes.
   $ rwc byoc delete --name default-byoc-environment # This may take 30-40 minutes.