> ## Documentation Index
> Fetch the complete documentation index at: https://docs.risingwave.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Bring your own cloud
> This guide outlines the services that RisingWave deploys in a BYOC environment and walks you through the process of enabling BYOC in a step-by-step manner.
The Bring Your Own Cloud (BYOC) plan offers you the flexibility to tailor your cloud infrastructure instead of depending on a hosted service. It allows you to utilize the advantages of your chosen cloud provider, maintain full control over your environment, and adjust configurations to suit your specific needs.
## Architecture overview
Before creating a BYOC deployment, familiarize yourself with the following architecture. In the BYOC environment, the entire data plane is deployed in the user's space. To manage the RisingWave clusters within this environment, we deploy two key services for operation delegation:
* **Agent Service**: This service manages Kubernetes (K8s) and cloud resources. It handles tasks such as managing RisingWave Pods, storage services (including AWS S3, GCS, and Azure Blob Storage), IAM roles/accounts associated with the RisingWave cluster, network endpoints, etc.
* **RWProxy**: This is a TCP proxy that routes SQL statements from the control plane to the appropriate RisingWave instances. Control plane will issue SQL queries to retrieve cluster metadata and health status.
## Prerequisites
BYOC uses the `rwc` (RisingWave Cloud CLI) to provision and manage environments. Before continuing, install and authenticate the CLI by following [Install the RisingWave Cloud CLI](/cloud/install-cli).
Once `rwc cluster list` runs successfully in your chosen region, you are ready to create a BYOC environment.
## Create a BYOC environment
Follow the steps below to create your own cloud environment.
1. Navigate to the [**Projects**](https://cloud.risingwave.com/project/home/) page and click **Create project**.
2. Select a deployment mode. Choose **BYOC**.
3. Select a service provider. Choose your cloud platform (AWS, GCP, or Azure) and your preferred region. See [Resource and permission](#resource-and-permission) for more details.
If you're using AWS, ensure you're not using the AWS account root user for any deployment or operational tasks. Always use IAM users or roles with appropriate permissions instead.
If you're using AWS with [AWS Firewall Manager (FMS)](https://docs.aws.amazon.com/fms/latest/userguide/what-is-fms.html) security group policies enabled, please contact our [support team](mailto:cloud-support@risingwave-labs.com) before creating a BYOC environment. FMS automatically attaches managed security groups to EC2 instances, which requires additional configuration to prevent node stability issues.
4. Prepare the BYOC environment section. Click **Setup now** and follow the steps to set up your BYOC environment. Please note that the final command `rwc byoc apply --name xxx` may take 30 to 40 minutes to complete, and a progress bar will keep you informed of its progress. During this time, it's crucial to ensure a stable internet connection. If the command is interrupted or fails due to network instability, you can safely retry it.
When you run the command `rwc byoc apply --name xxx`, it will deploy some resources in your AWS/GCP/Azure environment, such as AWS S3/Google Cloud Storage/Azure Blob Storage and EKS/GKE/AKS clusters. Please do not modify the configuration of these resources. If you encounter any issues during this process, please contact our [support team](mailto:cloud-support@risingwave-labs.com).
5. Select compute size in RWU. For most users, standard sizes offer the best performance and cost balance. Choose **Custom** if the standard sizes don't meet your workload requirements. To learn more about the nodes, see the [architecture of RisingWave](/get-started/architecture).
6. Click **Summary**, then **Confirm**, and run the command that appears to set up a BYOC cluster in your environment.
Once the cluster is successfully created, you can manage it through the portal just like hosted clusters.
## Configure custom settings
RisingWave provides several custom settings for BYOC deployments. To enable this feature, you need to create a configuration file containing the custom settings. These settings can be applied when creating a new BYOC environment or updating an existing one.
Below are all supported custom settings:
### General settings
| Setting | Type | Description |
| --------------------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `container_security_context` | Object | Kubernetes [SecurityContext](https://pkg.go.dev/k8s.io/api/core/v1#SecurityContext) applied to all containers in RisingWave-managed namespaces: `cloudagent`, `rwproxy`, `risingwave-operator-system`, and `rwc-*`. |
| `pod_security_admission_labels` | Map | [Pod Security Admission labels](https://kubernetes.io/docs/concepts/security/pod-security-admission/#pod-security-admission-labels-for-namespaces) applied to all RisingWave-managed namespaces. |
| `extra_tags` | Map | Custom tags applied to cloud vendor resources managed by RisingWave. |
| `disable_external_cluster_access` | Boolean | When set to `true`, disables external (Internet-facing) access to RisingWave clusters. |
### AWS settings (`aws_settings`)
| Setting | Type | Description |
| ------------------------------ | ------ | ------------------------------------------------------------------------------ |
| `eks_node_ami_release_version` | String | Custom AMI release version for EKS nodes. If omitted, the default AMI is used. |
### Metrics settings (`metrics_settings`)
| Setting | Type | Description |
| ------------------------------------ | ------- | ---------------------------------------------------------------------------------------- |
| `discarded_risingwave_metrics_regex` | List | A list of regular expressions matching metric names to discard when scraping RisingWave. |
| `risingwave_scraping_interval_secs` | Integer | Scraping interval in seconds for RisingWave metrics. |
### Create a configuration file
1. Create a file at path `BYOC_CONFIG` with desired custom settings. You can include only the settings you need and omit others:
```yaml theme={null}
# General settings
container_security_context:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
readOnlyRootFilesystem: true
runAsNonRoot: true
runAsUser: 65521
seccompProfile:
type: RuntimeDefault
pod_security_admission_labels:
pod-security.kubernetes.io/enforce: restricted
extra_tags:
foo: bar
disable_external_cluster_access: true
# AWS-specific settings
aws_settings:
eks_node_ami_release_version: 1.32.0-20241225
# Metrics settings
metrics_settings:
discarded_risingwave_metrics_regex:
- "^risingwave_.*_total$"
risingwave_scraping_interval_secs: 30
```
2. Save the file path `$BYOC_CONFIG`, as you will use it in the later steps.
### Apply to a BYOC environment
You may either create a new BYOC environment or update your existing one to apply the custom settings.
1. Run the following command to create a new BYOC environment with custom settings. `$BYOC_CONFIG` is the file path of the config file created in the previous step.
```bash theme={null}
$ rwc byoc create \
--cidr $BYOC_CIDR \
--cloud-account-id $ACCOUNT_ID \
--name $BYOC_NAME \
--custom-settings-path $BYOC_CONFIG
```
2. Apply custom settings to the new BYOC environment. This may take up to 30 minutes.
```bash theme={null}
$ rwc byoc apply --name $BYOC_NAME
```
When updating the existing BYOC environment, the existing cluster namespaces `rwc-*` will NOT adopt the security feature update.
1. Run the following command to trigger a BYOC environment update. `$BYOC_NAME` should be the name of your existing BYOC environment.
```bash theme={null}
$ rwc byoc update \
--name $BYOC_NAME \
--version $VERSION \
--custom-settings-path $BYOC_CONFIG
```
2. Run the following command to retrieve the cluster status, until the status becomes `Ready` again. This may take up to 30 minutes.
```bash theme={null}
$ rwc byoc describe --name $BYOC_NAME
```
## Resource and permission
When you customize your cloud platform, refer to the following notes to see what we've set up for you and the permissions you need to enable.
* **Required service-linked role**
The role `AWSServiceRoleForAutoScaling` needs to be in place. If it is not ready yet, you need to create it manually. See [Create a service-linked role](https://docs.aws.amazon.com/autoscaling/ec2/userguide/autoscaling-service-linked-role.html#create-service-linked-role-manual) for detailed steps.
* **Required quota increase**
For optimal performance, the quota for managed node groups per cluster should be increased to 36 or more. See [Service quotas](https://docs.aws.amazon.com/eks/latest/userguide/service-quotas.html#sq-text) for more details.
* **Required permissions for BYOC environment creation/deletion**
The IAM role/user requires the following roles to create or delete a BYOC environment:
* AmazonDynamoDBFullAccess
* AmazonEC2FullAccess
* AmazonElasticContainerRegistryPublicReadOnly
* AmazonPrometheusFullAccess
* AmazonS3FullAccess
* AmazonVPCFullAccess
* AWSKeyManagementServicePowerUser
* CloudWatchLogsFullAccess
* IAMFullAccess
Additionally, attach a policy that includes the following permissions:
```json theme={null}
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "BYOCAdditionalAccess",
"Effect": "Allow",
"Action": [
"eks:*",
"kms:*",
"rds:DescribeDBInstances",
"rds:StopDBInstance",
"rds:StartDBInstance",
"rds:CreateDBSubnetGroup",
"rds:DeleteDBSubnetGroup",
"rds:ModifyDBSubnetGroup",
"rds:DescribeDBSubnetGroups",
"rds:AddTagsToResource",
"rds:RemoveTagsFromResource",
"rds:ListTagsForResource"
],
"Resource": "*"
}
]
}
```
* **Resources provisioned in BYOC environment**
We will set up the following resources in a BYOC environment:
* 1 VPC: including VPC, its subnets, security, and IPs to host all BYOC resources.
* 1 EKS cluster: to host all service and RisingWave clusters workloads.
* 2 S3 buckets: for RisingWave cluster data and infra state data respectively.
* 2 Internal network load balancer: to expose Agent Service and RWProxy.
* (Optional) 1 External network load balancer: to expose RWProxy to the Internet.
* A few IAM roles for EKS and K8s workloads, and each role is granted the least privilege it requires.
* **Required permission for deployed services**
The deployed service will need the permissions to perform the following actions:
* AmazonDynamoDBFullAccess
* AmazonEC2FullAccess
* AmazonElasticContainerRegistryPublicReadOnly
* AmazonPrometheusFullAccess
* AmazonS3FullAccess
* AmazonVPCFullAccess
* AWSKeyManagementServicePowerUser
* CloudWatchLogsFullAccess
* IAMFullAccess
Additionally, attach a policy that includes the following permissions:
```json theme={null}
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "BYOCAdditionalAccess",
"Effect": "Allow",
"Action": [
"eks:*",
"kms:*",
"rds:DescribeDBInstances",
"rds:StopDBInstance",
"rds:StartDBInstance",
"rds:CreateDBSubnetGroup",
"rds:DeleteDBSubnetGroup",
"rds:ModifyDBSubnetGroup",
"rds:DescribeDBSubnetGroups",
"rds:AddTagsToResource",
"rds:RemoveTagsFromResource",
"rds:ListTagsForResource"
],
"Resource": "*"
}
]
}
```
* **Required APIs for BYOC environment creation/deletion**
You need to enable the following APIs to create or delete a BYOC environment:
* **Compute Engine API** for VPC resources provision.
* **Cloud DNS API** for VPC private service connect setup.
* **Kubernetes Engine API** for provisioning the GKE cluster the data plane is hosted.
* **Cloud Resource Manager API** for IAM provisioning.
* **Service Networking API** for Cloud SQL (as meta store) connection.
* **Cloud SQL Admin API** for Cloud SQL (as meta store) provisioning.
* **Required permission for BYOC environment creation/deletion**
Before running the command-line interface to create or delete a BYOC environment, you need to have a Google IAM (IAM user/Service account) with the following roles.
* [Kubernetes Engine Admin](https://cloud.google.com/iam/docs/understanding-roles#container.admin)
* [Compute Network Admin](https://cloud.google.com/iam/docs/understanding-roles#compute.networkAdmin)
* [Compute Security Admin](https://cloud.google.com/iam/docs/understanding-roles#compute.securityAdmin)
* [Storage Admin](https://cloud.google.com/iam/docs/understanding-roles#storage.admin)
* [Security Admin](https://cloud.google.com/iam/docs/understanding-roles#iam.securityAdmin)
* [Service Account Admin](https://cloud.google.com/iam/docs/understanding-roles#iam.serviceAccountAdmin)
These permissions are only required for creating or deleting a BYOC environment. Once the environment is up and running, limited permissions are needed to operate the services.
* **Resources provisioned in BYOC environment**
We will set up the following resources in a BYOC environment:
* 1 VPC: including VPC, its subnets, firewalls, IPs to host all BYOC resources.
* 1 GKE cluster: to host all service and RisingWave clusters workloads.
* 2 GCS buckets: for RisingWave cluster data and infra state data respectively.
* 2 Internal network load balancer: to expose Agent Service and RWProxy.
* (Optional) 1 External network load balancer: to expose RWProxy to the Internet.
* A few service accounts for GKE and K8s workloads, and each role is granted the least privilege it requires.
* **Required permission for deployed services**
We will provision a Google Service Account for the deployed services. The services require the following permissions:
* [Kubernetes Engine Admin](https://cloud.google.com/iam/docs/understanding-roles#container.admin) for infra management.
* [Compute Network Admin](https://cloud.google.com/iam/docs/understanding-roles#compute.networkAdmin) for managing private links for the source and sink of RisingWave clusters and infra management.
* [Compute Security Admin](https://cloud.google.com/iam/docs/understanding-roles#compute.securityAdmin) for infra management.
* [Storage Admin](https://cloud.google.com/iam/docs/understanding-roles#storage.admin) for managing GCS objects and bucket access for RisingWave clusters and infra management.
* [Security Admin](https://cloud.google.com/iam/docs/understanding-roles#iam.securityAdmin) for infra management.
* [Service Account Admin](https://cloud.google.com/iam/docs/understanding-roles#iam.serviceAccountAdmin) for managing the IAM service account RisingWave clusters runs as and infra management.
* **Required feature flags**
Feature flag `EnableAPIServerVnetIntegrationPreview` must be enabled for the subscription to deploy a BYOC environment, see [Instruction](https://learn.microsoft.com/en-us/azure/aks/api-server-vnet-integration#register-the-enableapiservervnetintegrationpreview-feature-flag) for more details.
The following providers must be registered in the subscription, see [Prerequisites](https://learn.microsoft.com/en-us/azure/azure-monitor/containers/kubernetes-monitoring-enable?tabs=cli#prerequisites) for more details.
* Microsoft.ContainerService
* Microsoft.Insights
* Microsoft.AlertsManagement
* Microsoft.Monitor
* **Required permission for BYOC environment creation/deletion**
We recommend utilizing a service principal or user with owner permissions of the Azure subscription to provision the infrastructure.
Additionally, the service principal or user should be granted the following Azure AD API permissions.
For service principal authentication (one of the following combinations):
* `AppRoleAssignment.ReadWrite.All` and `Application.Read.All`
* `AppRoleAssignment.ReadWrite.All` and `Directory.Read.All`
* `Application.ReadWrite.All`
* `Directory.ReadWrite.All`
For user principal authentication:
* `Global Administrator`
* **Resources provisioned in BYOC environment**
We will set up the following resources in a BYOC environment:
* 1 VPC: including VPC, its subnets, firewalls, IPs to host all BYOC resources.
* 1 AKS cluster: to host all service and RisingWave clusters workloads.
* 2 Azure storage accounts each with one blob container in it: for RisingWave cluster data and infra state data respectively.
* 2 Internal network load balancer: to expose Agent Service and RWProxy.
* (Optional) 1 External network load balancer: to expose RWProxy to the Internet.
* A few user-assigned identities for AKS workloads, and each identity is granted the least privilege it requires.
* **Required permission for deployed services**
* Role [Storage Blob Data Contributor](https://learn.microsoft.com/en-us/azure/role-based-access-control/built-in-roles/storage#storage-blob-data-contributor)
* Role [Network Contributor](https://learn.microsoft.com/en-us/azure/role-based-access-control/built-in-roles/networking#network-contributor)
* Role [Managed Identity Contributor](https://learn.microsoft.com/en-us/azure/role-based-access-control/built-in-roles/identity#managed-identity-contributor)
* Role [Role Based Access Control Administrator](https://learn.microsoft.com/en-us/azure/role-based-access-control/built-in-roles/general#role-based-access-control-administrator)
* Role [Monitoring Reader](https://learn.microsoft.com/en-us/azure/role-based-access-control/built-in-roles/monitor#monitoring-reader)
* A custom role with the following permissions:
- Microsoft.Resources/subscriptions/resourceGroups/read
- Microsoft.Resources/subscriptions/resourceGroups/write
- Microsoft.Resources/subscriptions/resourceGroups/delete
- Microsoft.Resources/subscriptions/resourcegroups/deployments/read
- Microsoft.Resources/subscriptions/resourcegroups/deployments/write
- Microsoft.Resources/subscriptions/resourcegroups/resources/read
- Microsoft.Resources/subscriptions/resourcegroups/deployments/operationstatuses/read
- Microsoft.Resources/subscriptions/resourcegroups/deployments/operations/read
- Microsoft.ManagedServices/\*
- microsoft.monitor/accounts/read
- microsoft.monitor/accounts/write
- microsoft.monitor/accounts/delete
- microsoft.monitor/accounts/privateEndpointConnectionsApproval/action
- Microsoft.Authorization/roleAssignments/read
- Microsoft.Authorization/roleAssignments/write
- Microsoft.Authorization/roleAssignments/delete
- Microsoft.OperationalInsights/workspaces/read
- Microsoft.OperationalInsights/workspaces/delete
- Microsoft.OperationalInsights/workspaces/sharedkeys/read
- Microsoft.OperationsManagement/solutions/read
- Microsoft.OperationsManagement/solutions/write
- Microsoft.OperationsManagement/solutions/delete
- Microsoft.ManagedIdentity/userAssignedIdentities/assign/action
- Microsoft.DBforPostgreSQL/flexibleServers/start/action
- Microsoft.DBforPostgreSQL/flexibleServers/stop/action
- Microsoft.DBforPostgreSQL/flexibleServers/read
## Data encryption
RisingWave Cloud implements comprehensive data encryption to protect your information:
### Data at rest
* **S3 storage**: All data stored in AWS S3 is automatically encrypted at rest using AWS default encryption keys
* **EBS volumes**: All EBS volumes are encrypted at rest using AWS KMS managed keys
* KMS keys are automatically rotated according to AWS security best practices
* This includes all data stored on EBS volumes attached to RisingWave nodes
### Data in transit
* Client connections to RisingWave are secured using SSL/TLS
## Delete a BYOC environment
Follow the steps below to delete a BYOC environment deployed in your cloud.
1. Delete all BYOC projects running in the environment. Navigate to the [**Projects**](https://cloud.risingwave.com/project/home/) page, click the delete icon to delete all of your BYOC projects.
2. Delete resources you created that are not managed by RisingWave, such as VPC Peerings, GCP Firewalls, and other common resources you might have used.
3. Open the terminal and execute the following commands:
```bash theme={null}
$ rwc byoc terminate --name default-byoc-environment # This may take 2-3 minutes.
$ rwc byoc delete --name default-byoc-environment # This may take 30-40 minutes.
```