Monitor a RisingWave cluster

If you run RisingWave in a Kubernetes cluster, you can monitor its performance, including the usage of resources like CPU, memory, and network, and the status of different nodes.

You can monitor your RisingWave cluster using various tools including Prometheus and Grafana, or through the RisingWave Console, which provides a built-in monitoring interface for cluster health, node status, and performance metrics. The console also offers diagnostic collection and metadata snapshot management for comprehensive cluster observability.

Install the monitoring stack

To monitor a RisingWave cluster, you can use the monitoring stack in the RisingWave-Operator repository. The stack includes Prometheus for collecting data, Grafana for visualization and alerting, and other necessary resources.

Clone or download the RisingWave-Operator repository to your environment.

git clone https://github.com/risingwavelabs/risingwave-operator.git

Run the script to install the monitoring stack. It will create the monitoring namespace and deploy all monitoring services in the namespace.

./monitoring/install.sh

Start monitoring

The RisingWave operator is now integrated with the Prometheus operator, which creates a service monitor for RisingWave and ensures that data is collected automatically.

1. Check if Prometheus is collecting data from RisingWave:

kubectl get servicemonitors -A -l risingwave/name

The expected output is like this:

NAMESPACE    NAME                         AGE
monitoring   risingwave-service-monitor   119m

2. Forward the web port of Grafana to localhost.

kubectl port-forward -n monitoring svc/prometheus-grafana 3000:http-web

The command above allows you to access Grafana from the local machine only. If you need to access Grafana from a different host, append --address 0.0.0.0 to the command to allow Grafana to accept requests from external hosts. The complete command is:

kubectl port-forward -n monitoring svc/prometheus-grafana 3000:http-web --address 0.0.0.0

3. Go to http://localhost:3000 if you access Grafana from a local machine, or http://<client_address>:3000 if you access Grafana from a different host, where <client_address> is the IP address of the machine running the Grafana service. When prompted, enter the default credentials (username: admin; password: prom-operator).
4. From Dashboards, select RisingWave -> RisingWave Dashboard. You can edit metrics in the dashboard and add new metrics.

Generate a Grafana dashboard for multiple RisingWave clusters

This section explains how to generate a Grafana dashboard for monitoring multiple RisingWave clusters deployed within a Kubernetes environment. These steps only generate the dashboard’s JSON model and can be performed on any machine with the necessary tools.

1. Clone the risingwave repository:

git clone https://github.com/risingwavelabs/risingwave.git

2. Follow the instructions in the RisingWave Grafana directory README to configure the required toolchain.

3. Generate the Grafana dashboard JSON model. Execute the following command to create the Grafana dashboard JSON model, enabling multi-cluster support:

DASHBOARD_NAMESPACE_FILTER_ENABLED=true \
DASHBOARD_RISINGWAVE_NAME_FILTER_ENABLED=true \
DASHBOARD_DYNAMIC_SOURCE=true \
DASHBOARD_UID=risingwave_dashboard \
./generate.sh

This command sets several environment variables to configure the dashboard generation:

• DASHBOARD_NAMESPACE_FILTER_ENABLED=true: Enables filtering by Kubernetes namespace.

• DASHBOARD_RISINGWAVE_NAME_FILTER_ENABLED=true: Enables filtering by RisingWave cluster name.

• DASHBOARD_DYNAMIC_SOURCE=true: Enables dynamic selection of the data source in Grafana.

• DASHBOARD_UID=risingwave_dashboard: Sets the unique identifier (UID) for the dashboard.

4. Import the generated JSON models (with the extension .gen.json) into your Grafana instance.

Troubleshooting

Namespace filter not working

If the namespace filter in your Grafana dashboard is not functioning correctly, the issue may stem from the absence of the risingwave_name label in your collected metrics. To resolve this, you can use the metricRelabelings and relabelings fields in the Prometheus Operator endpoint spec to append this label to your metrics.