Ingest data from GitHub webhook

Ingest GitHub events directly into your RisingWave database for real-time processing and analytics.

GitHub webhook allows you to build or set up integrations that subscribe to certain events on GitHub.com. When one of those events is triggered, GitHub sends an HTTP POST payload to the webhook’s configured URL. Webhooks can be used to update an external issue tracker, trigger CI builds, update a backup mirror, or even deploy to your production server.

This guide will walk through the steps to set up RisingWave as a destination for GitHub webhooks.

New to RisingWave webhooks? Before configuring GitHub, it’s recommended to run a quick local smoke test to confirm your RisingWave webhook listener is reachable and your table is ingesting events. See Webhook endpoint and Quick local smoke test.

1. Create a table in RisingWave

Next, create a table configured to accept webhook data from GitHub.

-- Optional (recommended): store the shared secret as a RisingWave secret.
-- If you don't have access to secrets, skip this and use the raw-string example below.
CREATE SECRET test_secret WITH (backend = 'meta') AS 'TEST_WEBHOOK';

CREATE TABLE wbhtable (
  data JSONB
) WITH (
  connector = 'webhook',
  [ is_batched = true ]
) VALIDATE SECRET test_secret AS secure_compare(
  headers->>'x-hub-signature-256', -- Example header value: `sha256=f37a93a68fef1505d75e920a15d0543199557be72d2182e5cf8c15d7f9a6260f`
  'sha256=' || encode(hmac(test_secret, data, 'sha256'), 'hex')
);

Parameter or clause	Description
data JSONB	Defines the name of column to store the JSON payload from the webhook. Currently, only JSONB type is supported for webhook tables.
is_batched	Optional, set to true to enable batch ingestion of multiple JSON lines in a single request.
SECRET <secret>	Use an existing secret to verify incoming webhook requests. This is optional. You can also use a raw string to verify.
headers->>'...'	Extracts the signature provided by GitHub in the x-hub-signature-256 HTTP header. In secure_compare() function, the whole HTTP header is interpreted as a JSONB object, and you can access the header value using the ->> operator, but only the lower-case header names in the ->> operator, otherwise the verification will fail.
'sha256=' || encode(...)	Computes the expected signature. In the example above, it generates an HMAC SHA-256 hash of the payload (data) using the secret (test_secret), encodes it in hexadecimal, and prefixes it with sha256=.
secure_compare(...)	Validates requests by matching the header signature against the computed signature, ensuring only authenticated requests are processed. The secure_compare() function compares two strings in a fixed amount of time, regardless of whether they are equal or not, ensuring that the comparison is secure and resistant to timing attacks.

In GitHub webhook, you can choose between SHA-1 and SHA-256 HMAC algorithms for signing the payload. The example above uses SHA-256 HMAC. If you want to use SHA-1, change x-hub-signature-256 into x-hub-signature, sha256 into sha1 in the VALIDATE clause.

Or you can also use raw string for verification without secret:

-- Example using SHA-1 and raw string 'TEST_WEBHOOK' to generate signature.
CREATE TABLE wbhtable (
  data JSONB
) WITH (
  connector = 'webhook'
) VALIDATE AS secure_compare(
  headers->>'x-hub-signature',
  'sha1=' || encode(hmac('TEST_WEBHOOK', data, 'sha1'), 'hex')
);

2. Set up webhook in GitHub

After configuring RisingWave to accept webhook data, set up GitHub to send events to your RisingWave instance.

RisingWave webhook URL

The webhook URL should follow this format: https://<HOST>/webhook/<database>/<schema_name>/<table_name>.

Parameter	Description
HOST	The hostname or IP address where your RisingWave instance is accessible. This could be a domain name or an IP address.
database	The name of the RisingWave database where your table resides
schema_name	The schema name of your table, typically public unless specified otherwise.
table_name	The name of the table you created to receive webhook data, e.g., wbhtable.

Configure webhook in GitHub

For more detailed instructions, refer to the GitHub documentation.

1. Go to your GitHub repository, and click on Settings tab.

2. In the left sidebar, click on webhooks > Add webhook.

3. Configure the webhook settings:

   • Payload URL: Enter your RisingWave webhook URL.
   • Content type: Select application/json.
   • Secret: Enter the same secret string you used when creating the RisingWave secret (e.g., 'TEST_WEBHOOK'). This ensures that GitHub signs the payloads using this secret, allowing RisingWave to validate them.
   • Which events would you like to trigger this webhook?: Choose the events you want to subscribe to. For testing purposes, you might start with Just the push event.
   • Active: Ensure the webhook is set to active.
4. Click Add webhook at the bottom of the page to save.

3. Push data from GitHub via webhook

With the webhook configured, GitHub will automatically send HTTP POST requests to your RisingWave webhook URL whenever the specified events occur (e.g., pushes to the repository). RisingWave will receive these requests, validate the signatures, and insert the payload data into the target table.

4. Further event processing

The data in the table is already ready for further processing. You can access the fields using data->'field_name' in SQL queries.

You can create a materialized view to extract specific fields from the JSON payload.

CREATE MATERIALIZED VIEW github_events AS
SELECT
  data->>'action' AS action,
  data->'repository'->>'full_name' AS repository_name,
  data->'sender'->>'login' AS sender_login,
  data->>'created_at' AS event_time
FROM wbhtable;

You can now query github_events like a regular table to perform analytics, generate reports, or trigger further processing.