Supported protobuf types

RisingWave supports a variety of protobuf data types, which are converted to equivalent types in RisingWave. This page provides an overview of the supported protobuf types and their corresponding RisingWave types.

Conversion

RisingWave converts well-known types from the protobuf library to specific types in RisingWave. The conversion is as follows:

Protobuf type	RisingWave type
any	jsonb
double	double precision
float	real
int32	int
int64	bigint
uint32	bigint
uint64	decimal
sint32	int
sint64	bigint
fixed32	bigint
fixed64	decimal
sfixed32	int
sfixed64	bigint
bool	boolean
string	varchar
bytes	bytea
enum	varchar
message	struct. See details in Nested messages.
messages_as_jsonb	jsonb. See details in Handle recursive definitions.
repeated	array
map	map. See details in Map.
google.protobuf.Struct	Not supported
google.protobuf.Timestamp	struct<seconds bigint, nanos int>
google.protobuf.Duration	struct<seconds bigint, nanos int>
google.protobuf.Any	struct<type_url varchar, value bytea>
google.protobuf.Int32Value	struct<value int>
google.protobuf.StringValue	struct<value varchar>

Nested messages

The nested fields are transformed into columns within a struct type. For example, a Protobuf message with the following structure:

Example

message NestedMessage {
  int32 id = 1;
  string name = 2;
}

Will be converted to struct<id int, name varchar> in RisingWave.

Handle recursive definitions

When detecting a recursive definition in the protobuf, RisingWave will reject the statement and show the circular dependency. Adding dependency items to messages_as_jsonb with full type name separated by comma can solve the case. For example:

CREATE TABLE opentelemetry_test 
WITH ( 
    ${RISEDEV_KAFKA_WITH_OPTIONS_COMMON}, 
    topic = 'opentelemetry_test' 
) 
FORMAT PLAIN 
ENCODE PROTOBUF ( 
    schema.registry = '${RISEDEV_SCHEMA_REGISTRY_URL}', 
    message = 'opentelemetry_test.OTLPTestMessage', 
    messages_as_jsonb = 'opentelemetry.proto.common.v1.ArrayValue,opentelemetry.proto.common.v1.KeyValueList,opentelemetry.proto.common.v1.AnyValue'
);

Field presence

RisingWave correctly handles protobuf field presence to distinguish between missing fields and fields explicitly set to their default values. The handling depends on the field type:

Regular primitive fields

Regular primitive fields (without the optional keyword) receive their default values when absent from the protobuf message:

Example

syntax = "proto3";
message Example {
  int32 regular_int = 1;      // Gets 0 if missing
  string regular_string = 2;  // Gets "" if missing
  bool regular_bool = 3;      // Gets false if missing
}

Optional fields

Fields marked with the optional keyword are set to NULL when absent:

Example

syntax = "proto3";
message Example {
  optional int32 optional_int = 1;      // Gets NULL if missing
  optional string optional_string = 2;  // Gets NULL if missing
}

Message fields

Message fields (nested messages) are set to NULL when absent:

Example

syntax = "proto3";
message InnerMessage {
  string content = 1;
}
message Example {
  InnerMessage inner = 1;  // Gets NULL if missing
}

Oneof fields

Fields within a oneof group are set to NULL when the oneof is not set or a different field in the group is set:

Example

syntax = "proto3";
message Example {
  oneof test_oneof {
    string oneof_string = 1;  // NULL if oneof_int is set or if oneof is not set
    int32 oneof_int = 2;      // NULL if oneof_string is set or if oneof is not set
  }
}

Only one field within a oneof can be non-NULL at a time.

Related topics

• CREATE SOURCE