Protocol

On this page

The wire shapes the worker and engine exchange.

record AggregateBindRequest

source

public record AggregateBindRequest( String function_name, byte[] arguments, byte[] input_schema, byte[] settings, byte[] secrets, byte[] attach_opaque_data, @Nullable String schema_name) implements ArrowSerializableRecord

Description

Wire DTO for the aggregate_bind request, opening an aggregate execution.

record AggregateBindResponse

source

public record AggregateBindResponse( byte[] output_schema, byte[] execution_id) implements ArrowSerializableRecord

Description

Wire DTO for the aggregate_bind response.

record AggregateCombineRequest

source

public record AggregateCombineRequest( String function_name, byte[] execution_id, byte[] merge_batch, byte[] attach_opaque_data, @Nullable String schema_name) implements ArrowSerializableRecord

Description

Wire DTO for the aggregate_combine request, merging partial aggregate states.

record AggregateCombineResponse

source

public record AggregateCombineResponse() implements ArrowSerializableRecord

Description

Empty acknowledgement for the aggregate_combine request.

record AggregateDestructorRequest

source

public record AggregateDestructorRequest( String function_name, byte[] execution_id, byte[] group_ids_batch, byte[] attach_opaque_data, @Nullable String schema_name) implements ArrowSerializableRecord

Description

Wire DTO for the aggregate_destructor request, releasing per-group aggregate state.

record AggregateDestructorResponse

source

public record AggregateDestructorResponse() implements ArrowSerializableRecord

Description

Empty acknowledgement for the aggregate_destructor request.

record AggregateFinalizeRequest

source

public record AggregateFinalizeRequest( String function_name, byte[] execution_id, byte[] group_ids_batch, byte[] output_schema, byte[] attach_opaque_data, @Nullable String schema_name) implements ArrowSerializableRecord

Description

Wire DTO for the aggregate_finalize request, producing per-group result rows.

record AggregateFinalizeResponse

source

public record AggregateFinalizeResponse(byte[] result_batch) implements ArrowSerializableRecord

Description

Wire DTO for the aggregate_finalize response.

record AggregateUpdateRequest

source

public record AggregateUpdateRequest( String function_name, byte[] execution_id, byte[] input_batch, byte[] attach_opaque_data, @Nullable String schema_name) implements ArrowSerializableRecord

Description

Wire DTO for the aggregate_update request, feeding a batch of input rows into groups.

record AggregateUpdateResponse

source

public record AggregateUpdateResponse() implements ArrowSerializableRecord

Description

Empty ack for aggregate_update.

record AttachCatalogInfo

source

public record AttachCatalogInfo( String alias, String target, String db_type, Map<String, String> options, boolean hidden, boolean required, String secret_ref) implements ArrowSerializableRecord

Description

A companion catalog the client should ATTACH when this VGI catalog attaches (lakehouse federation). Serialised as an item in CatalogAttachResult.attach_catalogs; mirrors vgi-python’s AttachCatalogInfo. The C++ VGI extension attaches each entry at VGI-attach time so multi-branch catalog-table branches can resolve tables in a companion DuckLake / Iceberg / Postgres / DuckDB.

Field order, types and nullability are part of the wire contract; the Arrow schema is derived reflectively from the record components.

record BindRequest

source

public record BindRequest( String function_name, byte[] arguments, String function_type, byte[] input_schema, byte[] settings, byte[] secrets, byte[] attach_opaque_data, byte[] transaction_opaque_data, boolean resolved_secrets_provided, @Nullable String at_unit, @Nullable String at_value, @Nullable CopyFromContext copy_from, @Nullable CopyToContext copy_to, @Nullable String schema_name) implements ArrowSerializableRecord

Description

Wire DTO for the VGI bind request, opening a table or function binding.

record BindResponse

source

public record BindResponse( byte[] output_schema, byte[] opaque_data, List<String> lookup_secret_types, List<String> lookup_scopes, List<String> lookup_names) implements ArrowSerializableRecord

Description

Wire DTO for VGI bind responses.

Mirrors vgi.BindResponseWire in vgi-go.

Members

BindResponse forSchema(byte[] outputSchema)

Builds a response carrying only an output schema, with no opaque data and no secret lookup.

record CardinalityRequest

source

public record CardinalityRequest( byte[] bind_call, byte[] bind_opaque_data) implements ArrowSerializableRecord

Description

Wire DTO for the table_function_cardinality request.

record CardinalityResponse

source

public record CardinalityResponse( @Nullable Long estimate, @Nullable Long max) implements ArrowSerializableRecord

Description

Wire shape for the table_function_cardinality RPC reply.

Both fields are nullable on the wire; null means “unknown” — DuckDB falls back to its own default.

record CatalogAttachRequest

source

public record CatalogAttachRequest( String name, byte[] options, String data_version_spec, String implementation_version) implements ArrowSerializableRecord

Description

Wire DTO for the catalog_attach request, opening a catalog attachment.

record CatalogAttachResult

source

public record CatalogAttachResult( byte[] attach_opaque_data, boolean supports_transactions, boolean supports_time_travel, boolean catalog_version_frozen, long catalog_version, boolean attach_opaque_data_required, String default_schema, List<byte[]> settings, List<byte[]> secret_types, List<byte[]> attach_catalogs, @Nullable String comment, Map<String, String> tags, boolean supports_column_statistics, List<byte[]> global_functions, String global_function_prefix, @Nullable String resolved_data_version, @Nullable String resolved_implementation_version) implements ArrowSerializableRecord

Description

Wire DTO for VGI catalog_attach responses. Mirrors vgi.CatalogAttachResultWire in vgi-go.

record CatalogVersionResponse

source

public record CatalogVersionResponse(long version) implements ArrowSerializableRecord

Description

Wire DTO for the catalog_version response.

record CopyFromContext

source

public record CopyFromContext( String format, String file_path, byte[] expected_schema) implements ArrowSerializableRecord

Description

Context for a COPY ... FROM read, threaded onto BindRequest.

Present only when the bind/init opens a COPY-FROM scan (null otherwise — set by the VGI extension’s copy_from_bind). The C++ extension serialises this as a nested, name-keyed Arrow struct field on the bind request, so it is matched by component name and defaults to null when the field is absent (ordinary scans). Mirrors vgi-python’s vgi.protocol.CopyFromContext.

The handler’s COPY options arrive through the normal BindRequest#arguments() (built from the COPY options), so they are not duplicated here.

record CopyFromFormatInfo

source

public record CopyFromFormatInfo( @Nullable String comment, Map<String, String> tags, String format_name, String handler, byte[] options, String direction, String description, boolean ordered) implements ArrowSerializableRecord

Description

A custom COPY ... FROM format advertised by a VGI catalog. Serialised as an item in catalog_copy_from_formats; mirrors the C++ CopyFromFormatInfoSchema and vgi-python’s CopyFromFormatInfo.

Field order, types and nullability are part of the wire contract (see farm.query.vgi.internal.CopyFromFormatInfoSerializer).

record CopyToContext

source

public record CopyToContext( String format, String file_path) implements ArrowSerializableRecord

Description

Context for a COPY ... TO write, threaded onto BindRequest.

Present only when the bind/init opens a COPY-TO sink (null otherwise — set by the VGI extension’s copy_to_bind). The C++ extension serialises this as a nested, name-keyed Arrow struct field on the bind request, so it is matched by component name and defaults to null when the field is absent (ordinary scans/sinks). Mirrors vgi-python’s vgi.protocol.CopyToContext.

The handler’s COPY options arrive through the normal BindRequest#arguments() (built from the COPY options); the source columns ride the existing BindRequest#input_schema(), so neither is duplicated here.

record DynamicToStringResponse

source

public record DynamicToStringResponse( List<String> keys, List<String> values) implements ArrowSerializableRecord

Description

Response shape for table_function_dynamic_to_string: two parallel string lists representing keys[i] -> values[i].

DuckDB renders these as Extra Info under EXPLAIN ANALYZE.

record FunctionExample

source

public record FunctionExample( String sql, String description, @Nullable String expected_output) implements ArrowSerializableRecord

Description

A documented usage example for a function, embedded in FunctionInfo#examples.

record FunctionInfo

source

public record FunctionInfo( @Nullable String comment, Map<String, String> tags, String name, String schema_name, @ArrowField(ArrowFieldType.DICT_INT16_UTF8) String function_type, byte[] arguments, byte[] output_schema, @ArrowField(ArrowFieldType.DICT_INT16_UTF8) @Nullable String stability, @ArrowField(ArrowFieldType.DICT_INT16_UTF8) @Nullable String null_handling, String description, List<FunctionExample> examples, List<String> categories, @Nullable Boolean projection_pushdown, @Nullable Boolean filter_pushdown, @Nullable Boolean sampling_pushdown, @Nullable Boolean late_materialization, List<String> supported_expression_filters, @ArrowField(ArrowFieldType.DICT_INT16_UTF8) @Nullable String order_preservation, @ArrowField(ArrowFieldType.INT32) @Nullable Integer max_workers, boolean supports_batch_index, @ArrowField(ArrowFieldType.DICT_INT16_UTF8) String partition_kind, @ArrowField(ArrowFieldType.DICT_INT16_UTF8) String order_dependent, @ArrowField(ArrowFieldType.DICT_INT16_UTF8) String distinct_dependent, boolean supports_window, boolean streaming_partitioned, boolean has_finalize, boolean source_order_dependent, boolean sink_order_dependent, boolean requires_input_batch_index, boolean input_from_args, List<String> required_settings, List<FunctionRequiredSecret> required_secrets) implements ArrowSerializableRecord

Description

Mirrors the C++ FunctionInfoSchema. Serialised as an item in catalog_schema_contents_functions.

Field order, types, and nullability are part of the wire contract.

record FunctionRequiredSecret

source

public record FunctionRequiredSecret( String secret_type, @Nullable String scope, @Nullable String secret_name) implements ArrowSerializableRecord

Description

A secret requirement declared by a function, embedded in FunctionInfo#required_secrets.

record GlobalInitResponse

source

public record GlobalInitResponse( byte[] execution_id, long max_workers, byte[] opaque_data) implements ArrowSerializableRecord

Description

Wire DTO for VGI init headers. Sent as the first batch of an init stream. Mirrors vgi.GlobalInitResponseWire in vgi-go.

Members

GlobalInitResponse of(byte[] executionId)

Builds a single-worker response with no opaque data.

record InitRequest

source

public record InitRequest( byte[] bind_call, byte[] output_schema, byte[] bind_opaque_data, List<Integer> projection_ids, byte[] pushdown_filters, List<byte[]> join_keys, String phase, byte[] execution_id, byte[] init_opaque_data, String order_by_column_name, String order_by_direction, String order_by_null_order, Long order_by_limit, Double tablesample_percentage, Long tablesample_seed, byte[] finalize_state_id, byte[] substream_id) implements ArrowSerializableRecord

Description

Wire DTO for the init request that binds and configures a function execution before streaming begins.

The phase field selects the execution mode (e.g. plain table, table-in-out, table buffering sink, or table buffering finalize/source), and the remaining fields carry the pushdown, ordering, sampling, and state parameters the C++ extension negotiated at bind time.

record ItemsResponse

source

public record ItemsResponse(List<byte[]> items) implements ArrowSerializableRecord

Description

Generic items list wire response. Used by every catalog query that returns zero or more serialised info records (schemas, tables, views, functions, etc.).

Mirrors vgi.ItemsResponseWire / vgi.CatalogsResponseWire.

Members

ItemsResponse empty()

Returns an items response carrying no items.

record MacroInfo

source

public record MacroInfo( @Nullable String comment, Map<String, String> tags, String name, String schema_name, @ArrowField(ArrowFieldType.DICT_INT16_UTF8) String macro_type, List<String> parameters, @Nullable byte[] parameter_default_values, String definition, @Nullable byte[] arguments_schema) implements ArrowSerializableRecord

Description

Mirrors the C++ MacroInfoSchema.

macro_type is a dictionary-encoded enum on the wire ("scalar" or "table"); parameter_default_values carries an IPC-encoded 1-row record batch holding default values for any named parameters that omit a positional default.

record SchemaInfo

source

public record SchemaInfo( @Nullable String comment, Map<String, String> tags, byte[] attach_opaque_data, String name, @Nullable Map<String, Long> estimated_object_count) implements ArrowSerializableRecord

Description

Mirrors the C++ SchemaInfoSchema.

estimated_object_count is consulted by the C++ extension to gate eager catalog loads — entries with count 0 for a kind (tables / views / macros / functions / indexes) skip the corresponding catalog_schema_contents_* RPC entirely, so workers with no objects of a kind avoid round-trips.

Members

SchemaInfo(String comment, Map<String, String> tags, byte[] attach_opaque_data, String name)

Convenience constructor with no estimated_object_count map (eager-load gating disabled).

record TableBufferingCombineRequest

source

public record TableBufferingCombineRequest( String function_name, byte[] execution_id, List<byte[]> state_ids, byte[] attach_opaque_data, byte[] transaction_id, @Nullable String schema_name) implements ArrowSerializableRecord

Description

Combine-phase request for table_buffering_combine — once per query after all input. Inner field order matches C++ BuildTableBufferingCombineInner.

record TableBufferingCombineResponse

source

public record TableBufferingCombineResponse( List<byte[]> finalize_state_ids) implements ArrowSerializableRecord

Description

Response from table_buffering_combine — the finalize partition keys.

record TableBufferingDestructorRequest

source

public record TableBufferingDestructorRequest( String function_name, byte[] execution_id, byte[] attach_opaque_data, byte[] transaction_id, @Nullable String schema_name) implements ArrowSerializableRecord

Description

Best-effort cleanup request for table_buffering_destructor.

record TableBufferingDestructorResponse

source

public record TableBufferingDestructorResponse() implements ArrowSerializableRecord

Description

Empty ack for table_buffering_destructor.

record TableBufferingProcessRequest

source

public record TableBufferingProcessRequest( String function_name, byte[] execution_id, byte[] input_batch, byte[] attach_opaque_data, byte[] transaction_id, Long batch_index, @Nullable String schema_name) implements ArrowSerializableRecord

Description

Sink-phase request for table_buffering_process — one input batch. Inner field order/types match the C++ BuildTableBufferingProcessInner.

record TableBufferingProcessResponse

source

public record TableBufferingProcessResponse(byte[] state_id) implements ArrowSerializableRecord

Description

Response from table_buffering_process — the worker-chosen state_id.

record TableInfo

source

public record TableInfo( @Nullable String comment, Map<String, String> tags, String name, String schema_name, byte[] columns, List<Integer> not_null_constraints, List<List<Integer>> unique_constraints, List<String> check_constraints, List<List<Integer>> primary_key_constraints, List<byte[]> foreign_key_constraints, boolean supports_insert, boolean supports_update, boolean supports_delete, boolean supports_returning, boolean supports_column_statistics, @Nullable byte[] scan_function, @Nullable byte[] insert_function, @Nullable byte[] update_function, @Nullable byte[] delete_function, @Nullable Long cardinality_estimate, @Nullable Long cardinality_max, @Nullable byte[] column_statistics, @Nullable byte[] bind_result, List<List<String>> required_filters)

Description

Mirrors the C++ TableInfoSchema. Hand-rolled by farm.query.vgi.internal.TableInfoSerializer because the schema has nested list<list<int32>> constraint shapes plus several optional binary “inline” fields the stock RecordCodec can’t currently emit with the right nullability.

record TableScanFunctionGetResponse

source

public record TableScanFunctionGetResponse( String function_name, byte[] arguments, List<String> required_extensions) implements ArrowSerializableRecord

Description

Response describing the table function used to scan a catalog table.

record TransactionBeginResponse

source

public record TransactionBeginResponse( @Nullable byte[] transaction_opaque_data) implements ArrowSerializableRecord

Description

Response from catalog_transaction_begin.

record ViewInfo

source

public record ViewInfo( @Nullable String comment, Map<String, String> tags, String name, String schema_name, String definition, Map<String, String> column_comments) implements ArrowSerializableRecord

Description

Mirrors the C++ ViewInfoSchema.