Catalogs

On this page

Exposing schemas, tables, and views to ATTACH.

struct AttachCatalogInfo

source

type AttachCatalogInfo struct {
// Alias is the ATTACH alias; also the SourceCatalog a catalog-table
// ScanBranch references. Namespace it by your catalog identity so two workers
// don't both claim the same name (collisions are rejected, never merged).
Alias  string
// Target is the ATTACH target — a path or DSN, e.g.
// "ducklake:sqlite:/data/meta.sqlite" or "postgres:dbname=... host=...".
Target  string
// DBType is the DuckDB db type (e.g. "ducklake", "postgres"). Empty => the
// extension infers it from the Target scheme prefix.
DBType  string
// Options are extra ATTACH options forwarded verbatim (e.g. DuckLake DATA_PATH).
Options  map[string]string
// Hidden attaches the companion excluded from duckdb_databases() (still
// resolvable by qualified name and by branches).
Hidden  bool
// Required: when true, a failure to attach fails the whole VGI ATTACH; when
// false it is logged and skipped.
Required  bool
// SecretRef optionally names a credential to inject into the companion's
// ATTACH options (opt-in on the client via attach_companion_secrets).
SecretRef  string
}

Description

AttachCatalogInfo is a companion catalog the client should ATTACH when this VGI catalog attaches (lakehouse federation). It is IPC-serialized into CatalogAttachResult.attach_catalogs; the C++ VGI extension attaches each entry at VGI-attach time so multi-branch catalog-table branches (and direct queries) can resolve tables in a companion DuckLake / Iceberg / Postgres / DuckDB.

struct AttachOptionSpec

source

type AttachOptionSpec struct {
Name    string
Description  string
Type    arrow.DataType
DefaultBatch  arrow.RecordBatch
Required  bool
}

Description

AttachOptionSpec describes an ATTACH-time option the worker accepts. Mirrors vgi-python’s AttachOptionSpec: wire format is the same as SettingSpec so DuckDB can parse both with shared code.

DefaultBatch, when non-nil, must be a single-row RecordBatch whose only column (name “value”) has type Type. Use this to carry defaults for types like list/struct/decimal/date/time/timestamp that are awkward to express as Go scalars. Callers that only need primitive defaults can use BuildDefaultValueBatch.

Required marks an option the caller must supply at ATTACH time. A catalog that cannot be attached without it advertises that at discovery, so a client can say so before attempting the attach rather than surfacing a failure that reads like an empty catalog. It is mutually exclusive with a default: an option that falls back to a value is by definition satisfiable without the caller.

struct CatalogDataVersionRelease

source

type CatalogDataVersionRelease struct {
// Version is the concrete published version (e.g. "1.0.0"), not a spec.
Version  string
// ReleasedAt is the release date (UTC). Nil when the worker doesn't track
// dates.
ReleasedAt  *time.Time
// Summary is a one-line human summary; empty string when unknown.
Summary  string
// NotesURL optionally links to detailed notes for this release.
NotesURL  *string
}

Description

CatalogDataVersionRelease is one published data version of a catalog. It mirrors vgi-python’s CatalogDataVersionRelease. Entries on CatalogInfo.Releases must be newest-first and unique by Version.

struct CatalogExample

source

type CatalogExample struct {
SQL    string
Description  string
ExpectedOutput  *string
}

Description

CatalogExample is a usage example attached to a FunctionInfo.

struct CatalogInfo

source

type CatalogInfo struct {
Name      string
ImplementationVersion  *string
DataVersionSpec    *string
// AttachOptionSpecs holds pre-serialized AttachOptionSpec records (one per
// declared ATTACH-time option). Surfaced to DuckDB via vgi_catalogs() so
// the extension can validate ATTACH options before attach.
AttachOptionSpecs  [][]byte
// Releases lists the concrete published data versions, newest-first. Empty
// when the worker doesn't track release history.
Releases  []CatalogDataVersionRelease
// SourceURL points at where this worker's code lives (repo, build, docs).
// Nil when the worker doesn't advertise a source location.
SourceURL  *string
}

Description

CatalogInfo is the discovery record returned by catalog_catalogs.

struct CatalogMacro

source

type CatalogMacro struct {
// Name is the macro name visible in SQL.
Name  string
// MacroType is "scalar" or "table".
MacroType  MacroType
// Parameters lists the parameter names in order.
Parameters  []string
// ParameterDefaultValues is the serialized Arrow IPC bytes of a 1-row
// RecordBatch containing default values (nil when no defaults).
ParameterDefaultValues  []byte
// Definition is the SQL expression (scalar) or query (table).
Definition  string
// Comment is a human-readable description.
Comment  string
// Tags are arbitrary key/value annotations attached to this macro;
// surfaced through MacroInfo.tags.
Tags  map[string]string
// ParameterDocs is an optional mapping of parameter name to a
// human/agent-facing description. Keys must appear in Parameters.
// Descriptions flow over the wire via the macro arguments_schema's
// vgi_doc field metadata (the same channel functions use for per-argument
// docs). Empty/nil means no per-parameter docs.
ParameterDocs  map[string]string
}

Description

CatalogMacro describes a macro to register in the catalog.

struct CatalogTable

source

type CatalogTable struct {
// Name is the table name visible in SQL.
Name  string
// Comment is a human-readable description.
Comment  string
// Columns is the explicit column schema. If nil and Function is set,
// columns are derived from the function's OnBind response.
Columns  *arrow.Schema
// Function is the backing table function (nil for handler-only tables).
Function  TableFunction
// FuncArgs are the arguments to pass when calling the backing function.
FuncArgs  []CatalogTableArg
// NotNull lists column names with NOT NULL constraints.
NotNull  []string
// Unique lists groups of column names for UNIQUE constraints.
Unique  [][]string
// Check lists check constraint expressions.
Check  []string
// PrimaryKey lists groups of column names for PRIMARY KEY constraints.
PrimaryKey  [][]string
// ForeignKey lists foreign key definitions.
ForeignKey  []ForeignKeyConstraint
// Defaults maps column names to their default values.
// Supported types: Sql (raw SQL), string, int/int64/int32, float64/float32, bool, nil.
Defaults  map[string]any
// ColumnComments maps column names to per-column comments surfaced
// through duckdb_columns().comment.
ColumnComments  map[string]string
// Generated maps column names to SQL expressions for generated (virtual)
// columns. Encoded as `generated_expression` Arrow field metadata.
Generated  map[string]string
// Statistics holds optimizer hints per column name. When non-empty,
// the catalog reports supports_column_statistics=true for this table and
// answers catalog_table_column_statistics_get from this map.
Statistics  map[string]*ColumnStatistics
// StatisticsCacheMaxAgeSeconds, when set, is emitted as
// cache_max_age_seconds schema metadata on the stats batch.
// Nil means cache indefinitely; 0 means do not cache.
StatisticsCacheMaxAgeSeconds  *int64
// SupportsTimeTravel indicates this table supports AT (VERSION/TIMESTAMP) queries.
SupportsTimeTravel  bool
// CardinalityEstimate, when non-nil, inlines the table's row-count estimate
// on TableInfo. The C++ extension uses it directly and skips the per-bind
// table_function_cardinality RPC. Use only for read-only / slow-changing
// tables where cardinality is statically known.
CardinalityEstimate  *int64
// CardinalityMax mirrors CardinalityEstimate for the cardinality upper bound.
CardinalityMax  *int64
// Tags are arbitrary key/value annotations attached to this table.
// They surface to DuckDB on TableInfo.tags and are visible through
// duckdb_tables().tags. Useful for category/coverage/example_queries
// metadata.
Tags  map[string]string
// RequiredFilters is the required WHERE-filter set in conjunctive normal
// form (CNF): an AND (outer list) of OR-groups (inner lists) of dotted-path
// column references. A group is satisfied when any one of its member paths
// has a WHERE filter; every group must be satisfied. So
// [][]string{{"accession_number"}, {"ticker", "cik"}} means
// "accession_number AND one of (ticker, cik)"; a singleton group
// [][]string{{"country"}} is a plain mandatory filter. Paths are top-level
// names ("country") or struct subfields ("bbox.xmin", "nested.outer.inner").
// Empty (default) means no enforcement — the zero-cost fast path. The VGI
// DuckDB extension's optimizer pass enforces this at bind time and throws
// BinderException listing any unsatisfied groups; satisfaction is
// prefix-based (a filter on a parent path satisfies its child paths).
RequiredFilters  [][]string
// InlineBind, when true and the table's columns are statically known,
// inlines the bind result onto TableInfo.bind_result so the C++ extension
// skips the per-scan bind RPC. Safe only for tables whose output schema is
// static (the resolved Columns schema is authoritative for every scan).
InlineBind  bool
}

Description

CatalogTable describes a table to register in the catalog.

struct CatalogTableArg

source

type CatalogTableArg struct {
// Position is the 0-based positional index, or -1 for named arguments.
Position  int
// Name is the argument name (for named args).
Name  string
// Value is the Go value (int64, float64, string, bool, []byte).
Value  interface{}
// Type is the Arrow type for serialization.
Type  arrow.DataType
}

Description

CatalogTableArg describes a single argument for a function-backed table.

struct CatalogView

source

type CatalogView struct {
// Name is the view name visible in SQL.
Name  string
// Definition is the SQL query backing the view.
Definition  string
// Comment is a human-readable description.
Comment  string
// Tags are arbitrary key/value annotations attached to this view;
// surfaced through ViewInfo.tags.
Tags  map[string]string
// ColumnComments maps output column names to per-column comments. The
// C++ extension aligns these by name against the columns DuckDB binds
// from the view definition and surfaces them via duckdb_columns().comment.
ColumnComments  map[string]string
}

Description

CatalogView describes a view to register in the catalog.

struct ColumnStatistics

source

type ColumnStatistics struct {
ColumnName  string
Type    arrow.DataType  // Arrow type of Min/Max (required if either is set)
Min    interface{}  // e.g. int64(1), float64(0.99), "Accounting", nil
Max    interface{}
HasNull    bool
HasNotNull  bool
DistinctCount  int64  // 0 treated as unknown
// ContainsUnicode and MaxStringLength apply only to string/binary columns.
ContainsUnicode    *bool
MaxStringLength    *int64
distinctCountSet  bool  // zero-value disambiguator
}

Description

ColumnStatistics describes optimizer hints for one column. Match the Python ColumnStatistics dataclass in vgi-python/vgi/catalog/catalog_interface.py.

Min/Max are stored as Go scalars; SerializeColumnStatistics handles the Arrow sparse-union encoding the wire protocol expects.

Methods

method SetDistinctCount

source

func (c *ColumnStatistics) SetDistinctCount(n int64)

SetDistinctCount sets the distinct-count estimator (including 0 as a valid value).

struct DefaultReadOnlyCatalog

source

type DefaultReadOnlyCatalog struct {
catalogName    string
schemas      map[string]*catalogSchemaInfo
version      int64
attachOpaqueData  []byte
}

Description

DefaultReadOnlyCatalog auto-generates from registered functions.

Methods

function NewDefaultReadOnlyCatalog

source

func NewDefaultReadOnlyCatalog(catalogName string, w *Worker) *DefaultReadOnlyCatalog

NewDefaultReadOnlyCatalog creates a catalog from registered functions.

struct ForeignKeyConstraint

source

type ForeignKeyConstraint struct {
// Columns are the column names in this table.
Columns  []string
// ReferencedTable is the name of the referenced table.
ReferencedTable  string
// ReferencedColumns are the column names in the referenced table.
ReferencedColumns  []string
// ReferencedSchema is the schema of the referenced table (empty = same schema).
ReferencedSchema  string
}

Description

ForeignKeyConstraint describes a foreign key relationship.

struct FunctionInfo

source

type FunctionInfo struct {
Name        string
SchemaName      string
FunctionType      FunctionType
ArgSchema      *arrow.Schema  // argument schema
OutputSchema      *arrow.Schema  // return schema
Stability      FunctionStability
NullHandling      NullHandling
Description      string
Comment        string
Tags        map[string]string
Examples      []CatalogExample
Categories      []string
ProjectionPushdown    *bool
FilterPushdown      *bool
SamplingPushdown    *bool
LateMaterialization    *bool
SupportedExpressionFilters  []string
OrderPreservation    OrderPreservation  // "" = null
MaxWorkers      int32
SupportsBatchIndex    bool      // opt-in per-batch batch_index tagging
PartitionKind      PartitionKind    // default PartitionKindNotPartitioned
OrderDependent      OrderDependence    // default NOT_ORDER_DEPENDENT
DistinctDependent    DistinctDependence  // default NOT_DISTINCT_DEPENDENT
SupportsWindow      bool
StreamingPartitioned    bool
HasFinalize      bool
// Table-buffering (sink/source) ordering hints. Default false.
SourceOrderDependent  bool
SinkOrderDependent  bool
RequiresInputBatchIndex  bool
// InputFromArgs marks a blended ("UNNEST-style") table-in-out function: its
// positional args ARE the per-row input columns (real typed args, no TABLE
// placeholder), so one registration serves literal / column / LATERAL call
// shapes. The C++ extension reads it to enter the in-out registration branch
// with real-typed args and drive the literal single-row scan-mode.
InputFromArgs    bool
RequiredSettings  []string
RequiredSecrets    []SecretRequirement

// catalogHome is the single catalog that owns this registration. It lives
// on the entry rather than in a name-keyed map because two catalogs served
// by one worker may each declare their own implementation of one name.
catalogHome  string
// unlisted hides the entry from every function listing without changing its
// home (it exists to back a catalog table).
unlisted  bool
}

Description

FunctionInfo describes a function in the catalog.

struct MacroDefault

source

type MacroDefault struct {
Name  string
Value  interface{}
Type  arrow.DataType
}

Description

MacroDefault describes a single parameter default value.

struct MacroInfo

source

type MacroInfo struct {
Name      string
SchemaName    string
Comment      string
Tags      map[string]string
MacroType    MacroType
Parameters    []string
ParameterDefaultValues  []byte
Definition    string
// ArgumentsSchema is the optional macro arguments schema, serialized as
// Arrow IPC bytes: one nullable field per parameter, in Parameters order,
// each carrying its description via the vgi_doc field-metadata key (the
// same channel functions use). nil when no per-parameter docs are supplied.
ArgumentsSchema  []byte
}

Description

MacroInfo describes a macro in the catalog for wire serialization.

Methods

function macroInfoFromCatalogMacro

source

func macroInfoFromCatalogMacro(cm CatalogMacro, schemaName string) (*MacroInfo, error)

macroInfoFromCatalogMacro builds the wire MacroInfo for a registered CatalogMacro in the given schema, including the per-parameter arguments_schema (carrying vgi_doc field metadata for documented parameters).

type MacroType

source

type MacroType string

Description

MacroType identifies the kind of macro.

Methods

function macroKindFilter

source

func macroKindFilter(s string) MacroType

macroKindFilter maps a schema_contents_macros “type” filter value (as sent by DuckDB) to a MacroType. Returns “” for an unrecognized value.

struct MissingAttachOptionsError

source

type MissingAttachOptionsError struct {
CatalogName  string
Missing    []string
}

Description

MissingAttachOptionsError reports an ATTACH that omitted options declared Required. Missing carries the names so a caller can act on them without parsing the message. Mirrors Python’s MissingAttachOptionsError, message included — the extension’s integration suite matches on its text.

Methods

method Error

source

func (e *MissingAttachOptionsError) Error() string

Error renders the missing option names, matching the Python and Go implementations’ message text.

struct ScanArg

source

type ScanArg struct {
Value  interface{}
Type  arrow.DataType
}

Description

ScanArg is a single argument value with its Arrow type.

struct ScanBranch

source

type ScanBranch struct {
// FunctionName is the function to call (a VGI table function or any native
// DuckDB function such as read_parquet).
FunctionName  string
// PositionalArguments / NamedArguments are passed to the function's bind.
PositionalArguments  []ScanArg
NamedArguments    map[string]ScanArg
// BranchFilter, when non-nil, is a SQL expression ANDed into every scan of
// this branch before filter pushdown. Used to make overlapping physical
// sources non-overlapping at scan time.
BranchFilter  *string
// Writable declares this branch as the INSERT target. At most one branch
// per table may be writable (the C++ extension enforces this at parse time).
Writable  bool
// SourceCatalog/SourceSchema/SourceTable define a *catalog-table* branch:
// when FunctionName is "" and SourceTable is set, the branch scans the base
// table SourceCatalog.SourceSchema.SourceTable in a companion catalog
// (lakehouse federation) instead of calling a table function. Nil = function
// branch.
SourceCatalog  *string
SourceSchema  *string
SourceTable  *string
}

Description

ScanBranch is one physical source backing a multi-branch scan. It mirrors vgi-python’s ScanBranch.

struct ScanBranchesResult

source

type ScanBranchesResult struct {
Branches  []ScanBranch
// RequiredExtensions is the union of DuckDB extensions needed across all
// branches, hoisted to the top level.
RequiredExtensions  []string
}

Description

ScanBranchesResult is the list of physical sources backing a multi-branch table. The branches list must be non-empty. It mirrors vgi-python’s ScanBranchesResult.

func type ScanFunctionGetHandler

source

type ScanFunctionGetHandler func(schemaName, tableName string, atUnit, atValue *string) (*ScanFunctionResult, error)

Description

ScanFunctionGetHandler is a callback for resolving table scan functions that are not backed by a registered CatalogTable with a Function field. atUnit and atValue carry time-travel AT clause parameters (both nil when absent).

struct ScanFunctionResult

source

type ScanFunctionResult struct {
// FunctionName is the name of the function to invoke.
FunctionName  string
// PositionalArguments are the positional arguments.
PositionalArguments  []ScanArg
// NamedArguments are the named arguments.
NamedArguments  map[string]ScanArg
// RequiredExtensions lists DuckDB extensions that must be loaded.
RequiredExtensions  []string
}

Description

ScanFunctionResult describes the function to call when scanning a catalog table.

struct SchemaInfo

source

type SchemaInfo struct {
Name      string
Comment      string
Tags      map[string]string
AttachOpaqueData  []byte
// EstimatedObjectCount, when non-nil, advertises the approximate per-kind
// population (e.g. {"table": 0, "view": 12}). A value of 0 is a hard
// guarantee — the C++ client skips the corresponding bulk RPC and any
// per-name lookup for that kind. Nil disables the optimisation entirely.
EstimatedObjectCount  map[string]int64
}

Description

SchemaInfo describes a schema in the catalog.

type SerializedItems

source

type SerializedItems = [][]byte

Description

SerializedItems is a list of Arrow-IPC-encoded items sent over the wire.

type Sql

source

type Sql string

Description

Sql represents a raw SQL expression that should be passed through verbatim (e.g. current_timestamp, nextval(‘seq’)).

struct TableColumnStatisticsResult

source

type TableColumnStatisticsResult struct {
Statistics  []ColumnStatistics
// CacheMaxAgeSeconds: nil means cache indefinitely; 0 means no cache.
CacheMaxAgeSeconds  *int64
}

Description

TableColumnStatisticsResult is the full reply to catalog_table_column_statistics_get.

func type TableGetHandler

source

type TableGetHandler func(schemaName, tableName string, atUnit, atValue *string) ([]byte, error)

Description

TableGetHandler is a callback for customizing catalog_table_get responses, e.g. to return version-specific schemas for time-travel queries. Return nil to fall through to the default table lookup.

struct TableInfo

source

type TableInfo struct {
Name        string
SchemaName      string
Comment        string
Tags        map[string]string
Columns        *arrow.Schema  // serialized as IPC schema bytes
NotNullConstraints    []int32
UniqueConstraints    [][]int32
CheckConstraints    []string
PrimaryKeyConstraints    [][]int32
ForeignKeyConstraints    [][]byte  // each []byte is an IPC-serialized FK RecordBatch
SupportsInsert      bool
SupportsUpdate      bool
SupportsDelete      bool
SupportsReturning    bool
SupportsColumnStatistics  bool

// Optional inlined function-discovery results. When populated (non-nil),
// the C++ extension uses the cached bytes and skips the corresponding
// catalog_table_{scan,insert,update,delete}_function_get RPC. Bytes are
// the IPC payload from SerializeScanFunctionResult.
ScanFunction  []byte
InsertFunction  []byte
UpdateFunction  []byte
DeleteFunction  []byte

// Optional inlined cardinality. When set, the C++ extension uses these
// directly and skips the table_function_cardinality RPC. Use nil to leave
// the field unset (per-bind RPC continues to fire).
CardinalityEstimate  *int64
CardinalityMax    *int64

// Optional inlined column statistics. Bytes are the IPC payload from
// SerializeColumnStatistics. When non-nil, the C++ extension skips the
// per-bind catalog_table_column_statistics_get and the per-scan
// table_function_statistics RPCs.
ColumnStatistics  []byte

// Optional inlined bind result. Bytes are the IPC payload from a bind
// response. When non-nil, the C++ extension threads it straight into
// bind_data and skips the per-scan bind RPC.
BindResult  []byte

// RequiredFilters is the required WHERE-filter set in conjunctive normal
// form (CNF): an AND (outer list) of OR-groups (inner lists) of dotted-path
// column references that MUST appear in a WHERE expression for any scan of
// this table. A group is satisfied when any one of its member paths has a
// filter; every group must be satisfied. So [["accession_number"],
// ["ticker","cik"]] means "accession_number AND one of (ticker, cik)".
// Paths are top-level names ("country") or struct subfields ("bbox.xmin").
// Empty (default) means no enforcement. The C++ extension's optimizer pass
// consults this at bind time and throws BinderException listing any
// unsatisfied groups. Satisfaction is prefix-based — a filter on a parent
// path satisfies every required child path.
//
// This is the trailing field of the TableInfo wire schema.
RequiredFilters  [][]string
}

Description

TableInfo describes a table in the catalog for wire serialization.

Methods

function tableInfoFromWritable

source

func tableInfoFromWritable(t *writableTable, schemaName string) (*TableInfo, error)

struct ViewInfo

source

type ViewInfo struct {
Name    string
SchemaName  string
Comment    string
Tags    map[string]string
Definition  string
ColumnComments  map[string]string
}

Description

ViewInfo describes a view in the catalog for wire serialization.

struct WritableCatalog

source

type WritableCatalog struct {
// Name is the SQL-visible catalog name (must match ATTACH '<name>').
Name  string
// Comment is an optional human-readable description.
Comment  string

mu      sync.Mutex
attachOpaqueData  []byte
version      int64
// schemas keyed by schema name (lower-case canonical form).
schemas  map[string]*writableSchema
// store persists schemas/tables/rows across worker processes.
store  *writableStore
}

Description

WritableCatalog is a name-bound writable catalog hosted by a Worker. It owns a per-catalog set of schemas and tables that can be created, dropped, and mutated via DDL/DML over the VGI protocol.

Storage is currently in-memory; use a fresh worker process for an empty starting state. Cross-process state sharing (analogous to the SQLite-backed aggregate storage) is a follow-up.

Methods

method AttachOpaqueData

source

func (c *WritableCatalog) AttachOpaqueData() []byte

AttachOpaqueData returns the attach_opaque_data assigned to this catalog after first attach.

function NewWritableCatalog

source

func NewWritableCatalog(name string) *WritableCatalog

NewWritableCatalog builds an empty writable catalog with one default schema “main”. The catalog uses a SQLite-backed store so DuckDB-spawned worker subprocesses see the same state.

struct catalogSchemaInfo

source

type catalogSchemaInfo struct {
info    *SchemaInfo
functions  []FunctionInfo
tables    []CatalogTable
views    []CatalogView
macros    []CatalogMacro
}

type onConflictAction

source

type onConflictAction string

Description

onConflictAction encodes the SQL ON-CONFLICT semantics carried as a dictionary-encoded string by the C++ extension.

Methods

function parseOnConflict

source

func parseOnConflict(s string) onConflictAction

struct writableDeleteFn

source

type writableDeleteFn struct{ w *Worker }

Methods

method ArgumentSpecs

source

func (f *writableDeleteFn) ArgumentSpecs() []ArgSpec

ArgumentSpecs declares the constant schema_name and table_name arguments.

method Finalize

source

func (f *writableDeleteFn) Finalize(ctx context.Context, p *ProcessParams, state interface{}) ([]arrow.RecordBatch, error)

Finalize emits no additional batches; per-batch counts are returned inline.

method Metadata

source

func (f *writableDeleteFn) Metadata() FunctionMetadata

Metadata reports the function as a volatile, internal writable table function.

method Name

source

func (f *writableDeleteFn) Name() string

Name returns the registered function name for the writable delete function.

method NewState

source

func (f *writableDeleteFn) NewState(p *ProcessParams) (interface{}, error)

NewState creates the per-call state holding the target schema and table names.

method OnBind

source

func (f *writableDeleteFn) OnBind(p *BindParams) (*BindResponse, error)

OnBind binds the output to a single-column “rows_deleted” count schema.

method OnInit

source

func (f *writableDeleteFn) OnInit(p *InitParams) (*GlobalInitResponse, error)

OnInit limits processing to a single worker to serialize mutations.

method Process

source

func (f *writableDeleteFn) Process(ctx context.Context, p *ProcessParams, state interface{}, batch arrow.RecordBatch, out *vgirpc.OutputCollector) error

Process deletes rows identified by their synthesized row IDs in the incoming batch and emits the deleted row count.

struct writableInsertFn

source

type writableInsertFn struct{ w *Worker }

Methods

method ArgumentSpecs

source

func (f *writableInsertFn) ArgumentSpecs() []ArgSpec

ArgumentSpecs declares the constant schema_name and table_name arguments.

method Finalize

source

func (f *writableInsertFn) Finalize(ctx context.Context, p *ProcessParams, state interface{}) ([]arrow.RecordBatch, error)

Finalize emits no additional batches; per-batch counts are returned inline.

method Metadata

source

func (f *writableInsertFn) Metadata() FunctionMetadata

Metadata reports the function as a volatile, internal writable table function.

method Name

source

func (f *writableInsertFn) Name() string

Name returns the registered function name for the writable insert function.

method NewState

source

func (f *writableInsertFn) NewState(p *ProcessParams) (interface{}, error)

NewState creates the per-call state holding the target schema and table names.

method OnBind

source

func (f *writableInsertFn) OnBind(p *BindParams) (*BindResponse, error)

OnBind binds the output to a single-column “rows_inserted” count schema.

method OnInit

source

func (f *writableInsertFn) OnInit(p *InitParams) (*GlobalInitResponse, error)

OnInit limits processing to a single worker to serialize mutations.

method Process

source

func (f *writableInsertFn) Process(ctx context.Context, p *ProcessParams, state interface{}, batch arrow.RecordBatch, out *vgirpc.OutputCollector) error

Process appends the incoming batch rows to the table and emits the inserted row count.

struct writableMutateState

source

type writableMutateState struct {
SchemaName  string
TableName  string
Count    int64

// catalog memoizes the resolved writable catalog for the life of the
// mutation (one DML statement). findWritableTable does a locked SQLite
// QueryRow + schema-IPC deserialize + gob decode; without this cache every
// input batch would repeat it. Unexported so gob ignores it on the HTTP
// continuation round-trip (it re-resolves after rehydration).
catalog  *WritableCatalog
}

Methods

method resolveCatalog

source

func (st *writableMutateState) resolveCatalog(w *Worker) (*WritableCatalog, error)

resolveCatalog returns the writable catalog owning this mutation’s table, resolving it once and caching it on the state.

struct writableScanFn

source

type writableScanFn struct{ w *Worker }

Methods

method ArgumentSpecs

source

func (f *writableScanFn) ArgumentSpecs() []ArgSpec

ArgumentSpecs declares the constant schema_name and table_name arguments identifying the writable table to scan.

method Cardinality

source

func (f *writableScanFn) Cardinality(params *BindParams) (*TableCardinality, error)

Cardinality estimates the result size from the table’s stored row count, returning zero if the table cannot be found.

method Metadata

source

func (f *writableScanFn) Metadata() FunctionMetadata

Metadata reports the function as a volatile, internal writable table function with projection pushdown enabled.

method Name

source

func (f *writableScanFn) Name() string

Name returns the registered function name for the writable scan function.

method NewState

source

func (f *writableScanFn) NewState(params *ProcessParams) (*writableScanState, error)

NewState creates the per-scan state tracking whether rows have been emitted.

method OnBind

source

func (f *writableScanFn) OnBind(params *BindParams) (*BindResponse, error)

OnBind looks up the target table and binds the output to its schema plus a synthesized row-ID column.

method Process

source

func (f *writableScanFn) Process(ctx context.Context, params *ProcessParams, state *writableScanState, out *vgirpc.OutputCollector) error

Process emits all stored rows once as a single batch, honoring projection pushdown, then finishes the stream.

struct writableScanState

source

type writableScanState struct {
Emitted bool  // exported for gob round-trip via the framework
}

struct writableSchema

source

type writableSchema struct {
name  string
comment  string
tables  map[string]*writableTable
}

struct writableStore

source

type writableStore struct {
mu  sync.Mutex
db  *sql.DB
once  sync.Once
openErr  error
}

Description

writableStore is a SQLite-backed persistence layer that mirrors the in-memory writableSchema/writableTable/row data so DuckDB-spawned worker subprocesses see the same writable catalog state. Same rationale and same SQLite file as the aggregate state store.

Schema:

wc_schema(catalog, name PK, comment, created_version)
wc_table(catalog, schema_name, name PK, schema_ipc, meta_blob, comment)
wc_row(catalog, schema_name, table_name, row_id INTEGER PK auto, data_blob)

meta_blob is a gob-encoded writableTableMeta capturing not-null / PK / unique / check / FK / defaults / column comments — keeps the table row narrow.

Methods

method ensureOpen

source

func (s *writableStore) ensureOpen() error

function newWritableStore

source

func newWritableStore() *writableStore

method rowsAppend

source

func (s *writableStore) rowsAppend(catalog, schemaName, tableName string, rows []map[string]interface{}) (int64, error)

rowsAppend writes new rows, returning the next row_id base.

method rowsCount

source

func (s *writableStore) rowsCount(catalog, schemaName, tableName string) (int64, error)

rowsCount returns the number of rows for cardinality estimates.

method rowsDelete

source

func (s *writableStore) rowsDelete(catalog, schemaName, tableName string, rowIDs []int64) (int64, error)

rowsDelete removes rows by row_id.

method rowsScan

source

func (s *writableStore) rowsScan(catalog, schemaName, tableName string) ([]map[string]interface{}, error)

rowsScan returns all rows ordered by row_id, attaching the row_id under rowIDFieldName.

method rowsUpdate

source

func (s *writableStore) rowsUpdate(catalog, schemaName, tableName string, updates []map[string]interface{}) (int64, error)

rowsUpdate replaces specified columns in rows identified by row_id.

method schemaDrop

source

func (s *writableStore) schemaDrop(catalog, name string, cascade bool) error

method schemaExists

source

func (s *writableStore) schemaExists(catalog, name string) (bool, error)

method schemaList

source

func (s *writableStore) schemaList(catalog string) ([]struct{ Name, Comment string }, error)

method schemaUpsert

source

func (s *writableStore) schemaUpsert(catalog, name, comment string) error

schemaUpsert writes a schema record. Returns existing comment if found.

method tableDrop

source

func (s *writableStore) tableDrop(catalog, schemaName, tableName string) error

method tableList

source

func (s *writableStore) tableList(catalog, schemaName string) ([]string, error)

method tableLoad

source

func (s *writableStore) tableLoad(catalog, schemaName, tableName string) (*writableTable, error)

tableLoad fetches a table definition and rehydrates it (without rows).

method tableUpsert

source

func (s *writableStore) tableUpsert(catalog, schemaName string, t *writableTable) error

tableUpsert writes a table definition record.

struct writableTable

source

type writableTable struct {
name  string
schema  *arrow.Schema
comment  string
// rows is the in-memory row store; each row is a map of column name
// to Go value (nil for NULL).
rows  []map[string]interface{}
// constraints captured at CREATE TABLE time.
notNull    []string
primaryKey  [][]string
unique    [][]string
check    []string
foreignKey  []ForeignKeyConstraint
defaults  map[string]any
columnComment  map[string]string
}

struct writableTableMeta

source

type writableTableMeta struct {
NotNull    []string
PrimaryKey  [][]string
Unique    [][]string
Check    []string
ForeignKey  []ForeignKeyConstraint
Defaults  map[string][]byte  // gob-encoded defaultValue per column
ColumnComment  map[string]string
}

Description

writableTableMeta is the gob-serializable per-table metadata.

Methods

function decodeTableMeta

source

func decodeTableMeta(data []byte) (*writableTableMeta, error)

method toDefaults

source

func (m *writableTableMeta) toDefaults() map[string]any

struct writableUpdateFn

source

type writableUpdateFn struct{ w *Worker }

Methods

method ArgumentSpecs

source

func (f *writableUpdateFn) ArgumentSpecs() []ArgSpec

ArgumentSpecs declares the constant schema_name and table_name arguments.

method Finalize

source

func (f *writableUpdateFn) Finalize(ctx context.Context, p *ProcessParams, state interface{}) ([]arrow.RecordBatch, error)

Finalize emits no additional batches; per-batch counts are returned inline.

method Metadata

source

func (f *writableUpdateFn) Metadata() FunctionMetadata

Metadata reports the function as a volatile, internal writable table function.

method Name

source

func (f *writableUpdateFn) Name() string

Name returns the registered function name for the writable update function.

method NewState

source

func (f *writableUpdateFn) NewState(p *ProcessParams) (interface{}, error)

NewState creates the per-call state holding the target schema and table names.

method OnBind

source

func (f *writableUpdateFn) OnBind(p *BindParams) (*BindResponse, error)

OnBind binds the output to a single-column “rows_updated” count schema.

method OnInit

source

func (f *writableUpdateFn) OnInit(p *InitParams) (*GlobalInitResponse, error)

OnInit limits processing to a single worker to serialize mutations.

method Process

source

func (f *writableUpdateFn) Process(ctx context.Context, p *ProcessParams, state interface{}, batch arrow.RecordBatch, out *vgirpc.OutputCollector) error

Process applies the incoming batch rows as updates to the table and emits the updated row count.

function BuildMacroArgumentsSchema

source

func BuildMacroArgumentsSchema(parameters []string, parameterDefaultValues []byte, parameterDocs map[string]string) ([]byte, error)

BuildMacroArgumentsSchema builds the macro arguments_schema describing macro parameters, mirroring the function arguments_schema mechanism: one nullable Arrow field per parameter, in parameters order. Each parameter’s field type is the type of its default value when known (derived from parameterDefaultValues, the serialized 1-row RecordBatch of typed defaults) else arrow.Null. The per-parameter description rides as field metadata under the same vgi_doc key functions use (UTF-8, presence-only — the key is omitted entirely when there is no doc).

Returns nil when there are no parameters (nothing to describe). The returned bytes are Arrow IPC schema bytes suitable for MacroInfo.ArgumentsSchema and the macro create-request arguments_schema slot.

function BuildMacroDefaultValues

source

func BuildMacroDefaultValues(defaults []MacroDefault) ([]byte, error)

BuildMacroDefaultValues builds the serialized Arrow IPC bytes for macro parameter defaults. Returns a 1-row RecordBatch where column names are parameter names and values are typed defaults.

function MacroParameterDocsFromSchema

source

func MacroParameterDocsFromSchema(argumentsSchema []byte) (map[string]string, error)

MacroParameterDocsFromSchema extracts per-parameter descriptions from a macro arguments_schema (Arrow IPC schema bytes). Inverse of BuildMacroArgumentsSchema’s vgi_doc handling: reads the vgi_doc field metadata (UTF-8) for each field. Fields without the key (undocumented) are omitted from the result. Returns an empty map when the input is empty or carries no docs.

function SerializeAttachCatalogInfo

source

func SerializeAttachCatalogInfo(info AttachCatalogInfo) ([]byte, error)

SerializeAttachCatalogInfo serializes one AttachCatalogInfo to IPC bytes matching AttachCatalogInfoSchema (alias, target, db_type, options, hidden, required, secret_ref).

function SerializeCatalogInfo

source

func SerializeCatalogInfo(info *CatalogInfo) ([]byte, error)

SerializeCatalogInfo serializes a CatalogInfo to IPC bytes.

function SerializeColumnStatistics

source

func SerializeColumnStatistics(stats []ColumnStatistics, cacheMaxAgeSeconds *int64) ([]byte, error)

SerializeColumnStatistics encodes per-column stats as the sparse-union IPC batch DuckDB’s VGI extension expects. See vgi-python’s catalog_interface.serialize_column_statistics for the reference layout.

function SerializeFunctionInfo

source

func SerializeFunctionInfo(info *FunctionInfo) ([]byte, error)

SerializeFunctionInfo serializes a FunctionInfo to IPC bytes.

function SerializeMacroInfo

source

func SerializeMacroInfo(info *MacroInfo) ([]byte, error)

SerializeMacroInfo serializes a MacroInfo to IPC bytes.

function SerializeScanBranch

source

func SerializeScanBranch(branch *ScanBranch) ([]byte, error)

SerializeScanBranch serializes one ScanBranch to IPC bytes (the per-branch blob carried in ScanBranchesResult.branches).

function SerializeScanFunctionResult

source

func SerializeScanFunctionResult(result *ScanFunctionResult) ([]byte, error)

SerializeScanFunctionResult serializes a ScanFunctionResult to IPC bytes.

function SerializeSchemaInfo

source

func SerializeSchemaInfo(info *SchemaInfo) ([]byte, error)

SerializeSchemaInfo serializes a SchemaInfo to IPC bytes.

function SerializeTableInfo

source

func SerializeTableInfo(info *TableInfo) ([]byte, error)

SerializeTableInfo serializes a TableInfo to IPC bytes.

function SerializeViewInfo

source

func SerializeViewInfo(info *ViewInfo) ([]byte, error)

SerializeViewInfo serializes a ViewInfo to IPC bytes.

function appendValue

source

func appendValue(b array.Builder, val interface{})

appendValue appends a Go value to the appropriate Arrow builder.

function applyColumnComments

source

func applyColumnComments(schema *arrow.Schema, comments map[string]string) (*arrow.Schema, error)

applyColumnComments adds “comment” metadata to Arrow schema fields for columns with per-column descriptions. Existing metadata is preserved.

function applyDefaults

source

func applyDefaults(schema *arrow.Schema, defaults map[string]any) (*arrow.Schema, error)

applyDefaults adds “default” metadata to Arrow schema fields for columns that have default values defined. Existing field metadata is preserved.

function applyGenerated

source

func applyGenerated(schema *arrow.Schema, generated map[string]string) (*arrow.Schema, error)

applyGenerated adds “generated_expression” metadata to Arrow schema fields for generated/virtual columns. Existing field metadata is preserved.

function arrowValueAccessor

source

func arrowValueAccessor(col arrow.Array) func(i int) interface{}

arrowValueAccessor resolves the concrete column type once and returns a per-row extractor, hoisting the type switch out of per-row loops. Unsupported types yield an accessor that returns nil (mirrors arrowValueAt’s default).

function arrowValueAt

source

func arrowValueAt(col arrow.Array, i int) interface{}

arrowValueAt extracts one cell as a Go scalar. For per-cell loops over a whole column, resolve arrowValueAccessor once instead (this convenience form re-runs the type switch on every call).

function batchToRows

source

func batchToRows(batch arrow.RecordBatch) ([]map[string]interface{}, error)

batchToRows converts a RecordBatch into a slice of column-name → Go-value maps.

function buildColumnFromValues

source

func buildColumnFromValues(mem memory.Allocator, f arrow.Field, rows []map[string]interface{}) (arrow.Array, error)

function buildInt8Array

source

func buildInt8Array(mem memory.Allocator, data []int8) arrow.Array

function buildStatChildArray

source

func buildStatChildArray(mem memory.Allocator, t arrow.DataType, target int8, codes []int8, stats []ColumnStatistics, wantMin bool) (arrow.Array, error)

buildStatChildArray builds the per-type child array of a sparse union. Only rows whose typeCode matches the target are populated with the real value; others get null.

function catalogNameOf

source

func catalogNameOf(attachOpaqueData []byte) string

catalogNameOf returns the catalog name carried in an unwrapped attach plaintext: the bytes before the first NUL separator, or the whole value when there is none. Alias-info catalogs (WithCatalogAliasInfo) mint “<name>\x00<random>” so each ATTACH is unique; plain catalogs/aliases use the bare name. Catalog-scoped function visibility compares against this name.

function columnGroupsByIndex

source

func columnGroupsByIndex(schema *arrow.Schema, groups [][]int32) [][]string

function columnsByIndex

source

func columnsByIndex(schema *arrow.Schema, idx []int32) []string

function defaultToSQL

source

func defaultToSQL(value any) string

defaultToSQL converts a Go default value to a SQL expression string.

function defaultsFromSchemaMetadata

source

func defaultsFromSchemaMetadata(schema *arrow.Schema) map[string]any

defaultsFromSchemaMetadata extracts the default-value SQL expression stored as Arrow field metadata (key “default”) set by DuckDB’s column definition serializer.

function encodeTableMeta

source

func encodeTableMeta(t *writableTable) []byte

function init

source

func init()

function makeUnionFields

source

func makeUnionFields(types []arrow.DataType, names []string) []arrow.Field

function pickScalar

source

func pickScalar(s ColumnStatistics, wantMin bool) interface{}

function resolveColumnGroupIndices

source

func resolveColumnGroupIndices(columns *arrow.Schema, groups [][]string) [][]int32

resolveColumnGroupIndices maps groups of column names to groups of indices.

function resolveColumnIndices

source

func resolveColumnIndices(columns *arrow.Schema, names []string) []int32

resolveColumnIndices maps column names to their indices in the schema.

function rowsToBatch

source

func rowsToBatch(schema *arrow.Schema, rows []map[string]interface{}) (arrow.RecordBatch, error)

rowsToBatch builds a RecordBatch from a slice of column-name → value maps.

function serializeAttachOptionSpec

source

func serializeAttachOptionSpec(spec AttachOptionSpec) ([]byte, error)

serializeAttachOptionSpec serializes an AttachOptionSpec to Arrow IPC bytes.

function serializeForeignKey

source

func serializeForeignKey(schemaName string, fk *ForeignKeyConstraint) ([]byte, error)

serializeForeignKey serializes a ForeignKeyConstraint to IPC bytes.

function serializeInlineBindResult

source

func serializeInlineBindResult(schema *arrow.Schema) ([]byte, error)

serializeInlineBindResult produces the IPC bytes for an inlined bind result carrying a static output schema. It mirrors the payload a regular bind RPC would return (vgi-python’s BindResponse(output_schema=…).serialize_to_bytes()): the given schema in output_schema, null opaque_data, and empty secret-lookup lists. Set on TableInfo.BindResult so the C++ extension threads it straight into bind_data and skips the per-scan bind RPC.

function serializeScanArgs

source

func serializeScanArgs(mem memory.Allocator, positional []ScanArg, named map[string]ScanArg) ([]byte, error)

serializeScanArgs builds and serializes the nested arguments batch. Positional args are named arg_0, arg_1, …; named args use their name.

function suppliedAttachOptionNames

source

func suppliedAttachOptionNames(optionsIPC []byte) (map[string]struct{}, error)

suppliedAttachOptionNames reads the option keys out of the serialized options batch. Each supplied option is a column of a single-row batch, so the column names are the keys; nil or empty bytes mean none were supplied.

function toFloat64

source

func toFloat64(v interface{}) float64

function toInt64

source

func toInt64(v interface{}) int64

function toString

source

func toString(v interface{}) string

function validateRequiredAttachOptions

source

func validateRequiredAttachOptions(catalogName string, specs []AttachOptionSpec, optionsIPC []byte) error

validateRequiredAttachOptions returns a *MissingAttachOptionsError when a spec marked Required has no corresponding entry in the supplied options. Names are matched case-insensitively, mirroring DuckDB’s handling of ATTACH option keys.

function validateRequiredFilters

source

func validateRequiredFilters(tableName string, columns *arrow.Schema, groups [][]string) error

validateRequiredFilters checks a table’s CNF required-filter groups. Each OR-group must be non-empty and contain no empty strings, and the leading dotted segment of every path must name a real column on the table. Returns nil when groups is empty (the no-enforcement fast path) or columns is nil.

function withSynthesizedRowID

source

func withSynthesizedRowID(s *arrow.Schema) *arrow.Schema

withSynthesizedRowID returns a new schema with __row_id (int64) appended if not already present.

function writableArgs

source

func writableArgs(args *Arguments) (string, string)

function writableCountBatch

source

func writableCountBatch(name string, n int64) arrow.RecordBatch

function writableCountSchema

source

func writableCountSchema(name string) *arrow.Schema