Skip to content
Query.Farm
Talk with Us

COPY formats

On this page

Custom COPY … FROM readers and COPY … TO writers.

interface CopyFromCommenter

Section titled “interface CopyFromCommenter”
source
type CopyFromCommenter interface {
CopyFromComment() string
}

Description

CopyFromCommenter is an optional interface; when implemented, the returned comment is surfaced by vgi_copy_formats(). Mirrors COPY_FROM_COMMENT.

struct CopyFromContext

Section titled “struct CopyFromContext”
source
type CopyFromContext struct {
// Format is the FORMAT name resolved at COPY bind time.
Format  string
// FilePath is the source path from the COPY ... FROM 'path' statement.
FilePath  string
// ExpectedSchema is the COPY target's schema (column names + types, in target
// order). The reader must emit batches whose schema matches this exactly —
// DuckDB inserts no cast between the scan and the INSERT.
ExpectedSchema  *arrow.Schema
}

Description

CopyFromContext is the COPY … FROM context threaded onto a bind/init when a COPY-FROM scan is opened. Mirrors Python’s vgi.protocol.CopyFromContext.

Present (non-nil on BindParams/ProcessParams) only when the scan was opened by a COPY … FROM statement against a custom format. The COPY options arrive through the function’s normal arguments (params.Args), not here.

interface CopyFromFunction

Section titled “interface CopyFromFunction”
source
type CopyFromFunction interface {
// Name returns the handler's registered function name.
Name() string
// Metadata returns descriptive metadata (description/categories/tags).
Metadata() FunctionMetadata
// ArgumentSpecs returns the COPY option specifications.
ArgumentSpecs() []ArgSpec
// CopyFromFormat returns the SQL FORMAT identifier users type.
CopyFromFormat() string
// Read parses the source at path and emits Arrow batches via out whose
// schema matches expectedSchema exactly. out.Finish() is called by the
// framework after Read returns.
Read(ctx context.Context, params *ProcessParams, path string, expectedSchema *arrow.Schema, out *vgirpc.OutputCollector) error
}

Description

CopyFromFunction is the interface a worker implements to serve a custom COPY … FROM format. Mirrors vgi-python’s CopyFromFunction base class.

A CopyFromFunction is, mechanically, an ordinary producer-mode table function (RegisterCopyFrom wraps it as one so it reuses the whole table bind/init/scan path). What makes it a COPY format is that CopyFromFormat() returns the SQL FORMAT identifier and the worker advertises it via catalog_copy_from_formats.

  • Name() is the handler’s registered function name (also visible in
duckdb_functions like any table function).
  • CopyFromFormat() is the bare SQL FORMAT identifier (the VGI extension
scopes it by the attach alias, e.g. "acme.<format>").
  • The COPY options are declared via ArgumentSpecs() (the file_path is
supplied by COPY, never as an option) and read in Read via params.Args.
  • Read parses the source and emits Arrow batches matching expectedSchema.

interface CopyFromSecretProvider

Section titled “interface CopyFromSecretProvider”
source
type CopyFromSecretProvider interface {
// SecretLookups returns the secrets to resolve at bind, typically scoped by the
// source path (params.CopyFrom.FilePath). Returning nil/empty requests none.
SecretLookups(params *BindParams) []SecretLookup
}

Description

CopyFromSecretProvider is an optional interface a CopyFromFunction may implement to forward CREATE SECRET credentials for secret-backed cloud sources (S3/GCS/HTTP/…). Mirrors Python’s CopyFromFunction.on_secrets.

SecretLookups is the COPY-FROM secret-bind hook: it is called during bind (only on the first pass) and returns the secrets to resolve — typically scoped by the source path (params.CopyFrom.FilePath). The framework’s two-phase secret bind resolves each lookup from the caller’s SecretManager and surfaces the resolved values on params.Secrets at Read time. Returning nil/empty requests nothing.

interface CopyToCommenter

Section titled “interface CopyToCommenter”
source
type CopyToCommenter interface {
CopyToComment() string
}

Description

CopyToCommenter is an optional interface; when implemented, the returned comment is surfaced by vgi_copy_formats(). Mirrors COPY_TO_COMMENT.

struct CopyToContext

Section titled “struct CopyToContext”
source
type CopyToContext struct {
// Format is the FORMAT name resolved at COPY bind time.
Format  string
// FilePath is the destination path from the COPY ... TO 'path' statement.
FilePath  string
}

Description

CopyToContext is the COPY … TO context threaded onto a bind/init when a COPY-TO sink is opened. Mirrors Python’s vgi.protocol.CopyToContext.

Present (non-nil on BindParams/ProcessParams) only when the bind/init was opened by a COPY … TO statement against a custom format. The COPY options arrive through the function’s normal arguments (params.Args), not here; the source columns ride params.OutputSchema / the bind input schema.

interface CopyToFunction

Section titled “interface CopyToFunction”
source
type CopyToFunction interface {
// Name returns the handler's registered function name.
Name() string
// Metadata returns descriptive metadata (description/categories/tags). Set
// SinkOrderDependent=true to request a single-thread, source-ordered sink.
Metadata() FunctionMetadata
// ArgumentSpecs returns the COPY option specifications.
ArgumentSpecs() []ArgSpec
// CopyToFormat returns the SQL FORMAT identifier users type.
CopyToFormat() string
// Write persists one input batch to an execution-scoped shard (called once
// per sink batch). params.CopyTo carries the destination format + path.
Write(ctx context.Context, params *ProcessParams, batch arrow.RecordBatch) error
// Close reads every shard back and performs the terminal write + close of
// the destination, exactly once. Called even for an empty COPY (zero rows).
Close(ctx context.Context, params *ProcessParams) error
}

Description

CopyToFunction is the interface a worker implements to serve a custom COPY … TO format. Mirrors vgi-python’s CopyToFunction base class.

Mechanically a CopyToFunction is a buffered (Sink+Combine) function with NO Source phase: RegisterCopyTo wraps it as a TableBufferingFunction so it reuses the table_buffering_process / table_buffering_combine machinery on both sides.

  • Write is called once per input batch (the buffered process() step, fanned
out across DuckDB's sink threads / per-thread workers). Persist the batch
to an execution_id-scoped shard via params.Storage (cross-process safe).
  • Close is called exactly once on the coordinator worker (the buffered
combine() step, driven by DuckDB's once-only copy_to_finalize). Read the
shards back and perform the terminal write+flush+close of the destination.

There is no finalize/drain phase, so the destination MUST be fully written and closed inside Close — a writer that forgets leaves a silent partial file.

Cross-process invariant: Write and Close may run on different worker processes (pool rotation / HTTP). Any shard state Close needs MUST live in execution_id-scoped storage (params.Storage), not in per-instance fields.

  • Name() is the handler’s registered function name (the TableBufferingFunction
name; visible in duckdb_functions like any table-buffering function).
  • CopyToFormat() is the bare SQL FORMAT identifier (the VGI extension scopes
it by the attach alias, e.g. "acme.<format>").
  • The COPY options are declared via ArgumentSpecs() (the file_path is supplied
by COPY, never as an option) and read in Write/Close via params.Args.
  • To require source order, return Metadata with SinkOrderDependent=true;
RegisterCopyTo surfaces ordered=true, which the extension maps to a
single-thread sink.

interface CopyToSecretProvider

Section titled “interface CopyToSecretProvider”
source
type CopyToSecretProvider interface {
// SecretLookups returns the secrets to resolve at bind, typically scoped by the
// destination path (params.CopyTo.FilePath). Returning nil/empty requests none.
SecretLookups(params *BindParams) []SecretLookup
}

Description

CopyToSecretProvider is an optional interface a CopyToFunction may implement to forward CREATE SECRET credentials for secret-backed cloud writes (S3/GCS/HTTP/…). Mirrors Python’s CopyToFunction.on_secrets.

SecretLookups is the COPY-TO secret-bind hook: it is called during bind (only on the first pass, before any secrets are resolved) and returns the secrets to resolve — typically scoped by the destination path (params.CopyTo.FilePath). The framework’s two-phase secret bind resolves each lookup from the caller’s SecretManager and surfaces the resolved values on params.Secrets at Write/Close time. Returning nil/empty requests nothing, so a writer that never touched credentials is unaffected.

struct copyFromAdapter

Section titled “struct copyFromAdapter”
source
type copyFromAdapter struct {
inner CopyFromFunction
}

Description

copyFromAdapter wraps a CopyFromFunction as a TypedTableFunc so it reuses the table bind/init/scan machinery. Mirrors Python’s CopyFromFunction.on_bind / process.

Methods

method ArgumentSpecs

Section titled “method ArgumentSpecs”
source
func (a *copyFromAdapter) ArgumentSpecs() []ArgSpec

method Metadata

Section titled “method Metadata”
source
func (a *copyFromAdapter) Metadata() FunctionMetadata

method Name

Section titled “method Name”
source
func (a *copyFromAdapter) Name() string

method NewState

Section titled “method NewState”
source
func (a *copyFromAdapter) NewState(params *ProcessParams) (*copyFromState, error)

NewState allocates the single-shot read guard.

method OnBind

Section titled “method OnBind”
source
func (a *copyFromAdapter) OnBind(params *BindParams) (*BindResponse, error)

OnBind binds the output schema to the COPY target’s schema. DuckDB forces the scan’s output types to the target table’s columns, so a COPY-FROM reader must produce exactly the expected schema.

method Process

Section titled “method Process”
source
func (a *copyFromAdapter) Process(ctx context.Context, params *ProcessParams, state *copyFromState, out *vgirpc.OutputCollector) error

Process drives Read once, then finishes the stream.

struct copyFromFormatRecord

Section titled “struct copyFromFormatRecord”
source
type copyFromFormatRecord struct {
formatName  string
handler    string
comment    string  // "" = no comment (encoded as null)
direction  string
description  string
tags    map[string]string
argSpecs  []ArgSpec
// ordered marks a COPY ... TO writer that needs source order. The C++
// extension maps it to a single-thread sink. Always false for FROM formats.
ordered  bool
}

Description

copyFromFormatRecord is the worker-side record advertised via catalog_copy_from_formats. Mirrors CopyFromFormatInfo on the wire.

struct copyFromState

Section titled “struct copyFromState”
source
type copyFromState struct {
Done bool
}

Description

copyFromState is the single-shot read guard for the producer-mode adapter. Exported field so it gob-encodes for HTTP rehydration.

struct copyToAdapter

Section titled “struct copyToAdapter”
source
type copyToAdapter struct {
inner CopyToFunction
}

Description

copyToAdapter wraps a CopyToFunction as a TableBufferingFunction so it reuses the buffered process/combine machinery. Mirrors Python’s CopyToFunction process()/combine() (final methods over write()/close()).

Methods

method ArgumentSpecs

Section titled “method ArgumentSpecs”
source
func (a *copyToAdapter) ArgumentSpecs() []ArgSpec

method Combine

Section titled “method Combine”
source
func (a *copyToAdapter) Combine(ctx context.Context, params *ProcessParams, stateIDs [][]byte) ([][]byte, error)

Combine performs the terminal write (→ Close) once on the coordinator and returns an empty finalize list — the COPY-TO path never drains output.

method Finalize

Section titled “method Finalize”
source
func (a *copyToAdapter) Finalize(ctx context.Context, params *ProcessParams, finalizeStateID []byte) ([]arrow.RecordBatch, error)

Finalize is never invoked on the COPY-TO path (Combine returns no finalize ids). Present to satisfy the TableBufferingFunction interface.

method Metadata

Section titled “method Metadata”
source
func (a *copyToAdapter) Metadata() FunctionMetadata

method Name

Section titled “method Name”
source
func (a *copyToAdapter) Name() string

method OnBind

Section titled “method OnBind”
source
func (a *copyToAdapter) OnBind(params *BindParams) (*BindResponse, error)

OnBind: a sink produces no rows — bind to an empty output schema. Mirrors Python’s CopyToFunction.on_bind. If the writer implements CopyToSecretProvider, its requested secret lookups are forwarded on the first bind pass so the two-phase secret bind resolves them (the resolved values reach Write/Close via params.Secrets).

method Process

Section titled “method Process”
source
func (a *copyToAdapter) Process(ctx context.Context, params *ProcessParams, batch arrow.RecordBatch) ([]byte, error)

Process sinks one input batch (→ Write) and returns the execution_id bucket so all of a query’s batches land in one bucket, mirroring Python.

function SerializeCopyFromFormatInfo

Section titled “function SerializeCopyFromFormatInfo”
source
func SerializeCopyFromFormatInfo(rec copyFromFormatRecord) ([]byte, error)

SerializeCopyFromFormatInfo serializes one copy-from/copy-to format record to IPC bytes matching CopyFromFormatInfoSchema (comment, tags, format_name, handler, options, direction, description, ordered). The options field carries the IPC-serialized Arrow argument schema built from the handler’s ArgSpecs — the same encoding as FunctionInfo.arguments — so option type/default/doc surface identically to vgi_function_arguments().