COPY formats

On this page

Reading and writing your own format through COPY … FROM / TO.

interface CopyFromFunctionConfig

source

export interface CopyFromFunctionConfig<TArgs = Record<string, unknown>>

Fields

namestring

Handler name (the function’s registered name; Meta.name).

formatstring

SQL FORMAT identifier users type, e.g. COPY t FROM 'x' (FORMAT myfmt).

read(params: CopyFromReadParams<TArgs>) => void | Promise<void>

Parse path and emit Arrow batches matching expectedSchema.

descriptionstringoptional

Function description (intrinsic documentation; Meta.description).

commentstring | nulloptional

Optional free-text comment surfaced by vgi_copy_formats().

directionstringoptional

Reserved for a future COPY ... TO; only "from" is supported today.

optionsRecord<string, CopyFromOption>optional

COPY options, keyed by option name. The file_path is NOT an option.

categoriesstring[]optional

tagsRecord<string, string>optional

examplesFunctionExample[]optional

requiredSettingsstring[]optional

requiredSecretsstring[]optional

onSecrets(params: { options: TArgs; path: string; bindCall: BindRequest; }) => CopySecretLookup[] | voidoptional

Optional secret-bind hook: forward CREATE SECRET credentials for secret-backed cloud sources (S3/GCS/HTTP/…). Called during bind (only on the first pass); return the secrets to resolve — typically scoped by the source path. The framework’s two-phase secret bind resolves each lookup from the caller’s SecretManager and surfaces the resolved values on processParams.secrets at read time. Mirrors vgi-python’s CopyFromFunction.on_secrets.

interface CopyFromOption

source

export interface CopyFromOption

Description

Declaration of a single COPY option (a named function argument).

Fields

typeVgiDataType

Arrow type of the option value (e.g. utf8(), int64()).

docstringoptional

Per-option description, surfaced by vgi_copy_formats().

defaultunknownoptional

Default value. When omitted the option is REQUIRED — the worker throws a clear error at COPY bind if the user does not supply it. Mirrors vgi-python’s Arg(..., default=...) (no default => required).

choicesunknown[]optional

Optional allowed value set, validated worker-side at bind. Mirrors vgi-python’s Arg(..., choices=[...]).

interface CopyFromReadParams

source

export interface CopyFromReadParams<TArgs = Record<string, unknown>>

Fields

pathstring

Source path from the COPY ... FROM 'path' statement.

optionsTArgs

Parsed COPY options (defaults applied).

expectedSchemaVgiSchema

The COPY target’s schema. Every emitted batch must have this exact schema (names + types, in order) — DuckDB inserts no cast before the INSERT.

processParamsTableProcessParams<TArgs>

Full process parameters (settings, secrets, storage).

outOutputCollector

Collector to emit batches / log. finish() is called for you.

interface CopyToCloseParams

source

export interface CopyToCloseParams<TArgs = Record<string, unknown>>

Description

Parameters for the terminal close hook.

Fields

optionsTArgs

Parsed COPY options (defaults applied).

filePathstring

Destination path from the COPY ... TO 'path' statement.

paramsTableBufferingParams<TArgs>

Full buffering parameters (settings, secrets, storage, executionId).

interface CopyToFunctionConfig

source

export interface CopyToFunctionConfig<TArgs = Record<string, unknown>>

Fields

namestring

Handler name (the function’s registered name; Meta.name).

formatstring

SQL FORMAT identifier users type, e.g. COPY t TO 'x' (FORMAT myfmt).

write(params: CopyToWriteParams<TArgs>) => void | Promise<void>

Persist one input batch to a shard (called per sink batch).

close(params: CopyToCloseParams<TArgs>) => number | void | Promise<number | void>

Write the destination and close it, once (called on the coordinator). Read the shards persisted by write and perform the terminal write. Called even for empty input. Return the row count (informational).

descriptionstringoptional

Function description (intrinsic documentation; Meta.description).

commentstring | nulloptional

Optional free-text comment surfaced by vgi_copy_formats().

directionstringoptional

COPY direction; only "to" is supported here.

orderedbooleanoptional

When true, the writer requires rows in source order — discovery advertises ordered=true and the extension uses a single-threaded sink (REGULAR_COPY_TO_FILE) so one worker receives every batch in source order. Mirrors vgi-python’s Meta.sink_order_dependent.

optionsRecord<string, CopyToOption>optional

COPY options, keyed by option name. The file_path is NOT an option.

categoriesstring[]optional

tagsRecord<string, string>optional

examplesFunctionExample[]optional

requiredSettingsstring[]optional

requiredSecretsstring[]optional

onSecrets(params: { options: TArgs; filePath: string; bindCall: BindRequest; }) => CopySecretLookup[] | voidoptional

Optional secret-bind hook: forward CREATE SECRET credentials for secret-backed cloud writes (S3/GCS/HTTP/…). Called during bind (only on the first pass); return the secrets to resolve — typically scoped by the destination filePath. The framework’s two-phase secret bind resolves each lookup from the caller’s SecretManager and surfaces the resolved values on params.secrets at write/close time. Mirrors vgi-python’s CopyToFunction.on_secrets.

interface CopyToOption

source

export interface CopyToOption

Description

Declaration of a single COPY option (a named function argument).

Fields

typeVgiDataType

Arrow type of the option value (e.g. utf8(), int64()).

docstringoptional

Per-option description, surfaced by vgi_copy_formats().

defaultunknownoptional

Default value. When omitted the option is REQUIRED — the worker throws a clear error at COPY bind if the user does not supply it. Mirrors vgi-python’s Arg(..., default=...) (no default => required).

choicesunknown[]optional

Optional allowed value set, validated worker-side at bind. Mirrors vgi-python’s Arg(..., choices=[...]).

genumberoptional

Optional inclusive lower bound for numeric options, validated worker-side at bind. Mirrors vgi-python’s Arg(..., ge=...).

lenumberoptional

Optional inclusive upper bound for numeric options, validated worker-side at bind. Mirrors vgi-python’s Arg(..., le=...).

interface CopyToWriteParams

source

export interface CopyToWriteParams<TArgs = Record<string, unknown>>

Description

Parameters for the per-batch write hook.

Fields

batchVgiBatch

One input batch from the COPY source.

optionsTArgs

Parsed COPY options (defaults applied).

filePathstring

Destination path from the COPY ... TO 'path' statement.

paramsTableBufferingParams<TArgs>

Full buffering parameters (settings, secrets, storage, executionId).

function defineCopyFromFunction

source

export function defineCopyFromFunction<TArgs = Record<string, unknown>>(
config: CopyFromFunctionConfig<TArgs>,
): VgiFunction

Description

Define a custom COPY ... FROM format reader, returned as a VgiFunction (kind "table") carrying the copyFromFormat metadata marker. Register it in the catalog’s function list like any table function; the catalog’s copyFromFormats() introspection picks it up and advertises the format.

function defineCopyToFunction

source

export function defineCopyToFunction<TArgs = Record<string, unknown>>(
config: CopyToFunctionConfig<TArgs>,
): TableBufferingVgiFunction

Description

Define a custom COPY ... TO format writer, returned as a TableBufferingVgiFunction (kind "table_buffering") carrying the copyToFormat metadata marker. Register it in the catalog’s function list like any function; the catalog’s copyFromFormats() introspection picks it up and advertises the format (direction "to"). The worker’s table_buffering_process / table_buffering_combine RPCs drive write/close.