Skip to content
Query.Farm
Talk with Us

vgi.copy_from_function

Module overview

Base class for custom COPY ... FROM format readers.

A :class:CopyFromFunction lets a VGI catalog act as a remote file-format reader: the user runs COPY target FROM 'path' (FORMAT <name>, opt val, ...) and the worker parses the source and streams Arrow batches that DuckDB inserts into the local target table.

Mechanically a CopyFromFunction is an ordinary producer-mode table function (so it reuses the entire table-function bind/init/scan path on both sides). What makes it a COPY format is twofold:

  • it sets :attr:CopyFromFunction.COPY_FROM_FORMAT to the SQL FORMAT identifier, and
  • the catalog advertises it via :meth:vgi.catalog.catalog_interface.ReadOnlyCatalogInterface.copy_from_formats, so the VGI DuckDB extension registers a DuckDB CopyFunction for it.

The COPY statement’s file path and the target table’s schema arrive on the bind through :class:vgi.protocol.CopyFromContext (params.bind_call.copy_from / params.init_call.bind_call.copy_from). The COPY options arrive as the function’s normal Arg-annotated arguments — declare them on FunctionArguments exactly like any other function; their doc becomes the option description surfaced by vgi_copy_formats() / vgi_function_arguments().

Subclasses implement :meth:read, emitting Arrow batches whose schema matches expected_schema exactly — DuckDB inserts no cast between the scan and the INSERT, so a type/arity mismatch is rejected by the extension at COPY bind.

class CopyFromFunction

Section titled “class CopyFromFunction”
source

Bases: TableFunctionGenerator[TArgs, _CopyFromState]

Description

Base class for custom COPY ... FROM format readers.

Subclass and:

  • set :attr:COPY_FROM_FORMAT to the SQL FORMAT identifier,
  • declare any options as Arg-annotated FunctionArguments (the source file_path is supplied by the COPY statement, not as an option),
  • implement :meth:read to parse the source and emit Arrow batches matching expected_schema.

Register the subclass in the catalog’s function list like any table function.

Attributes

attribute COPY_FROM_FORMAT

Section titled “attribute COPY_FROM_FORMAT”

str

: SQL FORMAT identifier users type, e.g. COPY t FROM 'x' (FORMAT myfmt).

attribute COPY_FROM_DIRECTION

Section titled “attribute COPY_FROM_DIRECTION”

str

: Reserved for a future COPY ... TO; only "from" is supported today.

attribute COPY_FROM_COMMENT

Section titled “attribute COPY_FROM_COMMENT”

str | None

: Optional free-text comment surfaced by vgi_copy_formats().

Methods

method on_bind

Section titled “method on_bind”
source
on_bind(params: BindParams[TArgs]) -> BindResponse

Bind the output schema to the COPY target’s schema.

DuckDB forces the scan’s output types to the target table’s columns, so a COPY-FROM reader must produce exactly expected_schema.

method on_secrets

Section titled “method on_secrets”
source
on_secrets(params: BindParams[TArgs]) -> None

Request the credentials this reader needs to reach its source.

Override to forward CREATE SECRET values to :meth:read for secret-backed cloud sources. Call params.secrets.get(secret_type, scope=..., name=...) — typically scoping by the source path (params.bind_call.copy_from.file_path) so DuckDB resolves the longest-prefix-matching secret. The framework issues a two-phase bind retry to resolve every requested secret from the caller’s secret store, then surfaces the resolved values on params.secrets (a :class:ResolvedSecrets) at :meth:read time. Pass required=True to get() to make a missing secret fail the bind.

Default: request nothing (no secrets forwarded).

method initial_state

Section titled “method initial_state”
source
initial_state(params: ProcessParams[TArgs]) -> _CopyFromState

Allocate the single-shot read guard.

method process

Section titled “method process”
source
process(
params: ProcessParams[TArgs],
state: _CopyFromState,
out: OutputCollector,
) -> None

Drive :meth:read once, then finish the stream.

method read

Section titled “method read”
source
read(
*,
path: str,
options: TArgs,
expected_schema: pa.Schema,
params: ProcessParams[TArgs],
out: OutputCollector,
) -> None

Parse path and emit Arrow batches via out.emit(...).

Parameters

path
Source path from the COPY … FROM ‘path’ statement.
options
Parsed COPY options (the FunctionArguments instance).
expected_schema
The COPY target’s schema. Every emitted batch must have this exact schema (names + types, in order).
params
Full process parameters (settings, secrets, storage, auth).
out
Collector to emit batches / log. finish() is called for you.
Inherited members (13)
  • get_metadata method · from MetadataMixin — Get the resolved metadata for this function class.
  • describe method · from MetadataMixin — Get metadata as a dictionary (for JSON serialization).
  • logger attribute · from Function
  • storage attribute · from Function
  • FunctionArguments attribute · from TableFunctionBase
  • bind method · from TableFunctionBase — Bind protocol entry point. Do not override; use on_bind().
  • on_init method · from TableFunctionBase — One-time setup after bind, before processing batches.
  • global_init method · from TableFunctionBase — Global init protocol entry point. Do not override; use on_init().
  • cardinality method · from TableFunctionBase — Return the cardinality for the output.
  • dynamic_to_string method · from TableFunctionBase — Return diagnostics rendered as Extra Info under EXPLAIN ANALYZE.
  • statistics method · from TableFunctionBase — Return per-output-column statistics for this invocation.
  • pushdown_filters method · from TableFunctionBase — Get deserialized pushdown filters, or None if not present.
  • on_cancel method · from TableFunctionGenerator