vgi.client
Module overview
VGI client package for communicating with VGI workers.
This package provides:
- Client: A class for programmatic interaction with VGI workers, including both function invocation and catalog operations
- ClientError: Exception raised by Client function operations
- CatalogClientMixin: Mixin class providing catalog operations
- OutputWriter: Helper for writing output in various formats
- main: CLI entry point
Usage (API):
from vgi.client import Client, ClientErrorfrom vgi.arguments import Arguments
with Client("./my_worker.py") as client: for batch in client.table_in_out_function( function_name="echo", arguments=Arguments(positional=[], named={}), input=input_batches, ): process(batch)Usage (Catalog API):
from vgi.client import Client
client = Client("./my_worker")result = client.catalog_attach( name="my_catalog", options={}, data_version_spec=None, implementation_version=None)Usage (CLI):
vgi-client --input data.parquet --function echovgi-client --input data.parquet --function sum_all_columnsclass CatalogClientMixin
Section titled “class CatalogClientMixin”Description
Mixin that adds catalog operations to a VGI Client.
Catalog methods spawn ephemeral connections under the hood — for
subprocess transport a pooled subprocess worker; for HTTP transport a
short-lived http_connect session reusing the Client’s shared
httpx2.Client (bearer token, headers); for TCP transport a short-lived
tcp_connect session. Browsing catalogs over HTTP is the canonical
non-DuckDB use case this mixin supports.
Other attributes expected from Client: _transport (subprocess,
http, or tcp), _base_url (HTTP base URL), _tcp_host / _tcp_port
(TCP endpoint), and _get_or_create_httpx_client() (shared HTTP client
factory).
Attributes
attribute server_path
Section titled “attribute server_path”str | Sequence[str]
Worker command used for subprocess transport — a string to be split, or an argv sequence taken as-is.
Methods
method catalogs
Section titled “method catalogs”catalogs() -> list[CatalogInfo]Get list of catalog discovery records from the worker.
method catalog_attach
Section titled “method catalog_attach”catalog_attach(
*,
name: str,
options: dict[str, Any] | None = None,
data_version_spec: str | None,
implementation_version: str | None,
) -> CatalogAttachResultAttach to a catalog.
method catalog_detach
Section titled “method catalog_detach”catalog_detach(*, attach_opaque_data: AttachOpaqueData) -> NoneDetach from a catalog.
method catalog_create
Section titled “method catalog_create”catalog_create(
*,
name: str,
on_conflict: OnConflict = OnConflict.ERROR,
options: dict[str, Any] | None = None,
) -> NoneCreate a new catalog.
method catalog_drop
Section titled “method catalog_drop”catalog_drop(*, name: str) -> NoneDrop a catalog.
method catalog_version
Section titled “method catalog_version”catalog_version(
*,
attach_opaque_data: AttachOpaqueData,
transaction_opaque_data: TransactionOpaqueData | None = None,
) -> intGet the current catalog version.
method catalog_transaction_begin
Section titled “method catalog_transaction_begin”catalog_transaction_begin(
*,
attach_opaque_data: AttachOpaqueData,
) -> TransactionOpaqueData | NoneBegin a new transaction.
method catalog_transaction_commit
Section titled “method catalog_transaction_commit”catalog_transaction_commit(
*,
attach_opaque_data: AttachOpaqueData,
transaction_opaque_data: TransactionOpaqueData,
) -> NoneCommit a transaction.
method catalog_transaction_rollback
Section titled “method catalog_transaction_rollback”catalog_transaction_rollback(
*,
attach_opaque_data: AttachOpaqueData,
transaction_opaque_data: TransactionOpaqueData,
) -> NoneRollback a transaction.
method schemas
Section titled “method schemas”schemas(
*,
attach_opaque_data: AttachOpaqueData,
transaction_opaque_data: TransactionOpaqueData | None = None,
) -> list[SchemaInfo]List schemas in the catalog.
method schema_get
Section titled “method schema_get”schema_get(
*,
attach_opaque_data: AttachOpaqueData,
transaction_opaque_data: TransactionOpaqueData | None = None,
name: str,
) -> SchemaInfo | NoneGet information about a schema.
method schema_create
Section titled “method schema_create”schema_create(
*,
attach_opaque_data: AttachOpaqueData,
transaction_opaque_data: TransactionOpaqueData | None = None,
name: str,
comment: str | None = None,
tags: dict[str, str] | None = None,
) -> NoneCreate a new schema.
method schema_drop
Section titled “method schema_drop”schema_drop(
*,
attach_opaque_data: AttachOpaqueData,
transaction_opaque_data: TransactionOpaqueData | None = None,
name: str,
ignore_not_found: bool = False,
cascade: bool = False,
) -> NoneDrop a schema.
method schema_contents
Section titled “method schema_contents”schema_contents(
*,
attach_opaque_data: AttachOpaqueData,
transaction_opaque_data: TransactionOpaqueData | None = None,
name: str,
type: SchemaObjectType,
) -> Sequence[TableInfo | ViewInfo | FunctionInfo | MacroInfo]List contents of a schema (tables, views, functions, macros).
method table_get
Section titled “method table_get”table_get(
*,
attach_opaque_data: AttachOpaqueData,
transaction_opaque_data: TransactionOpaqueData | None = None,
schema_name: str,
name: str,
) -> TableInfo | NoneGet information about a table.
method table_column_statistics
Section titled “method table_column_statistics”table_column_statistics(
*,
attach_opaque_data: AttachOpaqueData,
transaction_opaque_data: TransactionOpaqueData | None = None,
schema_name: str,
name: str,
) -> list[ColumnStatistics]Fetch a table’s column statistics, decoded.
Workers may inline statistics on TableInfo (see
TableInfo.column_statistics) or serve them lazily through this per-table
call, which is what TableInfo.supports_column_statistics advertises. Prefer
the inlined copy when present and fall back to this.
method table_create
Section titled “method table_create”table_create(
*,
attach_opaque_data: AttachOpaqueData,
transaction_opaque_data: TransactionOpaqueData | None = None,
schema_name: str,
name: str,
columns: SerializedSchema,
on_conflict: OnConflict = OnConflict.ERROR,
not_null_constraints: list[int] | None = None,
unique_constraints: list[list[int]] | None = None,
check_constraints: list[str] | None = None,
) -> NoneCreate a new table.
method table_drop
Section titled “method table_drop”table_drop(
*,
attach_opaque_data: AttachOpaqueData,
transaction_opaque_data: TransactionOpaqueData | None = None,
schema_name: str,
name: str,
ignore_not_found: bool = False,
cascade: bool = False,
) -> NoneDrop a table.
method table_scan_function_get
Section titled “method table_scan_function_get”table_scan_function_get(
*,
attach_opaque_data: AttachOpaqueData,
transaction_opaque_data: TransactionOpaqueData | None = None,
schema_name: str,
name: str,
at_unit: str | None = None,
at_value: str | None = None,
) -> ScanFunctionResultGet the scan function for a table.
Returns a ScanFunctionResult that tells the VGI DuckDB extension which
DuckDB function to call to obtain the table data.
method table_comment_set
Section titled “method table_comment_set”table_comment_set(
*,
attach_opaque_data: AttachOpaqueData,
transaction_opaque_data: TransactionOpaqueData | None = None,
schema_name: str,
name: str,
comment: str | None,
ignore_not_found: bool = False,
) -> NoneSet or clear the comment on a table.
method table_rename
Section titled “method table_rename”table_rename(
*,
attach_opaque_data: AttachOpaqueData,
transaction_opaque_data: TransactionOpaqueData | None = None,
schema_name: str,
name: str,
new_name: str,
ignore_not_found: bool = False,
) -> NoneRename a table.
method table_column_add
Section titled “method table_column_add”table_column_add(
*,
attach_opaque_data: AttachOpaqueData,
transaction_opaque_data: TransactionOpaqueData | None = None,
schema_name: str,
name: str,
column_definition: SerializedSchema,
ignore_not_found: bool = False,
if_column_not_exists: bool = False,
) -> NoneAdd a new column to a table.
method table_column_drop
Section titled “method table_column_drop”table_column_drop(
*,
attach_opaque_data: AttachOpaqueData,
transaction_opaque_data: TransactionOpaqueData | None = None,
schema_name: str,
name: str,
column_name: str,
ignore_not_found: bool = False,
if_column_exists: bool = False,
cascade: bool = False,
) -> NoneDrop a column from a table.
method table_column_rename
Section titled “method table_column_rename”table_column_rename(
*,
attach_opaque_data: AttachOpaqueData,
transaction_opaque_data: TransactionOpaqueData | None = None,
schema_name: str,
name: str,
column_name: str,
new_column_name: str,
ignore_not_found: bool = False,
) -> NoneRename a column.
method table_column_default_set
Section titled “method table_column_default_set”table_column_default_set(
*,
attach_opaque_data: AttachOpaqueData,
transaction_opaque_data: TransactionOpaqueData | None = None,
schema_name: str,
name: str,
column_name: str,
expression: SqlExpression,
ignore_not_found: bool = False,
) -> NoneSet the default value expression for a column.
method table_column_default_drop
Section titled “method table_column_default_drop”table_column_default_drop(
*,
attach_opaque_data: AttachOpaqueData,
transaction_opaque_data: TransactionOpaqueData | None = None,
schema_name: str,
name: str,
column_name: str,
ignore_not_found: bool = False,
) -> NoneRemove the default value from a column.
method table_column_type_change
Section titled “method table_column_type_change”table_column_type_change(
*,
attach_opaque_data: AttachOpaqueData,
transaction_opaque_data: TransactionOpaqueData | None = None,
schema_name: str,
name: str,
column_definition: SerializedSchema,
expression: SqlExpression | None = None,
ignore_not_found: bool = False,
) -> NoneChange the type of a column.
method table_not_null_drop
Section titled “method table_not_null_drop”table_not_null_drop(
*,
attach_opaque_data: AttachOpaqueData,
transaction_opaque_data: TransactionOpaqueData | None = None,
schema_name: str,
name: str,
column_name: str,
ignore_not_found: bool = False,
) -> NoneRemove NOT NULL constraint from a column.
method table_not_null_set
Section titled “method table_not_null_set”table_not_null_set(
*,
attach_opaque_data: AttachOpaqueData,
transaction_opaque_data: TransactionOpaqueData | None = None,
schema_name: str,
name: str,
column_name: str,
ignore_not_found: bool = False,
) -> NoneAdd NOT NULL constraint to a column.
method view_get
Section titled “method view_get”view_get(
*,
attach_opaque_data: AttachOpaqueData,
transaction_opaque_data: TransactionOpaqueData | None = None,
schema_name: str,
name: str,
) -> ViewInfo | NoneGet information about a view.
method view_create
Section titled “method view_create”view_create(
*,
attach_opaque_data: AttachOpaqueData,
transaction_opaque_data: TransactionOpaqueData | None = None,
schema_name: str,
name: str,
definition: str,
on_conflict: OnConflict = OnConflict.ERROR,
) -> NoneCreate a new view.
method view_drop
Section titled “method view_drop”view_drop(
*,
attach_opaque_data: AttachOpaqueData,
transaction_opaque_data: TransactionOpaqueData | None = None,
schema_name: str,
name: str,
ignore_not_found: bool = False,
cascade: bool = False,
) -> NoneDrop a view.
method view_rename
Section titled “method view_rename”view_rename(
*,
attach_opaque_data: AttachOpaqueData,
transaction_opaque_data: TransactionOpaqueData | None = None,
schema_name: str,
name: str,
new_name: str,
ignore_not_found: bool = False,
) -> NoneRename a view.
method view_comment_set
Section titled “method view_comment_set”view_comment_set(
*,
attach_opaque_data: AttachOpaqueData,
transaction_opaque_data: TransactionOpaqueData | None = None,
schema_name: str,
name: str,
comment: str | None,
ignore_not_found: bool = False,
) -> NoneSet or clear the comment on a view.
method macro_get
Section titled “method macro_get”macro_get(
*,
attach_opaque_data: AttachOpaqueData,
transaction_opaque_data: TransactionOpaqueData | None = None,
schema_name: str,
name: str,
) -> MacroInfo | NoneGet information about a macro.
method macro_create
Section titled “method macro_create”macro_create(
*,
attach_opaque_data: AttachOpaqueData,
transaction_opaque_data: TransactionOpaqueData | None = None,
schema_name: str,
name: str,
macro_type: MacroType,
parameters: list[str],
definition: str,
on_conflict: OnConflict = OnConflict.ERROR,
parameter_default_values: pa.RecordBatch | None = None,
arguments_schema: pa.Schema | None = None,
) -> NoneCreate a new macro.
method macro_drop
Section titled “method macro_drop”macro_drop(
*,
attach_opaque_data: AttachOpaqueData,
transaction_opaque_data: TransactionOpaqueData | None = None,
schema_name: str,
name: str,
ignore_not_found: bool = False,
) -> NoneDrop a macro.
class Client
Section titled “class Client”Bases: CatalogClientMixin
Description
Canonical VGI client — HTTP is the path other-language ports mirror.
Two transports:
- HTTP (
Client.from_http(base_url, bearer_token=...)). The canonical non-DuckDB path. Usesvgi_rpc.http.http_connectunder the hood; transparently resolves pointer batches returned by workers that externalize large outputs (demo storage, S3). Transparently externalizes large input batches when the server advertises upload-URL support. - Subprocess (
Client(server_path)). Python-only convenience for local workers. Uses shell subprocesses + aWorkerPoolfor reuse. Ports don’t need to mirror this.
Catalog operations (catalogs(), schema_contents(), etc.) are
provided by CatalogClientMixin and don’t require start(). They
open a short-lived connection per call (HTTP) or borrow a pooled
subprocess worker.
Function invocation (scalar_function, table_function,
table_in_out_function) requires start() — typically via the
context-manager protocol:
with Client.from_http("http://host:port", bearer_token="...") as c: for batch in c.table_function(function_name="sequence", ...): ...Attributes
attribute THREAD_JOIN_TIMEOUT
Section titled “attribute THREAD_JOIN_TIMEOUT”float
Seconds to wait for a worker thread to join during shutdown.
attribute PROCESS_WAIT_TIMEOUT
Section titled “attribute PROCESS_WAIT_TIMEOUT”float
Seconds to wait for a worker process to exit during shutdown.
attribute server_path
Section titled “attribute server_path”str | Sequence[str]
Subprocess-only. The VGI worker command. A string is
split with shlex.split; pass a sequence to give
the argv exactly, which is what you want for arguments carrying
spaces or quotes ([sys.executable, "-c", script]). No shell
is involved either way, so shell syntax — pipes, redirection,
VAR=value prefixes, ~ expansion — is not interpreted.
attribute passthrough_stderr
Section titled “attribute passthrough_stderr”Subprocess-only. If True, worker stderr is passed through to the parent process’s stderr in real-time.
attribute supports_resumable_scan
Section titled “attribute supports_resumable_scan”bool
Whether this client’s transport can drive :meth:table_scan_resumable.
True only for HTTP, whose producer streams round-trip state in continuation tokens. The pipe/subprocess transport holds a live stream with no serializable resume point.
Methods
method from_http
Section titled “method from_http”from_http(
base_url: str,
*,
bearer_token: str | None = None,
httpx_client: Any | None = None,
external_location: Any | None = None,
worker_limit: int | None = None,
attach_opaque_data: bytes | None = None,
) -> ClientCreate a Client bound to a remote HTTP VGI worker.
Canonical entry point for non-DuckDB callers (e.g. a TypeScript port browsing catalog contents). Subprocess-specific kwargs are not accepted; pool/stderr semantics do not apply.
method from_tcp
Section titled “method from_tcp”from_tcp(
host: str,
port: int,
*,
external_location: Any | None = None,
worker_limit: int | None = None,
attach_opaque_data: bytes | None = None,
) -> ClientCreate a Client bound to a running TCP VGI worker.
Connects via vgi_rpc.rpc.tcp_connect (raw Arrow-IPC framing). The
framing carries no authentication or encryption — only connect to
trusted endpoints on loopback or a trusted network; use
Client.from_http(...) for untrusted networks. Spin up a matching
worker with vgi-fixture-worker --tcp [HOST:]PORT.
method get_worker_stderr
Section titled “method get_worker_stderr”get_worker_stderr() -> strReturn all captured stderr from the worker processes.
Returns stderr output from the primary worker and all additional workers spawned for parallel processing. The output is accumulated in a shared buffer throughout the client’s lifetime.
This method is thread-safe and can be called while processing is ongoing, though the buffer may not yet contain all output until the workers have completed.
Note
This method only returns data when passthrough_stderr=False was set in the constructor. When passthrough_stderr=True, stderr goes directly to the parent process’s stderr and this method returns an empty string.
method start
Section titled “method start”start() -> NoneStart the primary worker subprocess.
Spawns the worker process using the server_path configured in init,
sets up RPC transport, and creates a typed VgiProtocol proxy for
method calls.
After this method returns, the client is ready to invoke functions via table_in_out_function(), table_function(), or scalar_function(). When using the context manager protocol (with statement), this method is called automatically.
The stderr buffer is cleared when start() is called, so any stderr from previous runs is discarded.
method stop
Section titled “method stop”stop(*, force: bool = False) -> intStop all worker subprocesses and clean up resources.
Terminates all workers in the following order:
- Stops all additional workers (spawned for parallel processing)
- Stops the primary worker
- Waits for all stderr drain threads to complete (with timeout)
- Resets all internal state
After this method returns, the client can be started again with start(). When using the context manager protocol (with statement), this method is called automatically on exit.
Cancelling an in-flight call
A graceful stop waits for a worker that is blocked inside a handler, so it
cannot be used to abandon a scan that has overrun its budget. Pass
force=True to SIGKILL direct subprocess workers first, which unblocks any
thread waiting on their output immediately. Note this only applies to direct
subprocess workers: a pooled worker is returned to its pool rather than owned
by this client, so construct the client with pool=None when you need to be
able to cancel it. HTTP and TCP workers are already prompt to close.
method server_capabilities
Section titled “method server_capabilities”server_capabilities() -> AnyReturn the HTTP server’s advertised capabilities.
Only valid for HTTP-mode clients. The returned
HttpServerCapabilities carries max_request_bytes,
upload_url_support, and max_upload_bytes — the fields the
client consults before deciding to externalize large input batches
via upload URLs (see Phase 4 of the whimsical-mccarthy plan).
method table_in_out_function
Section titled “method table_in_out_function”table_in_out_function(
*,
function_name: str,
schema_name: str,
input: Iterator[pa.RecordBatch],
arguments: Arguments | None = None,
bind_result_callback: Callable[[BindResponse], None] | None = None,
projection_ids: list[int] | None = None,
pushdown_filters: bytes | None = None,
settings: dict[str, Any] | None = None,
transaction_opaque_data: bytes | None = None,
) -> Generator[pa.RecordBatch]Invoke a table-in-out function on the worker and stream results.
For parallel processing (max_workers > 1), input batches are distributed round-robin across workers using dedicated threads. Output order may not match input order in parallel mode. Only the primary worker receives the FINALIZE phase and produces final aggregated output.
method table_buffering_function
Section titled “method table_buffering_function”table_buffering_function(
*,
function_name: str,
schema_name: str,
input: Iterator[pa.RecordBatch],
arguments: Arguments | None = None,
bind_result_callback: Callable[[BindResponse], None] | None = None,
projection_ids: list[int] | None = None,
pushdown_filters: bytes | None = None,
settings: dict[str, Any] | None = None,
transaction_opaque_data: bytes | None = None,
) -> Generator[pa.RecordBatch]Invoke a TableBufferingFunction (Sink+Source) and stream results.
This mirrors the C++ PhysicalVgiTableBufferingFunction operator
rather than the streaming INPUT/FINALIZE path used by
:meth:table_in_out_function. The sequence is:
bind→init(phase=TABLE_BUFFERING)on the primary worker. The sink init persists init metadata to cold storage so any pool worker can serve subsequent process/combine RPCs; its stream carries no data, so it is closed immediately after the header.table_buffering_process(unary) per input batch — the worker sinks the batch and returns an opaquestate_id.table_buffering_combine(unary) once at end-of-input — the worker hands allstate_ids to usercombine()and returns opaquefinalize_state_ids (the source-side partition keys).init(phase=TABLE_BUFFERING_FINALIZE, finalize_state_id=...)per finalize key — a producer stream driving userfinalize()per tick. Output batches are yielded in finalize-key order.table_buffering_destructor(unary, best-effort) for cleanup.
Unlike :meth:table_in_out_function this driver runs entirely on the
primary worker connection (process/combine are unary RPCs); the
worker buffers all input regardless, so the aggregate result is
identical to the distributed C++ path.
method table_function
Section titled “method table_function”table_function(
*,
function_name: str,
schema_name: str,
arguments: Arguments | None = None,
bind_result_callback: Callable[[BindResponse], None] | None = None,
projection_ids: list[int] | None = None,
pushdown_filters: bytes | None = None,
settings: dict[str, Any] | None = None,
transaction_opaque_data: bytes | None = None,
) -> Generator[pa.RecordBatch]Invoke a table function (source function) and stream output batches.
Table functions generate output batches without receiving input data. They are useful for data sources, generators, or functions that produce results based solely on their arguments.
For parallel processing (max_workers > 1), output is read from all workers concurrently using threads. Output order is non-deterministic.
method table_scan_resumable
Section titled “method table_scan_resumable”table_scan_resumable(
*,
function_name: str,
schema_name: str,
arguments: Arguments | None = None,
projection_ids: list[int] | None = None,
pushdown_filters: bytes | None = None,
settings: dict[str, Any] | None = None,
transaction_opaque_data: bytes | None = None,
resume_token: bytes | None = None,
) -> ResumableTableScanOpen (or resume) a resumable table-function scan.
Resumable variant of :meth:table_function: the returned
:class:ResumableTableScan yields (batch, token) one batch at a
time, surfacing the worker’s continuation token so a stateless caller
can persist it and resume on another process/node.
When resume_token is given, the scan continues from that token
(the bind/init is still issued — the upstream’s first turn is produced
and discarded — so the same function_name/projection/filters must
be supplied). When None, a fresh scan starts.
method table_scan_continue
Section titled “method table_scan_continue”table_scan_continue(
*,
resume_token: bytes,
output_schema: pa.Schema | None = None,
) -> ResumableTableScanResume a producer table scan from a continuation token WITHOUT re-binding.
The cheap counterpart to table_scan_resumable(resume_token=...): a continuation
token is a signed, self-describing snapshot of the worker’s producer state, so the
server recovers state + schemas + function identity from the token alone. This skips
the bind/init round-trip (and the discarded first turn) that
table_scan_resumable pays — the right primitive for a stateless relay that holds
a per-batch token and resumes on any node every batch.
The client must be started and connected to a worker that honours the token (the
token is verified against the caller’s auth identity, and routed by the same
init stream method that minted it). HTTP transport only.
method scalar_function
Section titled “method scalar_function”scalar_function(
*,
function_name: str,
schema_name: str,
input: Iterator[pa.RecordBatch],
arguments: Arguments | None = None,
bind_result_callback: Callable[[BindResponse], None] | None = None,
settings: dict[str, Any] | None = None,
secrets: dict[str, Any] | None = None,
transaction_opaque_data: bytes | None = None,
) -> Generator[pa.RecordBatch]Invoke a scalar function on the worker and stream results.
Scalar functions transform input batches to single-column output with 1:1 row mapping. Processing ends when input is exhausted.
For parallel processing (max_workers > 1), input batches are distributed round-robin across workers using dedicated threads. Output order may not match input order in parallel mode.
Inherited members (37)
catalogsmethod · from CatalogClientMixin — Get list of catalog discovery records from the worker.catalog_attachmethod · from CatalogClientMixin — Attach to a catalog.catalog_detachmethod · from CatalogClientMixin — Detach from a catalog.catalog_createmethod · from CatalogClientMixin — Create a new catalog.catalog_dropmethod · from CatalogClientMixin — Drop a catalog.catalog_versionmethod · from CatalogClientMixin — Get the current catalog version.catalog_transaction_beginmethod · from CatalogClientMixin — Begin a new transaction.catalog_transaction_commitmethod · from CatalogClientMixin — Commit a transaction.catalog_transaction_rollbackmethod · from CatalogClientMixin — Rollback a transaction.schemasmethod · from CatalogClientMixin — List schemas in the catalog.schema_getmethod · from CatalogClientMixin — Get information about a schema.schema_createmethod · from CatalogClientMixin — Create a new schema.schema_dropmethod · from CatalogClientMixin — Drop a schema.schema_contentsmethod · from CatalogClientMixin — List contents of a schema (tables, views, functions, macros).table_getmethod · from CatalogClientMixin — Get information about a table.table_column_statisticsmethod · from CatalogClientMixin — Fetch a table’s column statistics, decoded.table_createmethod · from CatalogClientMixin — Create a new table.table_dropmethod · from CatalogClientMixin — Drop a table.table_scan_function_getmethod · from CatalogClientMixin — Get the scan function for a table.table_comment_setmethod · from CatalogClientMixin — Set or clear the comment on a table.table_renamemethod · from CatalogClientMixin — Rename a table.table_column_addmethod · from CatalogClientMixin — Add a new column to a table.table_column_dropmethod · from CatalogClientMixin — Drop a column from a table.table_column_renamemethod · from CatalogClientMixin — Rename a column.table_column_default_setmethod · from CatalogClientMixin — Set the default value expression for a column.table_column_default_dropmethod · from CatalogClientMixin — Remove the default value from a column.table_column_type_changemethod · from CatalogClientMixin — Change the type of a column.table_not_null_dropmethod · from CatalogClientMixin — Remove NOT NULL constraint from a column.table_not_null_setmethod · from CatalogClientMixin — Add NOT NULL constraint to a column.view_getmethod · from CatalogClientMixin — Get information about a view.view_createmethod · from CatalogClientMixin — Create a new view.view_dropmethod · from CatalogClientMixin — Drop a view.view_renamemethod · from CatalogClientMixin — Rename a view.view_comment_setmethod · from CatalogClientMixin — Set or clear the comment on a view.macro_getmethod · from CatalogClientMixin — Get information about a macro.macro_createmethod · from CatalogClientMixin — Create a new macro.macro_dropmethod · from CatalogClientMixin — Drop a macro.
class ClientError
Section titled “class ClientError”Bases: Exception
Description
Error raised by Client operations.
The first line of str(ClientError) is the remote exception as the
worker raised it ({error_type}: {error_message}), so that whatever
a user typed into their raise ValueError(...) shows up at the top of
their traceback instead of being buried under VGI framing. Remote
traceback and worker-stderr excerpts, when present, follow after an
empty line.
Methods
method from_rpc_error
Section titled “method from_rpc_error”from_rpc_error(e: RpcError) -> ClientErrorCreate a ClientError from an RpcError, including remote traceback.
Lead with the user’s exception (error_type: error_message) so
the most actionable line is first. The Remote traceback section
trails and is only included when the worker produced one.
function main
Section titled “function main”main() -> NoneCLI entry point for vgi-client.
class OutputWriter
Section titled “class OutputWriter”Description
Handles writing output batches in various formats.
Supported formats
- json: JSON Lines format (one JSON object per row)
- csv: CSV with header
- parquet: Apache Parquet columnar format
- arrow-ipc: Apache Arrow IPC streaming format (useful for debugging)
The arrow-ipc format writes batches in the standard Arrow IPC streaming format, which can be read by any Arrow implementation. This is useful for:
- Debugging VGI protocol issues- Inspecting raw output data with tools like pyarrow or arrow CLI- Piping data to other Arrow-aware toolsAttributes
attribute output_file
Section titled “attribute output_file”Path to output file, “-” for stdout, or None for logging.
attribute format
Section titled “attribute format”Output format (“parquet”, “csv”, “json”, or “arrow-ipc”).
Methods
method write_batch
Section titled “method write_batch”write_batch(batch: pa.RecordBatch) -> NoneWrite a batch to the output destination in the configured format.
method close
Section titled “method close”close() -> NoneClose the underlying writer if one exists.
class ResumableTableScan
Section titled “class ResumableTableScan”Description
A resumable, one-batch-at-a-time handle on an upstream table-function scan.
Unlike :meth:Client.table_function (a live generator that hides the
server’s continuation token), each :meth:next returns (batch, token)
where token is the worker’s serialized producer state AFTER batch.
A stateless client (e.g. a load-balanced proxy) can persist token, drop
the connection, and resume on another node via
Client.table_scan_resumable(resume_token=token, ...).
Single-worker: reads the primary stream only (parallel max_workers>1
reads are unordered and not resumable from a single token).
Methods
method next
Section titled “method next”next() -> tuple[pa.RecordBatch | None, bytes | None]Return (batch, resume_token); (None, None) at end-of-stream.
resume_token resumes the scan AFTER batch on any node.
method close
Section titled “method close”close() -> NoneRelease the underlying stream (no-op over HTTP — stateless).
class ResumeUnsupported
Section titled “class ResumeUnsupported”Bases: ClientError
Description
Raised when a resumable scan is requested on a non-resumable transport.
Only the HTTP transport round-trips producer state in continuation tokens,
so only HTTP clients can drive :meth:Client.table_scan_resumable. On the
pipe/subprocess transport the stream is a live connection with no
serializable resume point; the caller must keep the live stream in-process
instead.
Inherited members (1)
from_rpc_errormethod · from ClientError — Create aClientErrorfrom anRpcError, including remote traceback.