Table-in-out functions

On this page

Streaming a relation through, batch by batch.

trait TableInOutFunction

source

pub trait TableInOutFunction: Send + Sync {
  fn name(&self) -> &str;
  fn metadata(&self) -> FunctionMetadata;
  fn argument_specs(&self) -> Vec<ArgSpec>;
  fn secret_lookups(&self, _params: &BindParams) -> Vec<crate::secrets::SecretLookup> {
      Vec::new()
  }
  fn on_bind(&self, params: &BindParams) -> Result<BindResponse> {
      let input = params
          .input_schema
          .clone()
          .ok_or_else(|| RpcError::value_error("table-in-out requires an input schema"))?;
      Ok(BindResponse {
          output_schema: input,
          opaque_data: Vec::new(),
      })
  }
  fn process(&self, params: &ProcessParams, batch: &RecordBatch) -> Result<Vec<RecordBatch>> {
      Ok(vec![project_batch(batch, &params.output_schema)?])
  }

  fn process_out(
      &self,
      params: &ProcessParams,
      batch: &RecordBatch,
      out: &mut TableInOutOutput,
  ) -> Result<()> {
      for b in self.process(params, batch)? {
          out.emit(b);
      }
      Ok(())
  }

  fn has_finish(&self) -> bool {
      false
  }

  fn finish(&self, _params: &ProcessParams) -> Result<Vec<RecordBatch>> {
      Ok(Vec::new())
  }
}

Description

A table-in-out VGI function.

struct EmitOptions

source

pub struct EmitOptions {
  pub metadata: Option<HashMap<String, String>>,
  pub cache_control: Option<CacheControl>,
  pub parent_rows: Option<Vec<i32>>,
}

Description

Options for one emitted table-in-out output batch (all optional).

constant PARENT_ROW_METADATA_KEY

source

pub const PARENT_ROW_METADATA_KEY: &str = “vgi_rpc.parent_row#b64”;

Description

Per-batch wire-metadata key carrying per-output-row provenance for the batched correlated-LATERAL operator: a base64-encoded raw little-endian int32[] where element i is the 0-based index (into this call’s input batch) of the row that produced output row i. Absent metadata = identity 1→1 map (the extension assumes it, and requires output rows == input rows).

struct TableInOutOutput

source

pub struct TableInOutOutput {
  pub(crate) items: Vec<(RecordBatch, Option<HashMap<String, String>>)>,
}

Description

Collects one process call’s output batches with optional per-batch metadata — the emit surface for [TableInOutFunction::process_out]. Plain 1→1 transforms keep returning Vec<RecordBatch> from [TableInOutFunction::process]; override process_out (and emit through this) only when a batch needs [EmitOptions] (cache control, LATERAL provenance, raw metadata).

Methods

method emit

source

pub fn emit(&mut self, batch: RecordBatch)

Emit a batch with no per-batch metadata.

method emit_with

source

pub fn emit_with(&mut self, batch: RecordBatch, opts: EmitOptions) -> Result<()>

Emit a batch with [EmitOptions] (metadata / cache control / LATERAL provenance). Mirrors the Python out.emit(batch, metadata=..., cache_control=..., parent_rows=...) kwargs.

function arc

source

pub fn arc(s: Schema) -> Arc<Schema>

Description

Build an Arc<Schema>.

function project_batch

source

pub fn project_batch(batch: &RecordBatch, schema: &SchemaRef) -> Result<RecordBatch>

Description

Project a batch to schema’s columns by name (projection pushdown).