COPY formats

On this page

Reading and writing your own format through COPY … FROM / TO.

trait CopyFromFunction

source

pub trait CopyFromFunction: Send + Sync {
  fn format(&self) -> &str;

  fn handler_name(&self) -> &str;

  fn comment(&self) -> Option<String> {
      None
  }

  fn metadata(&self) -> FunctionMetadata {
      FunctionMetadata::default()
  }

  fn argument_specs(&self) -> Vec<ArgSpec>;

  fn secret_lookups(&self, _params: &BindParams) -> Vec<crate::secrets::SecretLookup> {
      Vec::new()
  }

  fn read(
      &self,
      ctx: &CopyFromReadContext,
      out: &mut OutputCollector,
  ) -> Result<Vec<RecordBatch>>;

  fn read_stream(&self, _ctx: &CopyFromReadContext) -> Result<Option<Box<dyn TableProducer>>> {
      Ok(None)
  }
}

Description

A custom COPY ... FROM format reader.

Implement format, handler_name, argument_specs (the COPY options — the source file_path is supplied by the COPY statement, not an option), and read (parse the source and return Arrow batches matching the target schema). Register with Worker::register_copy_from.

trait CopyToFunction

source

pub trait CopyToFunction: Send + Sync {
  fn format(&self) -> &str;

  fn handler_name(&self) -> &str;

  fn comment(&self) -> Option<String> {
      None
  }

  fn metadata(&self) -> FunctionMetadata {
      FunctionMetadata::default()
  }

  fn argument_specs(&self) -> Vec<ArgSpec>;

  fn secret_lookups(&self, _params: &BindParams) -> Vec<crate::secrets::SecretLookup> {
      Vec::new()
  }

  fn ordered(&self) -> bool {
      false
  }

  fn write(&self, ctx: &CopyToWriteContext, batch: &RecordBatch) -> Result<()>;

  fn close(&self, ctx: &CopyToCloseContext) -> Result<i64>;
}

Description

A custom COPY ... TO format writer.

Implement format, handler_name, argument_specs (the COPY options — the destination file_path is supplied by the COPY statement, not an option), write (persist one batch to a shard), and close (terminal destination write). Register with Worker::register_copy_to.

struct CopyFromReadContext

source

pub struct CopyFromReadContext<'a> {
  pub path: &'a str,
  pub options: &'a Arguments,
  pub expected_schema: &'a SchemaRef,
  pub params: &'a ProcessParams,
}

Description

Context handed to [CopyFromFunction::read].

struct CopyFromTable

source

pub struct CopyFromTable(pub Arc<dyn CopyFromFunction>);

Description

Adapter that exposes a [CopyFromFunction] as an ordinary producer-mode [TableFunction], so the entire table bind/init/scan path is reused. The COPY-FROM context arrives via [ProcessParams::copy_from].

struct CopyToBuffering

source

pub struct CopyToBuffering(pub Arc<dyn CopyToFunction>);

Description

Adapter that exposes a [CopyToFunction] as a [TableBufferingFunction], so the entire buffering sink/combine RPC path is reused. There is no Source phase: combine returns an empty finalize list and finalize_producer is never invoked.

struct CopyToCloseContext

source

pub struct CopyToCloseContext<'a> {
  pub path: &'a str,
  pub options: &'a Arguments,
  pub storage: &'a Arc<dyn FunctionStorage>,
  pub execution_id: &'a [u8],
  pub input_schema: Option<&'a SchemaRef>,
  pub params: &'a BufferingParams,
}

Description

Context handed to [CopyToFunction::close] (terminal write, once).

struct CopyToWriteContext

source

pub struct CopyToWriteContext<'a> {
  pub path: &'a str,
  pub options: &'a Arguments,
  pub storage: &'a Arc<dyn FunctionStorage>,
  pub execution_id: &'a [u8],
  pub params: &'a BufferingParams,
}

Description

Context handed to [CopyToFunction::write] (per input batch).